A method and device for parsing emails with missing content

Abstract

The present application discloses a method and device for parsing emails with missing content, relating to the field of email technology. The method comprises: extracting and classifying the text content of the email text to obtain the category of the text content; the category includes plain text format, web page format, and attachment; using the text content in the plain text format and the web page format as the text to be decoded; decoding the text to be decoded according to the corresponding encoding method to obtain a decoded string; performing segmented character set detection on the decoded string according to a set byte length, and adding the character set corresponding to the character set detection result that meets the set conditions to a character set list; performing character conversion on the decoded string by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment to obtain the original email text. The present application realizes the effective parsing of emails with missing content.

Description

Technical Field

[0001] The present application relates to the field of email technology, and in particular to a method and device for parsing emails with missing content. Background Art

[0002] With the advent of the information age, electronic mail (EML) has become a vital means of communication for modern people. However, this has also brought with it a series of email security issues. An increasing number of insiders are illegally uploading sensitive information via email, resulting in significant losses and data leakage risks. Email security incidents are a frequent occurrence, and research on email security is crucial for improving network security, preventing data leaks, and facilitating data forensics.

[0003] In addition to the specific content of the message, emails also contain information such as the sender, recipient, subject, attachments, time, and various metadata. These texts are encoded and displayed through related email applications.

[0004] However, humans cannot directly understand encoded text and require the use of relevant tools for parsing. Existing email parsing methods and tools mainly parse based on the structure and key fields of the email format. These methods are limited to complete email content and cannot effectively parse emails with missing content. They often only parse partially formatted and structurally complete text content or the parsed text is still garbled, requiring manual screening and preventing rapid batch parsing of emails. Summary of the Invention

[0005] The purpose of this application is to provide a method and device for parsing emails with missing content, which can effectively parse emails with missing content.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a method for parsing emails with missing content, comprising:

[0008] Extracting and classifying the text content of the email text to obtain categories of the text content; the categories include plain text format, web page format and attachment; and using the text content in the plain text format and the web page format as the text to be decoded;

[0009] Decoding the text to be decoded according to the corresponding encoding method to obtain a decoded string;

[0010] Performing segmented character set detection on the decoded character string according to the set byte length, and adding the character sets corresponding to the character set detection results that meet the set conditions to the character set list;

[0011] The decoded character string is converted into characters by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment, thereby obtaining the original text of the email.

[0012] Optionally, extracting and classifying the email text to obtain the category of the text content may include:

[0013] Based on the encoding features, the regular matching method is used to extract the text content of the email text. If there is Base64-encoded text in the extracted text content, the Base64-encoded text is vectorized to obtain the text to be classified;

[0014] Inputting the file to be classified into a classification model to obtain a classification prediction result of the text content; the classification model is a trained neural network;

[0015] Based on the inherent order of the categories in the email, the classification prediction result is corrected to obtain the category of the text content.

[0016] Optionally, based on the inherent order of categories in the email, the classification prediction result is corrected to obtain the category of the text content, specifically including:

[0017] Input the classification prediction results into the first sequence;

[0018] Initialize a second sequence, and initialize the first element in the second sequence to the first element in the first sequence;

[0019] Traverse the elements of the first sequence except the first element, and for each element of the first sequence that is traversed: if the current traversed element is the same as the previous element, add the current traversed element to the second sequence; if the current traversed element is smaller than the previous element, add the previous element to the second sequence; if the current traversed element is larger than the previous element, determine whether the current traversed element is equal to the next element; if so, add the current traversed element to the second sequence; otherwise, add the previous element to the second sequence; after completing the traversal of the first sequence, obtain a corrected second sequence;

[0020] generating a ladder list according to the corrected second sequence, wherein the element values ​​in each ladder column of the ladder list are the same;

[0021] Calculate the average probability for each step;

[0022] Traversing the ladder list except the first ladder, for each ladder traversed: if the element value of the previous ladder is greater than the element value of the next ladder, and the average probability of the previous ladder is greater than or equal to the average probability of the next ladder, then correcting the values ​​of all elements of the next ladder to the element value of the previous ladder; if the average probability of the previous ladder is less than the average probability of the next ladder, then correcting the values ​​of all elements of the previous ladder to the element value of the next ladder, after the ladder list traversal is completed, obtaining a corrected ladder list;

[0023] According to the corrected ladder list, the category of the text content is obtained.

[0024] Optionally, the neural network includes an embedding layer, a first convolution module, a second convolution module, a long short-term memory network and a fully connected layer connected in sequence; the first convolution module and the second convolution module both include a one-dimensional convolution layer, a ReLU function, a maximum pooling layer and a Dropout layer connected in sequence.

[0025] Optionally, segment character set detection is performed on the decoded character string according to a set byte length, and character sets corresponding to character set detection results that meet set conditions are added to a character set list, specifically including:

[0026] Starting from the first byte of the decoded string, byte text of each set byte length is intercepted in sequence for character set detection. For each byte text, if the confidence in the detection result is greater than or equal to 0.9, the detection result is added to the character set list; otherwise, the byte text is shifted back by 1 byte and the text of the set byte length is intercepted to continue character set detection until the character set detection of the last byte text is completed.

[0027] Optionally, character conversion is performed on the decoded string by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment to obtain the original email text, specifically including:

[0028] Counting the character sets in the character set list, and sorting the character sets in descending order of the counted number to obtain a character set sorting list;

[0029] Each character set in the character set sorted list is traversed, and the decoded string is converted into characters using the currently traversed character set to obtain a current decoding result. The current decoding result is subjected to a semantic coherence judgment. If the current decoding result passes the semantic coherence judgment, the current decoding result is used as the original email text and traversing the character set sorted list is stopped. If the current decoding result does not pass the semantic coherence judgment, the next character set is traversed.

[0030] Optionally, the semantic coherence of the current decoding result is judged, specifically including:

[0031] Extract the features of the current decoding result;

[0032] The extracted feature parameters are input into a semantic coherence judgment model to output a semantic judgment result; the semantic judgment result is semantic coherence or semantic incoherence; the semantic coherence judgment model is obtained by training a machine learning model using a training set.

[0033] Optionally, after performing character conversion on the decoded string by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment and the original email text is obtained, the email parsing method for missing content further includes:

[0034] The language type of the original email text is detected, and if the detected language type is not the target language, the original email text is translated into the target language.

[0035] Secondly, this application provides a method for parsing emails with missing content, including:

[0036] Extract text content from email text based on key fields;

[0037] Decode the extracted text content;

[0038] If the decoding is successful, the original email text is obtained and the program stops;

[0039] If the decoding is unsuccessful, the text content of the email is extracted and classified to obtain the categories of the text content; the categories include plain text format, web page format and attachment; the text content in the plain text format and web page format is used as the text to be decoded;

[0040] Decoding the text to be decoded according to the corresponding encoding method to obtain a decoded string;

[0041] Performing segmented character set detection on the decoded character string according to the set byte length, and adding the character sets corresponding to the character set detection results that meet the set conditions to the character set list;

[0042] The decoded character string is converted into characters by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment, thereby obtaining the original text of the email.

[0043] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the above-mentioned methods for parsing emails with missing content.

[0044] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0045] The present application provides an email parsing method and device for missing content, which extracts and classifies text content from email text to obtain text content categories; the categories include plain text format, web page format and attachment; the plain text format and web page format and text content are used as text to be decoded; thereby, the text content of the attachment type is eliminated, reducing the impact on the original email text and improving decoding efficiency; the text to be decoded is decoded according to the corresponding encoding method to obtain a decoded string; the decoded string is segmented and character set detection is performed on the decoded string according to the set byte length, and the character set corresponding to the character set detection result that meets the set conditions is added to the character set list; the character sets in the character set list are traversed to perform character conversion on the decoded string until the decoding result passes the semantic coherence judgment, and the original email text is obtained, which solves the problem that the existing technical means cannot parse emails with missing content and realizes effective parsing of emails with missing content. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 This is a diagram of an application environment for an email parsing method for missing content in one embodiment of the present application;

[0048] Figure 2 A flowchart of a method for parsing emails with missing content provided in one embodiment of the present application;

[0049] Figure 3 This is an architectural diagram of a method for parsing emails with missing content, provided in one embodiment of the present application;

[0050] Figure 4 A schematic diagram of a text preprocessing process provided in one embodiment of the present application;

[0051] Figure 5 A schematic diagram of a decoding process provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of a character set detection process provided in an embodiment of the present application;

[0053] Figure 7 A flowchart of a method for parsing emails with missing content provided in another embodiment of the present application;

[0054] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0056] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0057] EML is a semi-structured text that belongs to the Multipurpose Internet Mail Extensions (MIME) type of text. When the content in the text is missing, existing extraction methods and tools cannot extract content based on the original text position features or field features. The main manifestations are: First, extraction errors and redundant information are extracted, causing interference for subsequent decoding. Second, it is impossible to distinguish between Base64-encoded plain text format (text / plain), web page format (text / html) and attachments (attachments), making it difficult to decode. Third, no content can be extracted.

[0058] The email parsing method for missing content provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the email to be processed to the server 104. After the server 104 receives the email to be processed, the server 104 extracts and classifies the text content of the email to obtain the category of the text content; the categories include plain text format, web page format, and attachment; the text content in the plain text format and the web page format are used as the text to be decoded; the text to be decoded is decoded according to the corresponding encoding method to obtain a decoded string; the decoded string is segmented and character set detection is performed on the decoded string according to a set byte length, and the character set corresponding to the character set detection result that meets the set conditions is added to the character set list; the decoded string is character converted by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment to obtain the original email text. The server 104 can feedback the obtained original email text to the terminal 102. In addition, in some embodiments, the email parsing method for missing content can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform email parsing on the email to be processed, or the server 104 can obtain the email to be processed from the data storage system and perform email parsing on the email to be processed.

[0059] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smartphones, tablet computers, and portable wearable devices. Portable wearable devices may include smart watches and smart bracelets. The server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers, or may be a cloud server.

[0060] In an exemplary embodiment, Figure 2 and Figure 3 As shown, a method for parsing mails with missing content is provided. The method is executed by a computer device, specifically a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used for explanation, including the following steps 201 to 204.

[0061] in:

[0062] Step 201: extract and classify the email text to obtain the categories of the text content; the categories include plain text format, web page format and attachment; and use the text content in the plain text format and web page format as the text to be decoded.

[0063] Step 202: Decode the text to be decoded according to the corresponding encoding method to obtain a decoded character string.

[0064] Step 203: Perform segmented character set detection on the decoded character string according to the set byte length, and add the character sets corresponding to the character set detection results that meet the set conditions to the character set list.

[0065] Step 204: performing character conversion on the decoded string by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment, thereby obtaining the original email text.

[0066] By implementing the above-mentioned steps 201 to 204, the present application provides a method for parsing emails with missing content, specifically a method for parsing emails with missing content, which removes the text content of the attachment type, reduces the impact on the original email text, and improves decoding efficiency; traverses the character set list to perform character conversion on the decoded string until the decoding result passes the semantic coherence judgment, and obtains the original email text. The decoded string is converted into character through a targeted character set, thereby achieving effective parsing of emails with missing content.

[0067] This application extracts and classifies text content based on regular matching and neural networks, wherein step 201 specifically includes:

[0068] Step 211: Based on the encoding features, a regular matching method is used to extract the text content of the email text. If Base64-encoded text exists in the extracted text content, the Base64-encoded text is vectorized to obtain the text to be classified.

[0069] If there is no Base64-encoded text in the extracted text content, the file to be classified is the text content extracted from the email text using the regular matching method; if there is Base64-encoded text in the extracted text content, the file to be classified is the text obtained by replacing the Base64-encoded text in the extracted text content with the vectorized Base64-encoded text.

[0070] Through regular matching, "noise" content such as header information, boundary characters, and feature fields is filtered out, and Base64-encoded and Quoted-printable-encoded text content are extracted respectively, which serves as preprocessing for subsequent text type classification and decoding.

[0071] The Base64 encoded regular expression is:

[0072] r"^([A-Za-z0-9+ / ]{4})(([A-Za-z0-9+\n / ]{1})*)([A-Za-z0-9+ / ]{4}|[A-Za-z0-9+ / ]{3}=|[A-Za-z0-9+ / ]{2}==)$".

[0073] The regular expression for Quoted-printable encoding is: r"=(?:[A-Fa-f0-9]{2})+".

[0074] Regarding n-grams, we draw on natural language processing to apply the idea of ​​using n-grams to analyze word sequences in text to detect topics to the study of file fragment type identification. Files can be compared to text, and the bytes that make up a file can be compared to word sequences.

[0075] For a single byte, n = 1, so this byte frequency distribution is also called a 1-gram or unigram. The frequency of each byte in a file or file fragment is counted. A byte is composed of 8 bits, and its value range is 2^8, or 0 to 255. Therefore, for each file fragment, the frequency of each byte can be represented, forming a 256-dimensional (0 to 255) feature vector.

[0076] If two consecutive bytes are counted each time, it is called a 2-gram or bigram, three consecutive bytes are called a 3-gram or trigram, and so on. 16 = 65536 different variations. The large number of features will lead to sparse data.

[0077] Secondly, Base64 has ˋA-Za-z0-9+ / =', a total of 65 characters, and uses the n-gram concept to serialize text.

[0078] First, count the number of occurrences of each character in the training data, and create a corresponding vocabulary from 0 to 64 according to the number of occurrences. Similarly, count the number of occurrences of every two characters, sort them, and add them to the vocabulary starting from 65, and finally add one <pad>Fill character.

[0079] The Base64-encoded text is vectorized (Quoted-printable encoded text is not vectorized), specifically including the following steps:

[0080] 1): Standardize the text length, that is, pad or truncate the Base64-encoded text to a fixed length (the fixed length is 72 characters in this example) to obtain a standardized length text.

[0081] 2): Construct a vocabulary table according to the 65 characters of Base64, and assign an index to each character in the vocabulary table, wherein the vocabulary table includes the 65 characters of Base64 and words composed of every two characters in the 65 characters.

[0082] 3): Vectorize the normalized length text to obtain a feature vector: convert each character into a corresponding index according to the vocabulary.

[0083] 4): Calculate the entropy of the Base64-encoded text and scale it to [0, 1], then add the entropy value to the feature vector.

[0084] 5) Calculate the complexity of the Base64-encoded text, specifically the Lempel-Ziv complexity, and scale it to between [0, 1], then add it to the feature vector to obtain the vectorized Base64-encoded text.

[0085] Step 2012: Input the file to be classified into a classification model to obtain a classification prediction result of the text content; the classification model is a trained neural network.

[0086] The neural network includes an embedding layer, a first convolution module, a second convolution module, a long short-term memory network (LSTM) and a fully connected layer connected in sequence; the first convolution module and the second convolution module both include a one-dimensional convolution layer, a ReLU function, a maximum pooling layer and a Dropout layer connected in sequence.

[0087] The output features of the second convolutional module are fed into an LSTM to better learn the correlations between features used for classification. Finally, the output of each sample generated by the LSTM is fed into the subsequent fully connected layer, which outputs the classification prediction result. The average accuracy reached approximately 88%.

[0088] Considering that the model's prediction accuracy cannot reach 100%, if the output results introduce misclassification, it will significantly interfere with subsequent decoding and translation. Therefore, the prediction results need to be corrected. The encoded text in the email, text / plain, text / html, and attachments, is sequentially arranged. The classification prediction results are sequentially ascending, showing a "staircase"-like upward trend (the three labels correspond to 1, 2, and 3, respectively). It is impossible for the following number to be smaller than the previous one, for example, [1, 1, 1, 2, 2, 2, 3, 3, 3]. If the number after the label is smaller than the previous one, for example, [1, 1, 1, 2, 2, 1, 3, 3], then a classification error has occurred. The error may occur at the sixth index position, where the value should not be 1 and may be 2 or 3. Alternatively, the values ​​at the fourth and fifth index positions may be incorrect, where the values ​​at these positions may both be 1. Based on this characteristic, a local and global calibration method is used for correction (step 2013).

[0089] Step 2013: Based on the inherent order of each category in the email, the classification prediction result is corrected to obtain the category of the text content.

[0090] Among them, step 2013 specifically includes:

[0091] Input the classification prediction results into the first sequence.

[0092] Initialize a second sequence, and initialize the first element in the second sequence to the first element in the first sequence.

[0093] Traverse the elements except the first element in the first sequence, and for each element in the first sequence that is traversed: if the current traversed element is the same as the previous element, add the current traversed element to the second sequence; if the current traversed element is smaller than the previous element, add the previous element to the second sequence; if the current traversed element is larger than the previous element, determine whether the current traversed element is equal to the next element; if so, add the current traversed element to the second sequence; otherwise, add the previous element to the second sequence; after completing the traversal of the first sequence, obtain a corrected second sequence.

[0094] The above is a local correction. Next, the sequence after local correction is globally corrected to ensure that the "steps" in the sequence do not violate monotonicity.

[0095] A ladder list is generated according to the corrected second sequence, wherein the element values ​​in each ladder column of the ladder list are the same.

[0096] Calculate the average probability for each step.

[0097] Traverse the steps except the first step in the step list, and for each step traversed: if the element value of the previous step is greater than the element value of the next step, and the average probability of the previous step is greater than or equal to the average probability of the next step, then correct the values ​​of all elements of the next step to the element value of the previous step; if the average probability of the previous step is less than the average probability of the next step, then correct the values ​​of all elements of the previous step to the element value of the next step. After the traversal of the step list is completed, a corrected step list is obtained.

[0098] According to the corrected ladder list, the category of the text content is obtained.

[0099] The average accuracy of the corrected results (ladder list) reached 97%, and the text categories were greatly improved.

[0100] Step 201 is the text preprocessing process, such as Figure 4 As shown, the text in Base64 encoding and Quoted-printable encoding is obtained, that is, the encoding method in step 202 includes Base64 encoding and Quoted-printable encoding.

[0101] According to its encoding method, Python's third-party library (base64, quopri) is used for decoding. The decoding is divided into two stages, such as Figure 5 shown.

[0102] Character set detection is performed based on byte string features. This application uses the encoding detection tool chardet developed by Mozilla to perform character set detection on each byte text, returning {'encoding':'utf-8','confidence':0.99,'language':"}, where encoding is the character encoding method, confidence is the confidence of the detection result, and language is the language.

[0103] If the text fragment may be incomplete, character set detection cannot be performed on the entire text, and the detection result may not be accurate, a segmented detection method is adopted for detection.

[0104] Among them, step 203 specifically includes: starting from the first byte of the decoded string, intercepting each byte text of the set byte length in turn for character set detection. For each byte text, if the confidence in the detection result is greater than or equal to 0.9, the detection result is added to the character set list; otherwise, the byte text is shifted back by 1 byte and the text of the set byte length is intercepted to continue character set detection until the character set detection of the last byte text intercepted is completed.

[0105] In an exemplary embodiment, the byte length is set to 24, and the character detection process specifically includes: performing a first-stage decoding on each 24-byte text, and the decoded result (detection result) is a byte string (for example, {'encoding':'utf-8', 'confidence':0.99, 'language':”}); if the confidence in the detection result is greater than or equal to 0.90, the result is added to the character set list, and then the next 24 bytes of text are detected. If the detection fails (for example, because of a decoding error or the detection result does not meet the confidence threshold), the text content is shifted backward by 1 byte and the next loop iteration is continued.

[0106] Wherein, step 204 specifically includes:

[0107] The character sets in the character set list are counted, and the character sets are sorted in descending order of the counted number to obtain a character set sorted list.

[0108] like Figure 6 As shown, each character set in the character set sorting list is traversed, and the decoded string is converted into a current decoding result using the currently traversed character set. The current decoding result is subjected to a semantic coherence judgment. If the current decoding result passes the semantic coherence judgment, the current decoding result is used as the original text of the email and the traversal of the character set sorting list is stopped. If the semantic coherence judgment is not passed, the next character set is traversed.

[0109] If the traversal of each character set in the character set sorting list is completed and the decoding result obtained by character conversion of the decoded string using the last character set still fails to pass the semantic coherence judgment, a reminder that the interpretation is unsuccessful is made and a log record is made.

[0110] In an exemplary embodiment, a machine learning method is used to perform binary classification on garbled and non-garbled data (whether the semantics are coherent or not), that is, to achieve binary classification of whether the semantic coherence judgment is passed or not.

[0111] The semantic coherence of the current decoding result is judged, specifically including:

[0112] Extract the features of the current decoding result.

[0113] The extracted feature parameters are input into a semantic coherence judgment model to output a semantic judgment result; the semantic judgment result is semantic coherence or semantic incoherence; the semantic coherence judgment model is obtained by training a machine learning model using a training set.

[0114] The samples in the training set include input data and label data. The input data is the 9 features of the sample text, and the label data is semantically coherent or semantically incoherent.

[0115] The machine learning model selected random forest and trained nine features, including perplexity, Flesch Reading Ease, Flesch-Kincaid Grade Level, fog index, SMOG index, Coleman-Liau index, automated readability index, LIX, and Dale-Chall readability score. The final model achieved a classification accuracy of 99.08% on the test set.

[0116] The feature names are explained as follows:

[0117] Perplexity is a metric used in natural language processing to measure the quality of a language model's predictions. The more accurate a model's predictions are, the lower its perplexity is, meaning it's more likely to be a sentence.

[0118]

[0119] Among them, perplexity(S) is the perplexity of text S, P() represents the probability function, Wi represents the i-th character or word, n represents the total number of characters and words, and P(Wi) represents the probability of the i-th character appearing.

[0120] Flesch Reading Ease: This metric assigns a score, with lower scores indicating more difficult text. Generally, a score between 60 and 70 is suitable for secondary school students, while a score between 0 and 30 is suitable for university graduates.

[0121] Flesch-Kincaid Grade Level: This metric indicates the grade level a U.S. student would need to understand the text. For example, a text with a score of 9 would be appropriate for 9th graders.

[0122] Gunning Fog Index: This metric assesses the reading difficulty of a text. The higher the score, the harder it is to read. It is generally recommended to keep this index below 12.

[0123] SMOG Index: This metric estimates the grade level required to read a text. It is often used to assess the difficulty of popular books.

[0124] Coleman-Liau Index: This index calculates the reading difficulty of a text using a formula and expresses it at a grade level. It is calculated based on the number of sentences and words in the text.

[0125] Automated Readability Index: This metric gives a single value representing the difficulty of reading a text, which corresponds to grade level.

[0126] LIX: This is a Swedish reading difficulty indicator that is calculated based on sentence length and word length in the text.

[0127] Dale-Chall Readability Score: This metric is based on a simple vocabulary list of about 3,000 words. If more than 5% of the words in a text are not on this list, the text is considered difficult to read.

[0128] In an exemplary embodiment, a semantic coherence judgment is performed on the current decoding result, specifically including: using a pre-trained language model to calculate the perplexity of the current decoding result; if the perplexity is greater than a set judgment threshold, it is determined that the current decoding result has not passed the semantic coherence judgment; otherwise, it is determined that the current decoding result has passed the semantic coherence judgment.

[0129] The pre-trained language model uses a pre-trained model loaded from the Hugging Face Transformers library. The higher the perplexity, the worse the semantic coherence of the text. The judgment threshold is set to 1000. When the perplexity exceeds 1000, the decoded result is considered unreadable and fails the semantic coherence judgment.

[0130] After step 204, the method for parsing emails with missing content further includes: detecting the language of the original email text, and if the detected language is not the target language, translating the original email text into the target language, and writing the translation result into a document for storage.

[0131] More specifically, the Python langdetect module is used to perform language detection on the decoded text. If the language detected is Chinese, it is directly written to the document, and this phase ends. Otherwise, the corresponding transformer translation model is called based on the detected language to perform translation, and the translation result is written to the document for storage.

[0132] In an exemplary embodiment, Figure 7 As shown, a method for parsing emails with missing content is provided, including:

[0133] Extract text content from email text based on key fields.

[0134] Decode the extracted text content.

[0135] If the decoding is successful, the original email text is obtained and the program stops. If the decoded content is not empty (None), the decoding is successful, otherwise the decoding is unsuccessful.

[0136] If the decoding is unsuccessful, the email text is extracted and classified to obtain the categories of the text content; the categories include plain text format, web page format and attachment; the text content in the plain text format and web page format categories is used as the text to be decoded.

[0137] The text to be decoded is decoded according to the corresponding encoding method to obtain a decoded character string.

[0138] The decoded character string is segmented and detected according to the set byte length, and the character set corresponding to the character set detection result that meets the set conditions is added to the character set list.

[0139] The decoded character string is converted into characters by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment, thereby obtaining the original text of the email.

[0140] In an exemplary embodiment, a Python email library is used to extract text content from the email text based on key fields, and the extracted text content is decoded.

[0141] Figure 7 The model classification method is specifically Figure 2 This provides a method for parsing emails with missing content. The model classification method primarily uses neural networks to extract content, and uses related tools for character encoding recognition and decoding, language detection, and offline translation to obtain the original email text in the target language.

[0142] This application method has the following advantages when processing missing email text:

[0143] 1. This application enables matching and classification extraction of missing email content. Using regular expressions, text content is categorized and extracted, filtering out "noise" information such as headers and boundary fields. Text is classified using a Convolutional Neural Network (CNN) + LSTM neural network. A "ladder" correction algorithm is proposed based on the sequential structure of email text. The corrected accuracy increases by 9% compared to the average, reaching 97%.

[0144] 2. When Base64 and Quoted-Printable encoding are missing, existing decoding tools cannot decode them properly and usually report an invalid encoding error. This application adopts segmented decoding and skipping of missing bytes to maximize the decoding and translation of missing content.

[0145] 3. Considering that the existing character set detection tools are not accurate in detecting short texts and texts with missing content, a segmented detection method is adopted, and the detection results are filtered by conditions and sorted by number of times, which greatly improves the accuracy of character set detection.

[0146] 4. Add a decoding result judgment function, which uses the output of the semantic coherence judgment model or the semantic coherence of the text to determine whether the decoded text is readable.

[0147] 5. This application can intelligently and automatically extract and parse batch email content, making up for the shortcomings of existing online email parsing tools or decoding tools that require manual selection of parsed email content, encoding method, and character set type, and is highly efficient.

[0148] 6. Added offline translation library function, making the decoded translated text intuitive and readable. The offline translation library function includes 5 language translation functions: English-Chinese, Russian-Chinese, Japanese-Chinese, Korean-Chinese, Vietnamese-Chinese, etc.

[0149] Based on the same inventive concept, the embodiments of the present application also provide a device for parsing emails with missing content, which is used to implement the aforementioned method for parsing emails with missing content. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the device for parsing emails with missing content provided below can be found in the limitations of the method for parsing emails with missing content above, and will not be repeated here.

[0150] In an exemplary embodiment, a device for parsing emails with missing content is provided, including:

[0151] The text content extraction and classification module is used to extract and classify the text content of the email text to obtain the category of the text content; the categories include plain text format, web page format and attachment; and the text content classified as the plain text format and the web page format is used as the text to be decoded.

[0152] The first decoding module is used to decode the text to be decoded according to the corresponding encoding method to obtain a decoded character string.

[0153] The character set detection module is used to perform segmented character set detection on the decoded character string according to a set byte length, and add the character set corresponding to the character set detection result that meets the set conditions to the character set list.

[0154] The second decoding module is configured to perform character conversion on the decoded string by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment to obtain the original text of the email.

[0155] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store email parsing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for parsing emails with missing content is implemented.

[0156] Those skilled in the art will understand that Figure 8 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application and does not constitute a limitation on the computer device to which the solution of the present application is applied. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above-mentioned method embodiments when executing the computer program.

[0157] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0158] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0160] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0161] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, data processing logic of programmable logic devices, and the like.

[0162] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.< / pad>

Claims

1. A method for parsing emails with missing content, characterized in that: The email parsing method for missing content includes: Extracting and classifying the text content of the email text to obtain categories of the text content; the categories include plain text format, web page format and attachment; and using the text content in the plain text format and the web page format as the text to be decoded; Decoding the text to be decoded according to the corresponding encoding method to obtain a decoded string; Performing segmented character set detection on the decoded string according to a set byte length, and adding the character set corresponding to the character set detection result that meets the set conditions to a character set list, specifically comprising: starting from the first byte of the decoded string, sequentially intercepting byte text of each set byte length for character set detection; for each byte text, if the confidence in the detection result is greater than or equal to 0.9, adding the detection result to the character set list; otherwise, shifting the byte text back by 1 byte and intercepting the text of the set byte length to continue character set detection until the character set detection of the last intercepted byte text is completed; The character set in the character set list is traversed to convert the decoded string into characters until the decoded result passes the semantic coherence judgment to obtain the original email text, specifically including: Counting the character sets in the character set list, and sorting the character sets in descending order of the counted number to obtain a character set sorting list; Each character set in the character set sorted list is traversed, and the decoded string is converted into characters using the currently traversed character set to obtain a current decoding result. The current decoding result is subjected to a semantic coherence judgment. If the current decoding result passes the semantic coherence judgment, the current decoding result is used as the original email text and traversing the character set sorted list is stopped. If the current decoding result does not pass the semantic coherence judgment, the next character set is traversed.

2. The method for parsing emails with missing content according to claim 1, wherein: Extract and classify the email content to obtain the category of the text content, including: Based on the encoding features, the regular matching method is used to extract the text content of the email text. If there is Base64-encoded text in the extracted text content, the Base64-encoded text is vectorized to obtain the text to be classified; Inputting the file to be classified into a classification model to obtain a classification prediction result of the text content; the classification model is a trained neural network; Based on the inherent order of the categories in the email, the classification prediction result is corrected to obtain the category of the text content.

3. The method for parsing emails with missing content according to claim 2, wherein: Based on the inherent order of the categories in the email, the classification prediction result is corrected to obtain the category of the text content, specifically including: Input the classification prediction results into the first sequence; Initialize a second sequence, and initialize the first element in the second sequence to the first element in the first sequence; Traverse the elements of the first sequence except the first element, and for each element of the first sequence that is traversed: if the current traversed element is the same as the previous element, add the current traversed element to the second sequence; if the current traversed element is smaller than the previous element, add the previous element to the second sequence; if the current traversed element is larger than the previous element, determine whether the current traversed element is equal to the next element; if so, add the current traversed element to the second sequence; otherwise, add the previous element to the second sequence; after completing the traversal of the first sequence, obtain a corrected second sequence; generating a ladder list according to the corrected second sequence, wherein the element values ​​in each ladder column of the ladder list are the same; Calculate the average probability for each step; Traversing the ladder list except the first ladder, for each ladder traversed: if the element value of the previous ladder is greater than the element value of the next ladder, and the average probability of the previous ladder is greater than or equal to the average probability of the next ladder, then correcting the values ​​of all elements of the next ladder to the element value of the previous ladder; if the average probability of the previous ladder is less than the average probability of the next ladder, then correcting the values ​​of all elements of the previous ladder to the element value of the next ladder, after the ladder list traversal is completed, obtaining a corrected ladder list; According to the corrected ladder list, the category of the text content is obtained.

4. The method for parsing emails with missing content according to claim 2, wherein: The neural network includes an embedding layer, a first convolution module, a second convolution module, a long short-term memory network and a fully connected layer connected in sequence; the first convolution module and the second convolution module both include a one-dimensional convolution layer, a ReLU function, a maximum pooling layer and a Dropout layer connected in sequence.

5. The method for parsing emails with missing content according to claim 1, wherein: Perform semantic coherence judgment on the current decoding result, including: Extract the features of the current decoding result; The extracted feature parameters are input into a semantic coherence judgment model to output a semantic judgment result; the semantic judgment result is semantic coherence or semantic incoherence; the semantic coherence judgment model is obtained by training a machine learning model using a training set.

6. The method for parsing emails with missing content according to claim 1, wherein: After the decoded string is converted by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment and the original email text is obtained, the email parsing method for missing content further includes: The language type of the original email text is detected, and if the detected language type is not the target language, the original email text is translated into the target language.

7. A method for parsing emails with missing content, characterized in that: The email parsing method for missing content includes: Extract text content from email text based on key fields; Decode the extracted text content; If the decoding is successful, the original email text is obtained and the program stops; If the decoding is unsuccessful, the text content of the email is extracted and classified to obtain the categories of the text content; the categories include plain text format, web page format and attachment; the text content in the plain text format and web page format is used as the text to be decoded; Decoding the text to be decoded according to the corresponding encoding method to obtain a decoded string; Performing segmented character set detection on the decoded string according to a set byte length, and adding the character set corresponding to the character set detection result that meets the set conditions to a character set list, specifically comprising: starting from the first byte of the decoded string, sequentially intercepting byte text of each set byte length for character set detection; for each byte text, if the confidence in the detection result is greater than or equal to 0.9, adding the detection result to the character set list; otherwise, shifting the byte text back by 1 byte and intercepting the text of the set byte length to continue character set detection until the character set detection of the last intercepted byte text is completed; The character set in the character set list is traversed to convert the decoded string into characters until the decoded result passes the semantic coherence judgment to obtain the original email text, specifically including: Counting the character sets in the character set list, and sorting the character sets in descending order of the counted number to obtain a character set sorting list; Each character set in the character set sorted list is traversed, and the decoded string is converted into characters using the currently traversed character set to obtain a current decoding result. The current decoding result is subjected to a semantic coherence judgment. If the current decoding result passes the semantic coherence judgment, the current decoding result is used as the original email text and traversing the character set sorted list is stopped. If the current decoding result does not pass the semantic coherence judgment, the next character set is traversed.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the email parsing method for missing content according to any one of claims 1 to 6.

Images