Email classification method and system based on multimodal large language model

Abstract

The present invention relates to a mail classification method and system based on a multimodal large language model, wherein the mail classification method comprises: preprocessing mail data to be classified, obtaining the text content of the multimodal data of the mail to be classified, randomly sampling mails with marked mail categories from preset mail categories as samples, analyzing the sample data through the multimodal large language model to obtain the category feature text of the marked mail category, and then inputting the obtained prompt words into the multimodal large language model through a constructed prompt word template according to the text content of the multimodal data of the mail to be classified and the category feature text of the marked mail category, thereby determining the mail category of the mail to be classified. The present invention can detect spam mails and classify mails according to different industry categories or subject categories, which can meet the needs of mail data management, improve the accuracy of mail classification, and reduce the cost required for mail classification.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a mail classification method and system based on a multimodal large language model. Background Art

[0002] With the rapid development of the Internet, email, as a convenient, fast and cheap means of communication, plays an important role in people's daily work and study. However, while email brings convenience to people, it also provides opportunities for spammers. People have to deal with a large number of emails every day. Therefore, automatic classification of emails is of great significance to improve work efficiency and reduce work burden.

[0003] Email classification is a task that determines the category of an email based on its subject, content, sender / receiver, and attachments. Traditional email classification methods usually use the text in the email as the basis for classification. In this context, email classification can be considered a text classification task. General text classification methods match text information by using regular expression rules, but the text features of emails are very complex. Regular expression rules often cannot fully cover all information and lack the ability to extract features of the key content of email texts. This method has great problems in terms of accuracy and generalization.

[0004] Large Language Model (LLM) is a language model based on deep learning technology. It can capture the semantics, grammar and contextual relationships in natural language by learning a large amount of text data. Common large language models include MiniCPM-V 2.6, GPT4-V, Google Gemini, LlaVa, etc. With the excellent performance of generative large language models in various tasks, people have also begun to study the use of deep learning to extract features from input text through natural language processing models, and then train based on labeled data, so that the model can obtain general email classification capabilities.

[0005] The existing email classification methods have the following problems:

[0006] 1. Only focusing on the binary classification problem of spam detection of emails ignores the data governance needs of emails as a kind of data. That is, while identifying spam, we should consider classifying emails according to other needs, such as classifying emails according to the categories listed in "GB / T 4754-2017 National Economic Industry Classification Standard" and "GB / T 13745-2009 Subject Classification and Code".

[0007] 2. Only single-modal data such as the email subject, content, sender and recipient are used, while the additional information brought by multi-modal data such as charts and attachments in the email is ignored.

[0008] 3. It is highly dependent on the quality of data annotation, and the quality assurance cost of data annotation is high.

[0009] In view of this, the present invention is proposed. Summary of the invention

[0010] The purpose of the present invention is to propose an email classification method and system based on a multimodal large language model, which can detect spam in emails and classify emails according to different industry categories or subject categories, meet the needs of email data management, improve the accuracy of email classification, and reduce the cost required for email classification.

[0011] In a first aspect, the present invention proposes an email classification method based on a multimodal large language model, comprising:

[0012] S1. Preprocess the mail Y to be classified to obtain the text content (yt, yi, ytb, ya) of the multimodal data of the mail Y to be classified, wherein yt is the original text content of the mail Y to be classified, yi is the text content of the image of the mail Y to be classified, ytb is the text content of the table of the mail Y to be classified, and ya is the summary text content of the text content of the mail Y to be classified;

[0013] S2. Preset email categories C={C1, C2}, C2={C21, C22, ..., C2i, ..., C2n}, where C1 is spam and C2 is normal email. The email categories C1 and C2 contain a small number of emails with categorized labels, i=1, 2, ..., n, where n is the total number of categories;

[0014] S3. The email data set with the classification labels is preset as M1, the email category of the email data set M1 is Md1={C_1,C_2}, the predicted email category list of the p email classification results is C_m1, and the email category description list of the p email classification results is Em1, where m=1, 2, ..., p;

[0015] S4. Clear the mail category description list Em1;

[0016] S5. Randomly sample k emails from each category C_r of the email category Md1={C_1,C_2} to obtain data M_k1, where r=1,2, write the data M_k1 into the constructed prompt word template Tm1 to obtain prompt word tm1, input the prompt word tm1 into the multimodal large language model A, obtain the category feature text emr of each category C_r of the email category Md1={C_1,C_2}, append the category feature text emr to the email description list Em1, and then obtain the category feature text list Em1={em1,em2} of each category C_r of the email category Md1={C_1,C_2};

[0017] S6. Write the category feature text list Em1={em1, em2} and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y into the constructed prompt word template Tm2 to obtain the prompt word tm2, input the prompt word tm2 into the multimodal large language model A, obtain the mail category C_r1 of the mail to be classified Y, and append the mail category C_r1 to the predicted mail category list C_m1;

[0018] S7. Execute steps S4-S6 p times to obtain a list of p predicted mail categories C_m1={C_1, C_2, ..., C_p} of the mail to be classified Y;

[0019] S8. Select the most appropriate category from the predicted email category list C_m1 as the final email category of the email to be classified Y, and save the final email category of the email to be classified Y and the text content (yt, yi, ytb, ya) of the multimodal data of the email to be classified Y into the email data set M1 with the classification label, for random sampling in subsequent email classification;

[0020] S9. If the final mail category of the mail to be classified Y belongs to C1, the mail classification task for the mail to be classified Y is stopped;

[0021] S10. If the final mail category of the mail to be classified Y belongs to C2, continue to classify the mail to be classified Y to obtain q classification results, and select the most suitable category from the q classification results as the final mail category of the mail to be classified Y.

[0022] Furthermore, the mail Y to be classified in S1 is preprocessed, including:

[0023] S11. Write the image content i and the table content tb of the mail to be classified Y into the constructed prompt word template T3 to obtain the prompt word TC3, input the prompt word TC3 into the multimodal large language model A, and obtain the text content yi of the image of the mail to be classified Y and the text content ytb of the table;

[0024] S12. Write the original text content yt of the mail to be classified, the text content yi of the image, and the text content ytb of the table into the constructed prompt word template T4 to obtain prompt words TC4, and input the prompt words TC4 into the multimodal large language model A to obtain a summary text ya of the text content of the mail to be classified Y;

[0025] S13. Save the text content (yt, yi, ytb, ya) of the preprocessed multimodal data of the mail to be classified Y into a mail database E, wherein the mail database E includes mails marked with classification labels and unclassified mails.

[0026] Furthermore, the prompt word template T3 constructed in S11 is as follows:

[0027] 1) The existing email Y to be classified originally has text content yt, image content i, and table content tb;

[0028] 2) Please analyze and describe this image in detail, giving a detailed introduction and summary;

[0029] 3) Please analyze and describe this table in detail, giving a detailed introduction and summary.

[0030] Furthermore, the prompt word template T4 constructed in S12 is as follows:

[0031] 1) The original text content yt of the existing email Y to be classified, the textual content yi of the image, and the textual content ytb of the table;

[0032] 2) Please summarize the main content of this email.

[0033] Furthermore, the multimodal large language model A described in S11 and S12 is the same, wherein the multimodal large language model A at least includes: MiniCPM-V 2.6, GPT4-V, Gemini, and LlaVa.

[0034] Furthermore, if Figure 2 As shown, the mail category set C2 described in S2 = {C21, C22, ..., C2i, ..., C2n}, C2i is the category name listed in the "GB / T 4754-2017 National Economic Industry Classification Standard", n is the total number of categories A to T, n=20.

[0035] Furthermore, if Figure 3 As shown, the mail category set C2 described in S2 = {C21, C22, ..., C2i, ..., C2n}, C2i is the subject name listed in "GB / T 13745-2009 Subject Classification and Code", and n = 58.

[0036] Furthermore, the prompt word template Tm1 constructed in S5 is:

[0037] 1) There are k emails belonging to category C_r [email r1, email r2, ..., email rk], where r = 1, 2;

[0038] 2) Based on the rule that k emails belong to category C2_r, please summarize some characteristics of this category.

[0039] Furthermore, the prompt word template Tm2 constructed as shown in S6 is:

[0040] 1) You are an excellent email classifier. There are r categories and their corresponding descriptions:

[0041] Category C_r: emr;

[0042] 2) Based on your rich experience, please analyze which category of the email category Md1={C_1,C_2} the text content (yt, yi, ytb, ya) of the multimodal data of the email to be classified Y belongs to.

[0043] Furthermore, the q-times email classification results of the email to be classified Y are obtained in S10, including:

[0044] S101. The email data set with the classification labels is preset as M2, the email categories of the email data set M2 are Md2={C2_1, C2_2, ..., C2_w, ..., C2_n}, the predicted email category list of q email classification results is C_m2, and the email category description list of q email classification results is Em2, where m=1, 2, ..., q;

[0045] S102. Clear the mail category description list Em2;

[0046] S103. Randomly sample k emails from each category C2_w of the email category Md2={C2_1, C2_2, ..., C2_w, ..., C2_n} to obtain data M_k2, write the data M_k2 into the constructed prompt word template Tm3 to obtain prompt word tm3, input the prompt word tm3 into the multimodal large language model A, and obtain the email category Md2={C2_1, C2_2, ..., C2_w, ..., C2_n}, the category feature text em_w of each category C2_w, the category feature text em_w is appended to the email description list Em2, and then the category feature text list Em2 of each category C2_w of the email category Md2={C2_1, C2_2, ..., C2_w, ..., C2_n} is obtained, wherein the prompt word template Tm3 is:

[0047] 1) There are k emails belonging to category C2_w [email w1, email w2, ..., email wk], where w = 1, 2, ..., n;

[0048] 2) Please summarize some characteristics of the category based on the rule that k emails belong to category C2_w;

[0049] S104. Write the category feature text list Em2={em_1, em_2, ..., em_w, ..., em_n} and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y into the constructed prompt word template Tm4 to obtain the prompt word tm4, input the prompt word tm4 into the multimodal large language model A, obtain the mail category C2_w1 of the mail to be classified Y, append the mail category C2_w1 to the predicted mail category list C_m2, wherein the prompt word template Tm4 is:

[0050] 1) You are an excellent email classifier. There are r categories and their corresponding descriptions:

[0051] Category C2_w:em_w;

[0052] 2) Based on your rich experience, please analyze which category of the email category Md2={C2_1,C2_2,…,C2_w,…,C2_n} the text content (yt, yi, ytb, ya) of the multimodal data of the email to be classified belongs to;

[0053] S105. Execute steps S102-S104 q times to obtain a list of q predicted mail categories of mail Y to be classified C_m2={C2_1, C2_2, ..., C_2q};

[0054] S106. Select the most appropriate category from the predicted email category list C_m2 as the final email category of the email to be classified Y, and save the final email category of the email to be classified Y and the text content (yt, yi, ytb, ya) of the multimodal data of the email to be classified Y to the email data set M2 marked with classification labels for random sampling in subsequent email classification.

[0055] Furthermore, in S8 and S10, the mail classification that appears most frequently in the classification results is selected as the classification standard for determining the final mail classification of the mail to be classified that best meets.

[0056] In a second aspect, the present invention proposes an email classification system based on a multimodal large language model, which adopts the email classification method based on a multimodal large language model as described in the first aspect, comprising:

[0057] The mail data preprocessing module D41 preprocesses the mail Y to be classified to obtain the text content (yt, yi, ytb, ya) of the multimodal data of the mail Y to be classified;

[0058] The mail category acquisition module D42 acquires mail categories C={C1, C2}, C2={C21, C22, ..., C2i, ..., C2n};

[0059] The email category feature text acquisition module D43 is pre-set as M1 an email data set with marked classification labels, the email category Md1={C_1, C_2} of the email data set M1, the predicted email category list of p email classification results is C_m1, the email category description list of p email classification results is Em1, where m=1, 2, ..., p, the email category description list Em1 is cleared, and k emails are randomly sampled from each category C_r of the email category Md1={C_1, C_2} to obtain data M_ k1, where r=1, 2, write the data M_k1 into the constructed prompt word template Tm1 to obtain the prompt word tm1, input the prompt word tm1 into the multimodal large language model A, obtain the category feature text emr of each category C_r of the email category Md1={C_1, C_2}, append the category feature text emr to the email description list Em1, and then obtain the category feature text list Em1={em1, em2} of each category C_r of the email category Md1={C_1, C_2};

[0060] The mail category classification result acquisition module D44 writes the category feature text list Em1={em1, em2} and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y into the constructed prompt word template Tm2 to obtain the prompt word tm2, inputs the prompt word tm2 into the multimodal large language model A, obtains the mail category C_r1 of the mail to be classified Y, appends the mail category C_r1 to the predicted mail category list C_m1, and executes the mail category feature text acquisition p times. The mail category classification result is obtained, and a list C_m1={C_1, C_2, ..., C_p} of p-times predicted mail categories of the mail to be classified is obtained. The most suitable category is selected from the mail category list C_m1 as the final mail category of the mail to be classified Y, and the final mail category of the mail to be classified Y and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y are saved to the mail data set M1 with the classification label, for random sampling in the subsequent mail classification;

[0061] The final mail category determination module D45 stops the mail classification task for the mail Y to be classified if the final mail category of the mail Y to be classified belongs to C1; if the final mail category of the mail Y to be classified belongs to C2, continues to classify the mail Y to be classified until the most appropriate category is selected as the final mail category of the mail Y to be classified.

[0062] In a third aspect, the present invention proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the email classification method based on the multimodal large language model as described above are implemented.

[0063] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the email classification method based on a multimodal large language model as described above.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. The present invention adopts a two-stage email classification model, which not only focuses on the detection and classification of spam emails but also focuses on the classification of normal emails. It can summarize and understand the characteristics of industry categories or subject categories through a large language model, and classify emails by industry categories or subject categories, which can more effectively protect the data governance process.

[0066] 2. The present invention uses a multi-modal large language model (MLLM), which retains the powerful context understanding ability of the large language model while introducing the ability to recognize, analyze and understand multiple modal data (such as charts, attachments, etc. in emails). It uses a multi-modal large language model to perform semantic analysis on all images, tables and other contents in the email to generate natural language text output, and then combines the text information of the email itself with the large language model for summary and abstraction, so as to improve the accuracy of email classification.

[0067] 3. The present invention analyzes and summarizes a small number of labeled emails through random sampling to obtain the text description features of the category, and adds the text description features of each category when constructing the prompt words, so that the large language model can better understand the email itself and the specific features of the various categories to be classified when making judgments, thereby obtaining more accurate classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0069] Figure 1 It is a schematic diagram of the email classification method based on the multimodal large language model of the present invention.

[0070] Figure 2 It is a schematic diagram of the category names listed in the "GB / T 4754-2017 National Economic Industry Classification Standard".

[0071] Figure 3 This is a schematic diagram of the subject names listed in "GB / T 13745-2009 Subject Classification and Code".

[0072] Figure 4 It is a structural schematic diagram of the mail classification system based on the multimodal large language model of the present invention.

[0073] Explanation of reference numerals: D41 module for preprocessing mail data to be classified; D42 module for obtaining mail categories; D43 module for obtaining mail category characteristic texts; D44 module for obtaining mail category classification results; D45 module for determining mail category classification results. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0076] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0077] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0078] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0079] The optional embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0080] First, as Figure 1 As shown, the present invention proposes an email classification method based on a multimodal large language model, comprising:

[0081] S1. Preprocess the mail Y to be classified to obtain the text content (yt, yi, ytb, ya) of the multimodal data of the mail Y to be classified, wherein yt is the original text content of the mail Y to be classified, yi is the text content of the image of the mail Y to be classified, ytb is the text content of the table of the mail Y to be classified, and ya is the summary text content of the text content of the mail Y to be classified;

[0082] S2. Preset email categories C={C1, C2}, C2={C21, C22, ..., C2i, ..., C2n}, where C1 is spam and C2 is normal email. The email categories C1 and C2 contain a small number of emails with categorized labels, i=1, 2, ..., n, where n is the total number of categories;

[0083] S3. The email data set with the classification labels is preset as M1, the email category of the email data set M1 is Md1={C_1,C_2}, the predicted email category list of the p email classification results is C_m1, and the email category description list of the p email classification results is Em1, where m=1, 2, ..., p;

[0084] S4. Clear the mail category description list Em1;

[0085] S5. Randomly sample k emails from each category C_r of the email category Md1={C_1,C_2} to obtain data M_k1, where r=1,2, write the data M_k1 into the constructed prompt word template Tm1 to obtain prompt word tm1, input the prompt word tm1 into the multimodal large language model A, obtain the category feature text emr of each category C_r of the email category Md1={C_1,C_2}, append the category feature text emr to the email description list Em1, and then obtain the category feature text list Em1={em1,em2} of each category C_r of the email category Md1={C_1,C_2};

[0086] S6. Write the category feature text list Em1={em1, em2} and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y into the constructed prompt word template Tm2 to obtain the prompt word tm2, input the prompt word tm2 into the multimodal large language model A, obtain the mail category C_r1 of the mail to be classified Y, and append the mail category C_r1 to the predicted mail category list C_m1;

[0087] S7. Execute steps S4-S6 p times to obtain a list of p predicted mail categories C_m1={C_1, C_2, ..., C_p} of the mail to be classified Y;

[0088] S8. Select the most appropriate category from the predicted email category list C_m1 as the final email category of the email to be classified Y, and save the final email category of the email to be classified Y and the text content (yt, yi, ytb, ya) of the multimodal data of the email to be classified Y into the email data set M1 with the classification label, for random sampling in subsequent email classification;

[0089] S9. If the final mail category of the mail to be classified Y belongs to C1, the mail classification task for the mail to be classified Y is stopped;

[0090] S10. If the final mail category of the mail to be classified Y belongs to C2, continue to classify the mail to be classified Y to obtain q classification results, and select the most suitable category from the q classification results as the final mail category of the mail to be classified Y.

[0091] Among them, the parameter k in S5 is a hyperparameter greater than or equal to 2, and the total number of categories of the email category Md1={C_1,C_2} is 2, so executing step S5 twice can obtain the category feature text list Em1={em1,em2} of each category C_r of the email category Md1={C_1,C_2}.

[0092] The mail category C_r1 of the mail to be classified Y obtained in S6 belongs to one of the mail categories Md1={C_1, C_2}.

[0093] The parameter p in S7 is also a hyperparameter. Generally, the larger the p is, the more accurate the final email category determination of the email Y to be classified will be.

[0094] In the context of machine learning, hyperparameters are parameters whose values ​​are set before starting the learning process, rather than parameter data obtained through training. Typically, hyperparameters need to be optimized to select a set of optimal hyperparameters to improve learning performance and results.

[0095] Furthermore, the mail Y to be classified in S1 is preprocessed, including:

[0096] S11. Write the image content i and the table content tb of the mail to be classified Y into the constructed prompt word template T3 to obtain the prompt word TC3, input the prompt word TC3 into the multimodal large language model A, and obtain the text content yi of the image of the mail to be classified Y and the text content ytb of the table;

[0097] S12. Write the original text content yt of the mail to be classified, the text content yi of the image, and the text content ytb of the table into the constructed prompt word template T4 to obtain prompt words TC4, and input the prompt words TC4 into the multimodal large language model A to obtain a summary text ya of the text content of the mail to be classified Y;

[0098] S13. Save the text content (yt, yi, ytb, ya) of the preprocessed multimodal data of the mail to be classified Y into a mail database E, wherein the mail database E includes mails marked with classification labels and unclassified mails.

[0099] Among them, the steps mentioned in S11-S13 of obtaining prompt words from the multimodal data text content through the prompt word template and inputting the prompt words into the multimodal large language model A to obtain the text content can all be obtained through existing technologies and will not be described here.

[0100] Furthermore, the prompt word template T3 constructed in S11 is as follows:

[0101] 1) The existing email Y to be classified originally has text content yt, image content i, and table content tb;

[0102] 2) Please analyze and describe this image in detail, giving a detailed introduction and summary;

[0103] 3) Please analyze and describe this table in detail, giving a detailed introduction and summary.

[0104] Furthermore, the prompt word template T4 constructed in S12 is as follows:

[0105] 1) The original text content yt of the existing email Y to be classified, the textual content yi of the image, and the textual content ytb of the table;

[0106] 2) Please summarize the main content of this email.

[0107] Furthermore, the multimodal large language model A described in S11 and S12 is the same, wherein the multimodal large language model A at least includes: MiniCPM-V 2.6, GPT4-V, Gemini, and LlaVa.

[0108] Furthermore, if Figure 2 As shown, the mail category set C2 in S2 = {C21, C22, ..., C2i, ..., C2n}, C2i is the category name listed in the "GB / T 4754-2017 National Economic Industry Classification Standard", n is the total number of categories A to T, n = 20;

[0109] Furthermore, if Figure 3 As shown, the mail category set C2 described in S2 is C2={C21, C22, ..., C2i, ..., C2n}, C2i is the subject name listed in "GB / T 13745-2009 Subject Classification and Code", and n=58;

[0110] Among them, the mail category set described in S2 is not limited to the categories listed in the "GB / T 4754-2017 National Economic Industry Classification Standard" and the subject names listed in the "GB / T 13745-2009 Subject Classification and Code". As long as there are some classification standards that can distinguish mail categories, they can be used for mail classification tasks.

[0111] Furthermore, the prompt word template Tm1 constructed in S5 is:

[0112] 1) There are k emails belonging to category C_r [email r1, email r2, ..., email rk], where r = 1, 2;

[0113] 2) Based on the rule that k emails belong to category C2_r, please summarize some characteristics of this category.

[0114] Furthermore, the prompt word template Tm2 constructed as shown in S6 is:

[0115] 1) You are an excellent email classifier. There are r categories and their corresponding descriptions:

[0116] Category C_r: emr;

[0117] 2) Based on your rich experience, please analyze which category of the email category Md1={C_1,C_2} the text content (yt, yi, ytb, ya) of the multimodal data of the email to be classified Y belongs to.

[0118] Furthermore, the q-times email classification results of the email to be classified Y are obtained in S10, including:

[0119] S101. The email data set with the classification labels is preset as M2, the email categories of the email data set M2 are Md2={C2_1, C2_2, ..., C2_w, ..., C2_n}, the predicted email category list of q email classification results is C_m2, and the email category description list of q email classification results is Em2, where m=1, 2, ..., q;

[0120] S102. Clear the mail category description list Em2;

[0121] S103. Randomly sample k emails from each category C2_w of the email category Md2={C2_1, C2_2, ..., C2_w, ..., C2_n} to obtain data M_k2, write the data M_k2 into the constructed prompt word template Tm3 to obtain prompt word tm3, input the prompt word tm3 into the multimodal large language model A, and obtain the email category Md2={C2_1, C2_2, ..., C2_w, ..., C2_n}, the category feature text em_w of each category C2_w, the category feature text em_w is appended to the email description list Em2, and then the category feature text list Em2 of each category C2_w of the email category Md2={C2_1, C2_2, ..., C2_w, ..., C2_n} is obtained, wherein the prompt word template Tm3 is:

[0122] 1) There are k emails belonging to category C2_w [email w1, email w2, ..., email wk], where w = 1, 2, ..., n;

[0123] 2) Please summarize some characteristics of the category based on the rule that k emails belong to category C2_w;

[0124] S104. Write the category feature text list Em2={em_1, em_2, ..., em_w, ..., em_n} and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y into the constructed prompt word template Tm4 to obtain the prompt word tm4, input the prompt word tm4 into the multimodal large language model A, obtain the mail category C2_w1 of the mail to be classified Y, append the mail category C2_w1 to the predicted mail category list C_m2, wherein the prompt word template Tm4 is:

[0125] 1) You are an excellent email classifier. There are r categories and their corresponding descriptions:

[0126] Category C2_w:em_w;

[0127] 2) Based on your rich experience, please analyze which category of the email category Md2={C2_1,C2_2,…,C2_w,…,C2_n} the text content (yt, yi, ytb, ya) of the multimodal data of the email to be classified belongs to;

[0128] S105. Execute steps S102-S104 q times to obtain a list of q predicted mail categories of mail Y to be classified C_m2={C2_1, C2_2, ..., C_2q};

[0129] S106. Select the most appropriate category from the predicted email category list C_m2 as the final email category of the email to be classified Y, and save the final email category of the email to be classified Y and the text content (yt, yi, ytb, ya) of the multimodal data of the email to be classified Y to the email data set M2 marked with classification labels for random sampling in subsequent email classification.

[0130] Among them, the parameter k described in S103 is the same as the parameter described in S5, which is a hyperparameter greater than or equal to 2. The total number of categories of the email category Md2={C2_1, C2_2, ..., C2_w, ..., C2_n} is n, so by executing step S103 n times, the category feature text list Em2={em_1, em_2, ..., em_w, ..., em_n} of each category C2_w of the email category Md2={C2_1, C2_2, ..., C2_w, ..., C2_n} can be obtained.

[0131] The mail category C2_w1 of the mail to be classified Y obtained in S104 belongs to one of the mail categories Md2={C2_1, C2_2, ..., C2_w, ..., C2_n}.

[0132] The parameter q in S105 is also a hyperparameter. Generally, the larger the value of q is, the more accurate the final email category determination of the email Y to be classified is.

[0133] Furthermore, in S8 and S10, the mail classification that appears most frequently in the classification results is selected as the classification standard for determining the final mail classification of the mail to be classified that best meets.

[0134] In the present invention, the mail category classification process of the mail to be classified Y includes random sampling of mails with marked classification labels, so there are multiple different prediction results in the predicted mail category lists C_m1 and C_m2. Since the multimodal large language model has a strong context understanding ability, selecting the mail category with the most occurrences in the predicted mail category lists C_m1 and C_m2 as the final mail category of the mail to be classified Y is the classification result with the highest probability of being correct.

[0135] Compared with traditional email classification models, the use of a multimodal large language model proposed in the present invention can integrate information from multiple modalities and understand more complex semantic relationships and contextual content. It can take advantage of its versatility when faced with complex emails without the need for specific rules, and can easily adapt to emails from different industries, significantly improving the accuracy, efficiency, and adaptability of email classification.

[0136] In addition, the present invention also proposes to analyze and summarize the labeled emails through random sampling, obtain the text description features of the category, and add the text description features of each category when constructing the prompt words, so that the large language model can better understand the specific features of the email itself and the various categories to be classified when making judgments, thereby obtaining more accurate classification results.

[0137] Second, as Figure 4 As shown, the present invention proposes an email classification system based on a multimodal large language model, which adopts the email classification method based on a multimodal large language model as described in the first aspect, including:

[0138] The mail data preprocessing module D41 preprocesses the mail Y to be classified to obtain the text content (yt, yi, ytb, ya) of the multimodal data of the mail Y to be classified;

[0139] The mail category acquisition module D42 acquires mail categories C={C1, C2}, C2={C21, C22, ..., C2i, ..., C2n};

[0140] The email category feature text acquisition module D43 is pre-set as M1 an email data set with marked classification labels, the email category Md1={C_1, C_2} of the email data set M1, the predicted email category list of p email classification results is C_m1, the email category description list of p email classification results is Em1, where m=1, 2, ..., p, the email category description list Em1 is cleared, and k emails are randomly sampled from each category C_r of the email category Md1={C_1, C_2} to obtain data M_ k1, where r=1, 2, write the data M_k1 into the constructed prompt word template Tm1 to obtain the prompt word tm1, input the prompt word tm1 into the multimodal large language model A, obtain the category feature text emr of each category C_r of the email category Md1={C_1, C_2}, append the category feature text emr to the email description list Em1, and then obtain the category feature text list Em1={em1, em2} of each category C_r of the email category Md1={C_1, C_2};

[0141] The mail category classification result acquisition module D44 writes the category feature text list Em1={em1, em2} and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y into the constructed prompt word template Tm2 to obtain the prompt word tm2, inputs the prompt word tm2 into the multimodal large language model A, obtains the mail category C_r1 of the mail to be classified Y, appends the mail category C_r1 to the predicted mail category list C_m1, and executes the mail category feature text acquisition p times. The mail category classification result is obtained, and a list C_m1={C_1, C_2, ..., C_p} of p-times predicted mail categories of the mail to be classified is obtained. The most suitable category is selected from the mail category list C_m1 as the final mail category of the mail to be classified Y, and the final mail category of the mail to be classified Y and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y are saved to the mail data set M1 with the classification label, for random sampling in the subsequent mail classification;

[0142] The final mail category determination module D45 stops the mail classification task for the mail Y to be classified if the final mail category of the mail Y to be classified belongs to C1; if the final mail category of the mail Y to be classified belongs to C2, continues to classify the mail Y to be classified until the most appropriate category is selected as the final mail category of the mail Y to be classified.

[0143] In a third aspect, the present invention proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the email classification method based on the multimodal large language model as described above are implemented.

[0144] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the email classification method based on a multimodal large language model as described above.

[0145] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0146] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0147] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0148] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0149] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mail classification method based on a multimodal large language model, characterized in that: include: S1. Preprocess the mail Y to be classified to obtain the text content (yt, yi, ytb, ya) of the multimodal data of the mail Y to be classified, wherein yt is the original text content of the mail Y to be classified, yi is the text content of the image of the mail Y to be classified, ytb is the text content of the table of the mail Y to be classified, and ya is the summary text content of the text content of the mail Y to be classified, wherein the preprocessing of the mail Y to be classified includes: S11. Write the image content i and the table content tb of the mail to be classified Y into the constructed prompt word template T3 to obtain the prompt word TC3, input the prompt word TC3 into the multimodal large language model A, and obtain the textual content yi of the image of the mail to be classified Y and the textual content ytb of the table; S12. Write the original text content yt of the mail to be classified, the text content yi of the image, and the text content ytb of the table into the constructed prompt word template T4 to obtain prompt words TC4, and input the prompt words TC4 into the multimodal large language model A to obtain a summary text ya of the text content of the mail to be classified Y; S13. Save the text content (yt, yi, ytb, ya) of the preprocessed multimodal data of the mail to be classified Y into a mail database E, wherein the mail database E includes mails marked with classification labels and unclassified mails; S2. Preset mail category C = {C1, C2}, C2 = {C21, C22, ..., C2i, ..., C2n}, where C1 is spam and C2 is normal mail. The mail categories C1 and C2 contain a small number of mails with categorized labels, i = 1, 2, ..., n, where n is the total number of categories; S3. The email data set with the classification labels is preset as M1, the email category of the email data set M1 is Md1 = {C_1, C_2}, the predicted email category list of the p email classification results is C_m1, and the email category description list of the p email classification results is Em1, where m = 1, 2, ..., p; S4. Clear the mail category description list Em1; S5. Randomly sample k emails from each category C_r of the email category Md1={C_1,C_2} to obtain data M_k1, where r=1,2, write the data M_k1 into the constructed prompt word template Tm1 to obtain prompt word tm1, input the prompt word tm1 into the multimodal large language model A, obtain the category feature text emr of each category C_r of the email category Md1={C_1,C_2}, append the category feature text emr to the email category description list Em1, and then obtain the email category description list Em1={em1,em2} composed of the category feature text emr of each category C_r of the email category Md1={C_1,C_2}; S6. Write the mail category description list Em1 = ​​{em1, em2} and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y into the constructed prompt word template Tm2 to obtain the prompt word tm2, input the prompt word tm2 into the multimodal large language model A, obtain the mail category C_r1 of the mail to be classified Y, and append the mail category C_r1 to the predicted mail category list C_m1; S7. Execute steps S4-S6 p times to obtain a list of p predicted mail categories of the mail to be classified Y C_m1 = {C_1, C_2, ..., C_p}; S8. Select the most appropriate category from the predicted mail category list C_m1 as the final mail category of the mail to be classified Y, and save the final mail category of the mail to be classified Y and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y into the mail data set M1 with the marked classification label, for random sampling in subsequent mail classification; S9. If the final mail category of the mail to be classified Y belongs to C1, the mail classification task for the mail to be classified Y is stopped; S10. If the final mail category of the mail to be classified Y belongs to C2, continue to classify the mail to be classified Y to obtain q classification results, and select the most suitable category from the q classification results as the final mail category of the mail to be classified Y.

2. According to claim 1, a method for email classification based on a multimodal large language model is characterized in that: The multimodal large language model A described in S11 and S12 includes at least: MiniCPM-V 2.6, GPT4-V, Gemini, and LlaVa.

3. The method for email classification based on a multimodal large language model according to claim 1, characterized in that: In S11, the constructed prompt word template T3 is: 1) The existing text content yt, image content i, and table content tb of the existing email Y to be classified; 2) Please analyze and describe this image in detail, giving a detailed introduction and summary; 3) Please analyze and describe this table in detail, giving a detailed introduction and summary; In S12, the constructed prompt word template T4 is: 1) The existing text content yt of the mail Y to be classified, the textual content yi of the image, and the textual content ytb of the table; 2) Please summarize the main content of this email.

4. The method for email classification based on a multimodal large language model according to claim 1, characterized in that: The mail category set C2 described in S2 = {C21, C22, ..., C2i, ..., C2n}, where C2i is at least a category listed in the "GB / T 4754-2017 National Economic Industry Classification Standard" and a subject name listed in the "GB / T 13745-2009 Subject Classification and Code".

5. The method for email classification based on a multimodal large language model according to claim 1, characterized in that: The prompt word template Tm1 constructed in S5 is: 1) There are k emails belonging to category C_r [email r1, email r2, ..., email rk], where r = 1, 2; 2) Based on the rule that k emails belong to category C2_r, please summarize some characteristics of this category.

6. The method for email classification based on a multimodal large language model according to claim 1, characterized in that: The cue word template Tm2 constructed as shown in S6 is: 1) You are an excellent email classifier. There are r categories and their corresponding descriptions: Category C_r: emr; 2) Based on your rich experience, please analyze which category of the email category Md1 = {C_1, C_2} the text content (yt, yi, ytb, ya) of the multimodal data of the email to be classified Y belongs to.

7. The method for email classification based on a multimodal large language model according to claim 1, characterized in that: In S10, the q-times mail classification results of the mail to be classified Y are obtained, including: S101. The email data set with the classification labels is preset as M2, the email categories of the email data set M2 are Md2 = {C2_1, C2_2, ..., C2_w, ..., C2_n}, the predicted email category list of q email classification results is C_m2, and the email category description list of q email classification results is Em2, where m = 1, 2, ..., q; S102. Clear the mail category description list Em2; S103. Randomly sample k emails from each category C2_w of the email category Md2 = {C2_1, C2_2, ..., C2_w, ..., C2_n} to obtain data M_k2, write the data M_k2 into the constructed prompt word template Tm3 to obtain prompt word tm3, input the prompt word tm3 into the multimodal large language model A, and obtain the email category Md2 = {C2_1, C2_2, ..., C2_w, ..., C2_n} The category feature text em_w of each category C2_w is appended to the mail category description list Em2, thereby obtaining the mail category Md2 = {C2_1, C2_2, ..., C2_w, ..., C2_n}. The mail category list Em2 = {em_1, em_2, ..., em_w, ..., em_n} composed of the category feature text em_w of each category C2_w is obtained, wherein the prompt word template Tm3 is: 1) There are k emails belonging to category C2_w [email w1, email w2, ..., email wk], where w = 1, 2, ..., n; 2) Based on the rule that k emails belong to category C2_w, please summarize some characteristics of this category; S104. Write the mail category list Em2 = {em_1, em_2, ..., em_w, ..., em_n} and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y into the constructed prompt word template Tm4 to obtain the prompt word tm4, input the prompt word tm4 into the multimodal large language model A, obtain the mail category C2_w1 of the mail to be classified Y, append the mail category C2_w1 to the predicted mail category list C_m2, wherein the prompt word template Tm4 is: 1) You are an excellent email classifier. There are r categories and their corresponding descriptions: Category C2_w:em_w; 2) Based on your rich experience, please analyze which category the text content (yt, yi, ytb, ya) of the multimodal data of the email to be classified Y belongs to among the email categories Md2 = {C2_1, C2_2, ..., C2_w, ..., C2_n}; S105. Execute steps S102-S104 q times to obtain the q-times predicted mail category list C_m2 = {C2_1, C2_2, ..., C_2q} of the mail to be classified Y; S106. Select the most appropriate category from the predicted email category list C_m2 as the final email category of the email to be classified Y, and save the final email category of the email to be classified Y and the text content (yt, yi, ytb, ya) of the multimodal data of the email to be classified Y to the email data set M2 marked with classification labels for random sampling in subsequent email classification.

8. The method for email classification based on a multimodal large language model according to claim 1, characterized in that: In S8 and S10, the mail classification that appears most frequently in the classification results is selected as the classification standard for determining the final mail classification of the mail to be classified that best meets.

9. An email classification system based on a multimodal large language model, using the email classification method based on a multimodal large language model as claimed in any one of claims 1 to 8, comprising: The mail data preprocessing module to be classified preprocesses the mail Y to be classified to obtain the text content (yt, yi, ytb, ya) of the multimodal data of the mail Y to be classified; The mail category acquisition module acquires mail categories C = {C1, C2}, C2 = {C21, C22, ..., C2i, ..., C2n}; the mail category feature text acquisition module, presetting the mail data set with marked classification labels as M1, the mail category Md1 of the mail data set M1 = {C_1, C_2}, the predicted mail category list of p mail classification results as C_m1, the mail category description list of p mail classification results as Em1, where m = 1, 2, ..., p, clear the mail category description list Em1, and select each category C from the mail category Md1 = {C_1, C_2}. _r randomly samples k emails to obtain data M_k1, where r=1, 2, writes the data M_k1 into the constructed prompt word template Tm1 to obtain prompt word tm1, inputs the prompt word tm1 into the multimodal large language model A, obtains the category feature text emr of each category C_r of the email category Md1={C_1, C_2}, appends the category feature text emr to the email category description list Em1, and then obtains the email category description list Em1={em1, em2} composed of the category feature text emr of each category C_r of the email category Md1={C_1, C_2}; The mail category classification result acquisition module writes the mail category description list Em1={em1, em2} and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y into the constructed prompt word template Tm2 to obtain the prompt word tm2, inputs the prompt word tm2 into the multimodal large language model A, obtains the mail category C_r1 of the mail to be classified Y, appends the mail category C_r1 to the predicted mail category list C_m1, and performs p times of mail category feature text acquisition and The mail category classification result is obtained, and a list C_m1 = {C_1, C_2, ..., C_p} of p-times predicted mail categories of the mail to be classified is obtained. The most suitable category is selected from the mail category list C_m1 as the final mail category of the mail to be classified Y, and the final mail category of the mail to be classified Y and the text content (yt, yi, ytb, ya) of the multimodal data of the mail to be classified Y are saved in the mail data set M1 with the classification label, for random sampling in the subsequent mail classification; The final mail category determination module stops the mail classification task for the mail Y to be classified if the final mail category of the mail Y to be classified belongs to C1; if the final mail category of the mail Y to be classified belongs to C2, continues to classify the mail Y to be classified until the most appropriate category is selected as the final mail category of the mail Y to be classified.

10. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the email classification method based on the multimodal large language model as claimed in any one of claims 1 to 8 is implemented.

Images