A mail labeling method, device, equipment and storage medium
By extracting various types of data to be labeled from the raw data of emails and distributing them to multiple labeling terminals, and by adjusting the weight values of the labeling terminals, the problems of low efficiency and low accuracy in email labeling were solved, and higher accuracy in classification labels was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING LINGXING TECH CO LTD
- Filing Date
- 2023-05-30
- Publication Date
- 2026-05-19
AI Technical Summary
In existing technologies, email labeling is inefficient and prone to generating incorrect labels, especially for complex emails where the labeling accuracy is low.
By extracting different types of data to be labeled from the raw data of emails to be labeled, distributing them to multiple labeling terminals, and collaboratively determining classification labels based on the labeling results of multiple labeling terminals, the accuracy of labeling is improved by using preset content extraction rules and the weight values of the labeling terminals.
This significantly reduces the probability of generating incorrect labels, improves the accuracy of email label classification, and ensures the accuracy of subsequent training samples.
Smart Images
Figure CN116662882B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of email annotation technology, and in particular to an email annotation method, apparatus, device and storage medium. Background Technology
[0002] With the rapid development of internet technology and the emergence of various communication technologies, email, as a formal communication medium in the workplace, still plays a vital role. However, spam (or threatening emails) is rampant in the use of email, accounting for a high proportion of all internet emails and seriously affecting the normal operation of email systems.
[0003] In related technologies, machine learning models are typically used to identify spam (or threatening emails). However, machine learning models usually require a large number of training samples to be collected beforehand, and then the model is iteratively trained based on these samples. Therefore, the accuracy of the classification labels on the training samples directly affects the quality of the machine learning module. Currently, traditional email labeling is generally done by professional labelers, but manual labeling is inefficient with large-scale training samples and is prone to producing incorrect labels. To improve labeling efficiency and appropriately reduce the probability of incorrect labels, intelligent labeling models are often used. Intelligent labeling modules can handle simple labeling, but for more complex labeling, such as emails with variable content, the accuracy of intelligent labeling modules is often relatively low. Summary of the Invention
[0004] This application provides an email labeling method, apparatus, device, and storage medium to improve the accuracy of email labeling classification tags.
[0005] The specific technical solutions provided in this application are as follows:
[0006] In a first aspect, embodiments of this application provide an email annotation method, including:
[0007] Obtain the raw data of the email to be labeled, and extract the data to be labeled from the raw data according to the preset content extraction rules, wherein the type of the data to be labeled includes text and / or images;
[0008] The data to be labeled is sent to multiple labeling terminals respectively;
[0009] Receive annotation results returned by each annotation terminal, wherein the annotation results are obtained by the corresponding annotation terminal in response to a preset operation on the data to be annotated;
[0010] Based on multiple annotation results, the category tags of the emails to be annotated are determined.
[0011] The email annotation method provided in this application can extract different types of data to be annotated from the original data of the email to be annotated by using preset content extraction rules. The data to be annotated is then distributed to multiple annotation terminals. Based on the annotation results returned by multiple annotation terminals for the data to be annotated, the classification tags of the email to be annotated can be determined collaboratively, which greatly reduces the probability of generating incorrect tags and improves the accuracy of the classification tags for email annotation.
[0012] In one possible implementation, determining the category tag for the email to be tagged based on multiple tagging results includes:
[0013] Based on the multiple annotation results and the correspondence between the annotation results and the label values, the label values corresponding to the multiple annotation results are obtained;
[0014] Based on the label values corresponding to the multiple annotation results and the weight value of the annotation terminal corresponding to each label value, the label reference value corresponding to the data to be annotated is obtained;
[0015] If the tag reference value is within a preset range, the data to be labeled is sent to the target terminal for labeling, and the category tag of the email to be labeled is determined based on the target labeling result returned by the target terminal.
[0016] If the tag reference value is not within the preset range, then the category tag of the email to be tagged is determined based on the tag reference value.
[0017] In one possible implementation, after determining the classification tag for the email to be tagged based on the target tagging result returned by the target terminal, the method further includes:
[0018] From the plurality of annotation terminals, select each candidate annotation terminal that is consistent with the target annotation result;
[0019] Increase the weight value of candidate annotation terminals whose weight value is less than the first preset value;
[0020] Reduce the weight value of the annotation terminals (excluding the candidate annotation terminals) whose weight value is greater than the second preset value.
[0021] In one possible implementation, determining the category tag of the email to be tagged based on the tag reference value includes:
[0022] If the tag reference value is greater than the maximum value of the preset range, then the tag corresponding to the maximum value will be used as the category tag of the email to be tagged.
[0023] If the reference value of the tag is less than the minimum value of the preset range, then the tag corresponding to the minimum value will be used as the category tag of the email to be tagged.
[0024] In one possible implementation, the step of extracting the data to be labeled from the original data according to a preset content extraction rule includes:
[0025] Extract the body text data and / or body image data and / or body image OCR text data from the original data of the email to be annotated; and / or
[0026] Extract the attachment text data and / or attachment image data and / or attachment image OCR text data from the attachments of the email to be annotated from the original data;
[0027] The main text data and / or main image data and / or main image OCR text data, and / or the attachment text data and / or attachment image data and / or attachment image OCR text data, shall be used as the data to be annotated.
[0028] Secondly, embodiments of this application provide an email annotation device, comprising:
[0029] The data extraction module is used to acquire the raw data of the email to be labeled, and extract the data to be labeled from the raw data according to the preset content extraction rules, wherein the type of the data to be labeled includes text and / or images;
[0030] The sending module is used to send the data to be labeled to multiple labeling terminals respectively;
[0031] The receiving module is used to receive the annotation results returned by each annotation terminal, wherein the annotation results are obtained by the corresponding annotation terminal in response to a preset operation on the data to be annotated;
[0032] The determination module is used to determine the category tag of the email to be labeled based on multiple labeling results.
[0033] In one possible implementation, the determining module is specifically used for:
[0034] Based on the multiple annotation results and the correspondence between the annotation results and the label values, the label values corresponding to the multiple annotation results are obtained;
[0035] Based on the label values corresponding to the multiple annotation results and the weight value of the annotation terminal corresponding to each label value, the label reference value corresponding to the data to be annotated is obtained;
[0036] If the tag reference value is within a preset range, the data to be labeled is sent to the target terminal for labeling, and the category tag of the email to be labeled is determined based on the target labeling result returned by the target terminal.
[0037] If the tag reference value is not within the preset range, then the category tag of the email to be tagged is determined based on the tag reference value.
[0038] In one possible implementation, after determining the category tag for the email to be tagged based on the target tagging result returned by the target terminal, the determining module is further configured to:
[0039] From the plurality of annotation terminals, select each candidate annotation terminal that is consistent with the target annotation result;
[0040] Increase the weight value of candidate annotation terminals whose weight value is less than the first preset value;
[0041] Reduce the weight value of the annotation terminals (excluding the candidate annotation terminals) whose weight value is greater than the second preset value.
[0042] In one possible implementation, the determining module is specifically used for:
[0043] If the tag reference value is greater than the maximum value of the preset range, then the tag corresponding to the maximum value will be used as the category tag of the email to be tagged.
[0044] If the reference value of the tag is less than the minimum value of the preset range, then the tag corresponding to the minimum value will be used as the category tag of the email to be tagged.
[0045] In one possible implementation, the data extraction module is specifically used for:
[0046] Extract the body text data and / or body image data and / or body image OCR text data from the original data of the email to be annotated; and / or
[0047] Extract the attachment text data and / or attachment image data and / or attachment image OCR text data from the attachments of the email to be annotated from the original data;
[0048] The main text data and / or main image data and / or main image OCR text data, and / or the attachment text data and / or attachment image data and / or attachment image OCR text data, shall be used as the data to be annotated.
[0049] Thirdly, embodiments of this application provide an electronic device, including:
[0050] Memory is used to store computer programs or instructions;
[0051] A processor for executing a computer program or instructions in the memory such that the method described in any of the first aspects is performed.
[0052] Fourthly, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the processor to perform the method described in any one of the first aspects above.
[0053] Fifthly, embodiments of this application provide a computer program product comprising: computer program code, which, when executed on a computer, causes the computer to perform the method described in any one of the first aspects.
[0054] Furthermore, the technical effects of any of the implementation methods in the second to fifth aspects can be found in the technical effects of different implementation methods in the first aspect, and will not be repeated here.
[0055] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0056] Figure 1A This is a schematic diagram of an application scenario in an embodiment of this application;
[0057] Figure 1B This is a schematic diagram of the structure of an email annotation system in an embodiment of this application;
[0058] Figure 2 This is a flowchart illustrating an email annotation method according to an embodiment of this application;
[0059] Figure 3 This is a flowchart illustrating a method for determining data to be labeled in an embodiment of this application;
[0060] Figure 4A This is a screenshot of the email to be labeled in an embodiment of this application;
[0061] Figure 4B This is a schematic diagram of the data extraction process in an embodiment of this application;
[0062] Figure 5 This is a schematic diagram of a process for determining the classification tags of emails to be labeled in an embodiment of this application;
[0063] Figure 6 This is a flowchart illustrating a method for adjusting the weight value of a labeling terminal according to an embodiment of this application;
[0064] Figure 7 This is a schematic diagram of the logical architecture of an email annotation device according to an embodiment of this application;
[0065] Figure 8 This is a schematic diagram of the physical architecture of an electronic device according to an embodiment of this application. Detailed Implementation
[0066] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0067] It should be noted that the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.
[0068] In this embodiment, to address the problem of low accuracy in email labeling classification under related technologies, the present embodiment extracts the data to be labeled from the raw data of the received email to be labeled according to preset content extraction rules. This data is then sent to multiple labeling terminals, and the labeling results returned by each terminal are received. Based on these multiple labeling results, the classification label of the email to be labeled is determined. Thus, by using preset content extraction rules, different types of data to be labeled can be extracted from the raw data of the email to be labeled. This data is then distributed to multiple labeling terminals, and the classification label of the email to be labeled can be collaboratively determined based on the labeling results returned by the multiple terminals. This significantly reduces the probability of generating incorrect labels and improves the accuracy of email labeling classification.
[0069] The preferred embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.
[0070] See Figure 1A The diagram shown illustrates an application scenario of an embodiment of this application. This application scenario includes a mail server 100, a mail annotation system 200, and multiple annotation terminals 300.
[0071] The email annotation system 200 is used to obtain raw data of multiple emails to be annotated from the email server 100. The emails to be annotated include any one of the following: emails reported by users, emails identified as bad cases by the online model after manual verification, and emails randomly sampled online that were not identified by the model. Then, for each email to be annotated, the following steps are performed: extract the data to be annotated from the raw data according to the preset content extraction rules, wherein the type of the data to be annotated includes text and / or images; and send the data to be annotated to multiple annotation terminals 300 respectively.
[0072] Each annotation terminal 300 is used to respond to a preset operation on the received data to be annotated, obtain the annotation result of the data to be annotated, and return the annotation result to the email annotation system 200.
[0073] The email labeling system 200 is also used to determine the category tag of the email to be labeled based on multiple labeling results after receiving the labeling results returned by each labeling terminal.
[0074] In some feasible embodiments, the email annotation system 200 is specifically used to extract the data to be annotated from the original data of the annotated email, which includes multiple types of data, such as text data and image data. The multiple types of data can be sent to each annotation terminal together, or they can be pushed to each annotation terminal separately. The data to be annotated extracted from the same email carries the identifier of the annotated email.
[0075] In some feasible embodiments, if the email annotation system 200 sends the various types of data contained in the data to be annotated separately to each annotation terminal, the email annotation system 200 can select the annotation results of each annotation terminal for the various types of data to be annotated in the email to be annotated from the multiple annotation results received based on the identifier of the email to be annotated, and summarize the annotation results of the annotation terminal for the email to be annotated based on the annotation results of the various types of data to be annotated returned by the corresponding annotation terminal.
[0076] In the above application scenarios, please refer to Figure 1B As shown, the email annotation system 200 includes an incremental data collection cluster 201 and a real-time processing cluster 202. Among them,
[0077] The incremental acquisition cluster 201 consists of multiple servers, which are used to obtain the raw data of the emails to be labeled from the mail server 100 and send the obtained raw data to the message queue 203 in a unified format.
[0078] The real-time processing cluster 202 consists of multiple processing nodes, which listen to the message queue 203, receive the raw data of the email to be labeled, and extract the data to be labeled from the raw data according to the preset content extraction rules. The data to be labeled includes text and / or images. Then, the data to be labeled is sent to multiple labeling terminals 300, and the labeling results returned by each labeling terminal 300 are received. Based on the multiple labeling results, the category tag of the email to be labeled is determined. Finally, the email to be labeled and the category tag are stored in the preset database 204.
[0079] In this embodiment, the introduction of incremental acquisition clusters and real-time processing clusters ensures the stability, reliability, and low latency of the email annotation system; furthermore, the message queue guarantees the high throughput of the email annotation system. Therefore, under complex daily production conditions, Figure 1B The demonstrated email annotation system can handle the collection, distribution, and processing of tens of millions of data points, ensuring the smooth progress of subsequent collaborative annotation. In practice, the system can distribute the data to be annotated to various annotation terminals within the annotation team via a real-time processing cluster, or to annotation terminals outside the team. This allows for flexible management and organization of the annotation team, further improving the accuracy of email annotation classification labels. Consequently, it becomes possible to iteratively train a more accurate model based on the training samples obtained using the aforementioned email annotation method.
[0080] After introducing the application scenarios of the embodiments of this application, Figure 2 This paper illustrates an email annotation method in an embodiment of this application, which is applied to an email annotation system. It should be noted that the following detailed description of the processing flow of this application's technical solution only uses the processing of one email to be annotated as an example. In practice, the email annotation system can use the same processing flow to process multiple emails to be annotated to determine the corresponding classification tags for each email, thereby obtaining a massive amount of training samples. See also... Figure 2 As shown, the specific process of an email annotation method in this application embodiment is as follows:
[0081] Step 400: Obtain the raw data of the email to be labeled, and extract the data to be labeled from the raw data according to the preset content extraction rules. The data to be labeled includes text and / or images.
[0082] In this embodiment of the application, when obtaining the original data of the email to be labeled in step 400, the email labeling system obtains the original data of the email to be labeled from the email server. The email to be labeled may be an email reported by a user, a bad case that has been identified incorrectly by the online model by manual verification, or an email that has not been identified by the model and is randomly sampled online.
[0083] Typically, the main content of an email is the email body. However, analysis of a large number of spam (or threatening emails) reveals that senders of spam (or threatening emails) employ various hiding and packaging methods to avoid being blocked by anti-spam systems (or defense systems, etc.). Email attachments thus become carriers of harmful content. Therefore, in this embodiment, after obtaining the original data of the email to be labeled, refer to... Figure 3 As shown, when extracting the data to be labeled from the original data according to the preset content extraction rules in step 400, the following steps can be performed:
[0084] Step 500: Extract the body text data and / or body image data and / or body image OCR text data from the original data of the email to be annotated; and / or, extract the attachment text data and / or attachment image data and / or attachment image OCR text data from the attachments of the email to be annotated from the original data.
[0085] Among them is optical character recognition (OCR).
[0086] For specific implementation details, please refer to... Figure 4A As shown, assume the email to be annotated includes the email body and an attachment. The email body includes text (denoted as body text) and an image (denoted as body image), and the attachment is an image (denoted as attachment image), such as an image containing an invoicing link. Then, refer to... Figure 4B As shown, after obtaining the original data of the email to be annotated, the body text data and body image data of the email body to be annotated are extracted from the original data. Then, the body image OCR text data is identified from the body image data. The attachment image data of the attachment to be annotated is extracted from the original data, and the attachment image OCR text data is identified from the attachment image data.
[0087] In some feasible embodiments, the above-mentioned main text image data includes image feature information corresponding to the main text image, and the attachment image data includes image feature information corresponding to the attachment image.
[0088] In some feasible embodiments, if the attachment is a compressed file, the email annotation system extracts the attachment data from the original data, then decompresses the attachment data to obtain attachment image data and / or attachment text data, and identifies attachment image OCR text data from the attachment image data.
[0089] Step 510: Take the main text data and / or main image data and / or main image OCR text data, and / or, attachment text data and / or attachment image data and / or attachment image OCR text data as the data to be annotated.
[0090] In the embodiments of this application, please refer to Figure 4B As shown, the main text data, main image data, main image OCR text data, attachment image data, and attachment image OCR text data are used as the data to be annotated.
[0091] Step 410: Send the data to be labeled to multiple labeling terminals respectively.
[0092] In the embodiments of this application, please refer to Figure 4B As shown, when performing step 410, the main text data, main image data, main image OCR text data, attachment image data, and attachment image OCR text data can be combined into one file, copied multiple times, and sent to multiple annotation terminals respectively; alternatively, the main text data, main image data, main image OCR text data, attachment image data, and attachment image OCR text data can each be combined into one file and sent to multiple annotation terminals respectively.
[0093] For specific implementation details, please refer to... Figure 2 As shown, the above-mentioned data to be labeled can be pushed to multiple labeling terminals simultaneously through a real-time processing cluster, with the help of each processing node, so as to ensure the real-time nature and consistency of the distributed content.
[0094] Step 420: Receive the annotation results returned by each annotation terminal, wherein the annotation results are obtained by the corresponding annotation terminal in response to a preset operation on the data to be annotated.
[0095] In practice, annotators can obtain the data to be annotated through the annotation terminal, display the data on the terminal's display interface, and perform preset operations on the display interface to annotate the data to obtain the annotation results. Then, the annotation results are sent to the email annotation system through the annotation terminal, and the email annotation system receives the annotation results returned by each annotation terminal.
[0096] Step 430: Determine the category tags for the emails to be annotated based on multiple annotation results.
[0097] In the embodiments of this application, see the following: Figure 5 As shown, when executing step 430, the following steps are specifically performed:
[0098] Step 4301: Based on the multiple annotation results and the correspondence between the annotation results and the label values, obtain the label values corresponding to the multiple annotation results.
[0099] In this embodiment of the application, the correspondence between the annotation result and the tag value can be preset based on the following: if the annotation result indicates that the email to be annotated is spam (or a threatening email), then the tag value corresponding to the annotation result is 1; if the annotation result indicates that the email to be annotated is normal (or a non-threatening email), then the tag value corresponding to the annotation result is -1.
[0100] In practice, when executing step 4301, the label value corresponding to each annotation result can be obtained from the correspondence between the annotation results and the label values.
[0101] Step 4302: Based on the label values corresponding to multiple annotation results and the weight value of the annotation terminal corresponding to each label value, obtain the label reference value corresponding to the data to be annotated.
[0102] In this embodiment, a weight value is pre-configured for each annotation terminal to characterize the annotation reliability of that terminal. For example, an initial weight value can be configured for each annotation terminal, and the weight value of each annotation terminal can be dynamically adjusted based on subsequent adjustment methods. See still for further details. Figure 1B As shown, the aforementioned weight values can be stored in the preset database 204. In specific implementation, when executing step 4302, the label reference value l corresponding to the data to be labeled can be calculated using the following formula:
[0103]
[0104] Among them, l i For the annotation result of the i-th annotation terminal, in this embodiment of the application, l i The value is 1 or -1, representing whether the email to be labeled is a positive sample (spam or a threatening email) or a negative sample (normal email or a non-threatening email), respectively; the weight value w i (0 <w i <1) Represents the credibility of the annotation of the i-th annotation terminal; n is the total number of annotation terminals.
[0105] Step 4303: If the tag reference value is within the preset range, the data to be labeled is sent to the target terminal for labeling, and the category tag of the email to be labeled is determined based on the target labeling result returned by the target terminal.
[0106] The aforementioned preset interval can be (-r, r), where, n represents the total number of labeled terminals.
[0107] See also Figure 1AAs shown, this application scenario also includes a target terminal (not shown in the figure). The annotators for the target terminal are different from those for the annotation terminal. The annotators for the target terminal are usually more experienced annotators or algorithm engineers.
[0108] In practice, the aforementioned tag reference values are compared with a preset range. If the tag reference value is determined to be within the preset range, it is determined that the tagging result of the email to be tagged is disputed. Then, step 4303 is executed, the data to be tagged is sent to the target terminal for tagging, and the category tag of the email to be tagged is determined based on the target tagging result returned by the target terminal. Since the taggers corresponding to the target terminal are usually more experienced taggers or algorithm engineers, the target tagging result returned by the target terminal is more authoritative and reliable. Therefore, the category tag of the email to be tagged determined based on the target tagging result is also more accurate.
[0109] In this embodiment of the application, after performing step 4303, refer to... Figure 6 As shown, the weight values of some annotation terminals are further adjusted by performing the following steps to improve the overall accuracy of email annotation classification tags:
[0110] Step 600: Select candidate annotation terminals that are consistent with the target annotation result from multiple annotation terminals.
[0111] Step 610: Increase the weight value of candidate annotation terminals whose weight value is less than the first preset value.
[0112] The first preset value can be 1. When performing step 610, the weight values of candidate annotation terminals whose weight values are less than the first preset value are appropriately increased.
[0113] In practice, the specific adjustment method for the weight value can be set according to the actual situation. For example, the specific adjustment method can be to increase a fixed value, such as a fixed value of 0.05; the specific adjustment method can also be to increase the weight value according to a certain gradient based on historical annotation data (such as the cumulative number of correct judgments), such as increasing the weight value by 0.01 if the cumulative number of correct judgments is 1; increasing it by 0.03 if the cumulative number of correct judgments is 2; ...; increasing it by 0.1 if the cumulative number of correct judgments is a, etc.
[0114] Step 620: Reduce the weight values of annotation terminals (excluding candidate annotation terminals) whose weight values are greater than the second preset value.
[0115] The second preset value can be 0. In practice, the specific adjustment method of the weight value can also be set according to the actual situation, such as lowering the fixed value, or lowering it according to the gradient based on historical annotation.
[0116] In this embodiment of the application, after performing steps 610 and 620, the adjusted weight values are stored in a preset database. Figure 1B (As shown)
[0117] Step 4304: If the tag reference value is not within the preset range, determine the category tag of the email to be tagged based on the tag reference value.
[0118] In practice, the above-mentioned tag reference value is compared with the preset range. If it is determined that the tag reference value is not within the preset range, it is determined that there is no dispute about the tagging result of the email to be tagged. Then, step 4304 is executed to determine the category tag of the email to be tagged based on the tag reference value.
[0119] In this embodiment of the application, when performing step 4304, the following two situations are included, but not limited to:
[0120] In case one, if the tag reference value is greater than the maximum value of the preset range, the tag corresponding to the maximum value will be used as the category tag for the email to be tagged.
[0121] Scenario 2: If the tag reference value is less than the minimum value of the preset range, then the tag corresponding to the minimum value will be used as the category tag for the email to be tagged.
[0122] In some feasible embodiments, see still Figure 4B As shown, in step 410, if the email annotation system sends the body text data, body image data, body image OCR text data, attachment image data, and attachment image OCR text data as separate files to multiple annotation terminals, the email annotation system will receive annotation results returned by each annotation terminal for the body text data, body image data, body image OCR text data, attachment image data, and attachment image OCR text data.
[0123] Therefore, when executing step 4301, firstly, based on the annotation results returned by any annotation terminal for the email to be annotated, the annotation result of the corresponding annotation terminal for the email to be annotated is determined. If one of the annotation results indicates that the email to be annotated is spam (or a threatening email), then the annotation result of the annotation terminal for the email to be annotated is determined to be spam (or a threatening email). If all the annotation results returned by the annotation terminal indicate that the email to be annotated is normal (or a non-threatening email), then the annotation result of the annotation terminal for the email to be annotated is determined to be normal (or a non-threatening email).
[0124] Then, based on the annotation results of multiple annotation terminals for the email to be annotated, and the correspondence between the annotation results and the tag values, the tag values corresponding to the multiple annotation results are obtained, thereby obtaining the tag reference value, and based on the tag reference value, the classification tag of the email to be annotated is determined.
[0125] Thus, by adopting the email annotation method provided in this application embodiment, different types of data to be annotated can be extracted from the original data of the email to be annotated through preset content extraction rules. The different types of data to be annotated are distributed to multiple annotation terminals to collaboratively determine the classification labels of the email to be annotated. This greatly reduces the probability of generating incorrect classification labels and improves the accuracy of the classification labels of email annotation. This ensures that subsequent machine learning based on a large number of training samples with classification labels can obtain a high-precision model.
[0126] Based on the same inventive concept, see [reference] Figure 7 As shown in the figure, this application provides an email annotation device, including:
[0127] The data extraction module 710 is used to acquire the raw data of the email to be labeled, and extract the data to be labeled from the raw data according to the preset content extraction rules, wherein the type of the data to be labeled includes text and / or images;
[0128] The sending module 720 is used to send the data to be labeled to multiple labeling terminals respectively;
[0129] The receiving module 730 is used to receive the annotation results returned by each annotation terminal, wherein the annotation results are obtained by the corresponding annotation terminal in response to a preset operation on the data to be annotated;
[0130] The determination module 740 is used to determine the category tag of the email to be labeled based on multiple labeling results.
[0131] In one possible implementation, the determining module 740 is specifically used for:
[0132] Based on the multiple annotation results and the correspondence between the annotation results and the label values, the label values corresponding to the multiple annotation results are obtained;
[0133] Based on the label values corresponding to the multiple annotation results and the weight value of the annotation terminal corresponding to each label value, the label reference value corresponding to the data to be annotated is obtained;
[0134] If the tag reference value is within a preset range, the data to be labeled is sent to the target terminal for labeling, and the category tag of the email to be labeled is determined based on the target labeling result returned by the target terminal.
[0135] If the tag reference value is not within the preset range, then the category tag of the email to be tagged is determined based on the tag reference value.
[0136] In one possible implementation, after determining the category tag of the email to be tagged based on the target tagging result returned by the target terminal, the determining module 740 is further configured to:
[0137] From the plurality of annotation terminals, select each candidate annotation terminal that is consistent with the target annotation result;
[0138] Increase the weight value of candidate annotation terminals whose weight value is less than the first preset value;
[0139] Reduce the weight value of the annotation terminals (excluding the candidate annotation terminals) whose weight value is greater than the second preset value.
[0140] In one possible implementation, the determining module 740 is specifically used for:
[0141] If the tag reference value is greater than the maximum value of the preset range, then the tag corresponding to the maximum value will be used as the category tag of the email to be tagged.
[0142] If the reference value of the tag is less than the minimum value of the preset range, then the tag corresponding to the minimum value will be used as the category tag of the email to be tagged.
[0143] In one possible implementation, the data extraction module 710 is specifically used for:
[0144] Extract the body text data and / or body image data and / or body image OCR text data from the original data of the email to be annotated; and / or
[0145] Extract the attachment text data and / or attachment image data and / or attachment image OCR text data from the attachments of the email to be annotated from the original data;
[0146] The main text data and / or main image data and / or main image OCR text data, and / or the attachment text data and / or attachment image data and / or attachment image OCR text data, shall be used as the data to be annotated.
[0147] Based on the same inventive concept, this application provides an electronic device, which is described below with reference to... Figure 8 To describe an electronic device 80 according to this embodiment of the present application. Figure 8 The electronic device 80 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0148] like Figure 8 As shown, the electronic device 80 is presented in the form of a general-purpose electronic device. The components of the electronic device 80 may include, but are not limited to: at least one processor 81, at least one memory 82, and a bus 83 connecting different system components (including memory 82 and processor 81).
[0149] Bus 83 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or a local bus using any of the various bus structures.
[0150] The memory 82 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 821 and / or cache memory 822, and may further include read-only memory (ROM) 823.
[0151] The memory 82 may also include a program / utility 825 having a set (at least one) of program modules 824, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.
[0152] Electronic device 80 can also communicate with one or more external devices 84 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 80, and / or with any device that enables electronic device 80 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 85. Furthermore, electronic device 80 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 86. As shown, network adapter 86 communicates with other modules used in electronic device 80 via bus 83. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 80, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0153] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium that, when executed by a processor, enables the processor to perform the method described in any of the preceding claims. Since the principle by which the computer-readable storage medium solves the problem is similar to that of the email annotation method, the implementation of the computer-readable storage medium can be found in the implementation of the method; repeated details will not be elaborated further.
[0154] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to execute any of the methods described above. Since the principle by which the above computer program product solves the problem is similar to that of the email annotation method, the implementation of the above computer program product can be referred to the implementation of the method, and repeated details will not be elaborated further.
[0155] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0156] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.
[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more processes in a flowchart and / or one or more blocks in a block diagram.
[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.
[0159] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. An email annotation method, characterized in that, include: Obtain the raw data of the email to be labeled, and extract the data to be labeled from the raw data according to the preset content extraction rules, wherein the type of the data to be labeled includes text and / or images; The data to be labeled is sent to multiple labeling terminals respectively; Receive annotation results returned by each annotation terminal, wherein the annotation results are obtained by the corresponding annotation terminal in response to a preset operation on the data to be annotated; Based on multiple annotation results and the correspondence between the annotation results and the tag values, the tag values corresponding to the multiple annotation results are obtained. The annotation result in the correspondence indicates that the tag value is 1 when the email to be annotated is spam or a threatening email, and -1 when it is a normal email. Based on the label values corresponding to the multiple annotation results and the weight value of the annotation terminal corresponding to each label value, the label reference value corresponding to the data to be annotated is obtained. The label reference value is compared with a preset interval, where the preset interval is (-r, r), and r is a positive number less than one-third of the total number of annotation terminals n. If the tag reference value is within (-r, r), it is determined that the tagging result of the email to be tagged is disputed. The data to be tagged is sent to the target terminal for tagging, and the category tag of the email to be tagged is determined based on the target tagging result returned by the target terminal. If the tag reference value is not within (-r, r), then it is determined that there is no dispute regarding the tagging result of the email to be tagged, and the classification tag of the email to be tagged is determined according to the tag reference value.
2. The method as described in claim 1, characterized in that, After determining the category tag for the email to be tagged based on the target tagging results returned by the target terminal, the method further includes: From the plurality of annotation terminals, select each candidate annotation terminal that is consistent with the target annotation result; Increase the weight value of candidate annotation terminals whose weight value is less than the first preset value; Reduce the weight value of the annotation terminals (excluding the candidate annotation terminals) whose weight value is greater than the second preset value.
3. The method as described in claim 1, characterized in that, The step of determining the category tag for the email to be tagged based on the tag reference value includes: If the tag reference value is greater than the maximum value of the preset range, then the tag corresponding to the maximum value will be used as the category tag of the email to be tagged. If the reference value of the tag is less than the minimum value of the preset range, then the tag corresponding to the minimum value will be used as the category tag of the email to be tagged.
4. The method according to any one of claims 1-3, characterized in that, The step of extracting the data to be labeled from the original data according to the preset content extraction rules includes: Extract the body text data and / or body image data and / or body image OCR text data from the original data of the email to be annotated; and / or Extract the attachment text data and / or attachment image data and / or attachment image OCR text data from the attachments of the email to be annotated from the original data; The main text data and / or main image data and / or main image OCR text data, and / or the attachment text data and / or attachment image data and / or attachment image OCR text data, shall be used as the data to be annotated.
5. A mail annotation device, characterized in that, include: The data extraction module is used to acquire the raw data of the email to be labeled, and extract the data to be labeled from the raw data according to the preset content extraction rules, wherein the type of the data to be labeled includes text and / or images; The sending module is used to send the data to be labeled to multiple labeling terminals respectively; The receiving module is used to receive the annotation results returned by each annotation terminal, wherein the annotation results are obtained by the corresponding annotation terminal in response to a preset operation on the data to be annotated; The determination module is used to obtain the label value corresponding to the multiple annotation results based on the multiple annotation results and the correspondence between the annotation results and the label value. The annotation result in the correspondence represents a label value of 1 when the email to be annotated is spam or a threatening email, and a label value of -1 when it is a normal email. Based on the label values corresponding to the multiple annotation results and the weight value of the annotation terminal corresponding to each label value, a label reference value corresponding to the data to be annotated is obtained. The label reference value is compared with a preset interval (-r, r), where r is a positive number less than one-third of the total number of annotation terminals n. If the label reference value is within (-r, r), it is determined that the annotation result of the email to be annotated is disputed, and the data to be annotated is sent to the target terminal for annotation. Based on the target annotation result returned by the target terminal, the category label of the email to be annotated is determined. If the label reference value is not within (-r, r), it is determined that the annotation result of the email to be annotated is not disputed, and the category label of the email to be annotated is determined according to the label reference value.
6. The apparatus as claimed in claim 5, characterized in that, After determining the category tag for the email to be tagged based on the target tagging results returned by the target terminal, the determining module is further configured to: From the plurality of annotation terminals, select each candidate annotation terminal that is consistent with the target annotation result; Increase the weight value of candidate annotation terminals whose weight value is less than the first preset value; Reduce the weight value of the annotation terminals (excluding the candidate annotation terminals) whose weight value is greater than the second preset value.
7. An electronic device, characterized in that, include: Memory is used to store computer programs or instructions; A processor for executing a computer program or instructions in the memory such that the method described in any one of claims 1-4 is performed.
8. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor, the processor is able to perform the method as described in any one of claims 1-4.
9. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of claims 1-4.