Systems and methods for automatically extracting classification training data

Abstract

A method for training a multi-class classification model includes receiving training data corresponding to a plurality of classes. For each class of the plurality of classes, the method includes training a binary classification model configured to determine whether a training data observation belongs to the class, and for each training data observation identified as belonging to the class, extracting one or more class identifying features from the training data observation based on activations of an intermediate attention layer in the binary classification model. The multi-class classification model is trained using the class identifying features extracted for each class of the plurality of classes.

Description

Technical Field

[0001] This invention generally relates to the field of data classification using deep learning models, and more specifically to the classification of text documents.

[0002] Cross-references to related applications

[0003] This application claims the benefit of U.S. Patent Application No. 17 / 125,120, entitled “SYSTEMS ANDMETHODS FOR AUTOMATIC EXTRACTION OF CLASSIFICATION TRAINING DATA”, filed on December 17, 2020, with the United States Patent and Trademark Office. Background Technology

[0004] Deep learning models can be used to classify various forms of data. These models are known to be trained on a training data corpus, which includes a large number of data observations with pre-assigned categories. Using machine learning techniques, the deep learning model assigns previously unseen data observations to one of several categories seen in the training data. Generally, the larger the training data corpus, the higher the accuracy of the trained model, and the greater the number of categories that can be accurately trained.

[0005] An example of a classification model is a text classification model, where an input text document is assigned to one of several categories. For instance, an email management tool can use a text classification model to automatically assign incoming emails to one of several known classes. The email management tool can then automatically take actions based on the assigned class, such as generating a response or a follow-up task.

[0006] Figure 1 This is a schematic diagram illustrating a conventional document classifier 100 known in the art, such as an email management tool using a text classification model. The document classifier 100 is configured to receive input text 10 in the form of a text document (e.g., an email). The document classifier 100 then assigns the input text 10 to one of a plurality (N) document classes 20. Where feedback is available, such as through supervision, sampling, or customer feedback, an update mechanism can then improve the text classification model by providing positive reinforcement for correct results and negative reinforcement for incorrect results.

[0007] In the initial training phase, document classifier 100 receives training data 30 in the form of multiple email documents. The email documents in the training data 30 can be referred to as observations. Each email document in the training data 30 has a pre-assigned category; for example, each email can be categorized based on the author's intent or purpose. In some implementations, for example, the training data 30 may include replies to multiple outgoing email communications (such as marketing emails). Replies can be categorized into one or more classes, such as "not interested," "interested," "existing customer," "wrong person," "unsubscribe," "left the company," "contact later," etc.

[0008] Following conventional machine learning techniques, the parameters of the text classification model are randomized in the initial state, and email documents are assigned to multiple classes 20. An update mechanism is used to improve the parameters of the text classification model by comparing these assignments with the known classifications of each email document. This process is repeated until the document classifier 100 can reliably assign the email documents of the training data 30 to the correct class 20.

[0009] At this stage, the intention is to enable text classification models to accurately assign previously unseen email documents. However, the number of documents needed to train a text classification model to a satisfactory level of accuracy may exceed the number of available training documents (e.g., a number of stored and classified emails). Furthermore, some email documents may be particularly difficult to classify correctly. Below are examples of such email documents that might be misclassified by multi-class classification models:

[0010] "I apologize for the long delay in replying. My manager and I reconsidered your offer yesterday. While it seems like a good fit, we currently do not have the budget for it. We will contact you if it becomes a priority in the future. Thank you for your time!"

[0011] This example is easily misclassified as "interesting" instead of "not interested".

[0012] Furthermore, creating additional training data by labeling unclassified training corpora can be laborious and inefficient. Additionally, remedying low model accuracy by using update mechanisms on real-world data over a period of time can lead to an unacceptable number of errors in the early stages. Moreover, if some training documents are irrelevant to the class, even a large amount of training data can hinder the speed and efficiency of the training process.

[0013] Therefore, there is a need to improve the training data corpus. This invention aims to address this need. Summary of the Invention

[0014] It should be understood that the following overview and detailed description are exemplary and illustrative, and are intended to provide further explanation of the claimed invention. The following overview and description are not intended to limit or restrict the scope of the invention to the specific features mentioned in the overview or description.

[0015] Generally, the present invention provides systems and methods related to training multi-class classification models, including receiving training data corresponding to multiple classes. Embodiments of the present invention include: for each of the multiple classes, training a binary classification model configured to determine whether a training data observation belongs to that class; and for each training data observation identified as belonging to that class, extracting one or more class identification features from the training data observation based on the activation of an intermediate attention layer in the binary classification model.

[0016] In at least one embodiment, a method for detecting markers is provided. The method includes: receiving training data corresponding to a plurality of classes by a processor; and, for each of the plurality of classes: training a binary classification model by the processor, the binary classification model being configured to determine whether a training data observation belongs to that class; for each training data observation identified as belonging to that class, extracting one or more class-identifying features from the training data observation based on activation of an intermediate attention layer in the binary classification model; and training a multi-class classification model by the processor using the class-identifying features extracted for each of the plurality of classes.

[0017] In at least another embodiment, a processing apparatus includes at least one processor configured to perform a method comprising the steps of: receiving training data corresponding to a plurality of classes; and, for each of the plurality of classes: training a binary classification model by the processor, the binary classification model being configured to determine whether training data observations belong to that class; for each training data observation identified as belonging to that class, extracting one or more class identification features from the training data observations based on activation of an intermediate attention layer in the binary classification model; and training a multi-class classification model by the processor using the class identification features extracted for each of the plurality of classes.

[0018] In at least one further embodiment of the invention, a computer-readable medium includes a non-transitory computer-readable medium configured to store instructions that, when executed by a processor, cause the processor to perform a method comprising the steps of: receiving training data corresponding to a plurality of classes; and, for each of the plurality of classes: training a binary classification model by the processor, the binary classification model being configured to determine whether training data observations belong to that class; for each training data observation identified as belonging to that class, extracting one or more class identification features from the training data observations based on activation of an intermediate attention layer in the binary classification model; and training a multi-class classification model by the processor using the class identification features extracted for each of the plurality of classes.

[0019] These and further and other objects and features of the invention will be apparent in this disclosure, which includes the foregoing and ongoing written description and drawings. Attached Figure Description

[0020] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate exemplary embodiments and, together with the accompanying drawings, are further intended to enable those skilled in the art to implement and use these embodiments, as well as other embodiments that are obvious to those skilled in the art.

[0021] Figure 1 This is a schematic diagram illustrating a conventional document classifier system known in the art;

[0022] Figure 2 This is a schematic diagram illustrating a processing apparatus according to an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram illustrating the model architecture for a binary classification model according to an embodiment of the present invention;

[0024] Figure 4 This is a block diagram illustrating a document classification system according to an embodiment of the present invention; and

[0025] Figure 5 This is a flowchart illustrating a method according to an embodiment of the present invention. Detailed Implementation

[0026] The invention will now be described more fully with reference to the accompanying drawings. The following description is exemplary, as several embodiments are described (e.g., by using the terms “preferred,” “for example,” or “in one embodiment”); however, this should not be considered limiting or as illustrating only the embodiments of the invention, as the invention encompasses other embodiments not specifically described in this specification, including substitutions, modifications, and equivalents within the spirit and scope of the invention. Furthermore, the use of the terms “invention,” “this invention,” “implementation,” and similar terms throughout the specification is extensive and does not imply that the invention is required to be or limited to any particular aspect described, or that such description is the only way the invention can be made or used. Additionally, the invention may be described in the context of a particular application; however, the invention can be used in various applications not specifically described.

[0027] In these figures, similar reference numerals may be used for similar elements having similar functions, even in different figures. The described embodiments, along with their detailed construction and elements, are provided merely to aid in a comprehensive understanding of the invention. Therefore, it will be apparent that the invention can be implemented in many ways and does not require any of the specific features described herein. Furthermore, well-known functions or constructions are not described in detail, as they would obscure the invention with unnecessary detail. Unless otherwise specifically indicated, any signal arrows in the figures / graphs should be considered exemplary only, not restrictive. Moreover, since the scope of the invention is best defined by the appended claims, this description should not be construed as limiting, but is merely for the purpose of illustrating the general principles of the invention.

[0028] It should be understood that although terms such as first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. Purely by way of non-limiting example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to also include the plural forms. It should also be noted that in some alternative embodiments, the mentioned functions and / or actions may not occur in the order shown in at least one of these figures. Purely by way of non-limiting example, depending on the described or depicted functions and / or actions, two figures shown consecutively may actually be performed substantially simultaneously or sometimes in reverse order.

[0029] Unless otherwise specifically stated, or otherwise understood in the context in which they are used, conditional language such as “can,” “may,” “may,” or “may” is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language is not generally intended to imply that features, elements, and / or steps are required in any way by one or more embodiments, or that one or more embodiments necessarily include logic for determining, with or without user input or prompting, whether such features, elements, and / or steps are included in any particular embodiment or will be performed in any particular embodiment.

[0030] This invention relates to a processing apparatus and method for training a classification model. Specifically, the processing method is applied to a new training data corpus before training the classification model, and the processed training data is used to train the classification model.

[0031] The classification model can be a document classification model and can be configured to assign multiple text documents to multiple document classes. For example, a document classification model can receive multiple incoming emails and assign each incoming email to a class based on the author's intent. Alternatively, in some implementations, the classification can be, for example, an image classification model.

[0032] The attached image Figure 2 A schematic diagram of an embodiment of the processing apparatus 200 according to the present invention is shown. The processing apparatus 200 includes a data input 210, a memory 220, a data processing module 230, and an output 240. The processing apparatus 200 is configured to train a multi-class classification model. The multi-class classification model may be a document classification model.

[0033] Data input 210 is configured to receive training data 30 corresponding to multiple classes. In some examples, training data 30 can be received via a local connection, via a network connection, or retrieved from local memory.

[0034] In the case where the multi-class classification model is a document classification model, the training data 30 may include multiple text documents. Each of the multiple classes can correspond to the intended purpose of the text documents. The data input 210 is configured to provide the data processing module 230 with the training data 30 corresponding to the multiple classes.

[0035] Memory 220 is configured to store instructions for data processing module 230. Memory 220 may include, for example, one or more hard disk drives, solid-state drives, or a combination thereof, as needed. Memory 220 stores, for example, algorithms, programs, applications, software, etc., for execution by a processor such as data processing module 230. When executed, the instructions cause data processing module 230 to perform processing methods, such as executing a training method for a multi-class classification model.

[0036] In some examples, memory 220 is also configured to store operational data. This operational data can be persistently stored; for example, memory 220 can record the final or intermediate output of the data processing module. In some examples, memory 220 can store training data 30. Memory 220 can store multiple training data corpora, such as training data received at different points in time. In some examples, the data can be temporarily stored. Memory 220 can include one or more transient storage media, such as flash memory. Memory 220 can store parameters or variables required for computation only for a short period of time.

[0037] Data processing module 230 includes one or more processors. Data processing module 230 is configured to receive training data 30 corresponding to multiple classes from data input 210. Alternatively, in some examples, data processing module 230 may receive training data 30 from memory 220. Data processing module 230 is configured to train a binary classification model 231 for each of the multiple classes.

[0038] Each binary classification model 231 is configured to determine whether a data observation (e.g., observations in training data 30) belongs to that class. For example, a binary classification model 231 for a class (such as the previously mentioned "not interested" class) can determine whether an email document in training data 30 belongs to that class. The binary classification model 231 is configured to process the data observations and generate a positive or negative output based on whether the observation belongs to the class.

[0039] In some implementations, the binary classification model 231 includes one or more algorithms, processes, software fragments, code, etc., stored in memory 220. The binary classification model 231 includes a multi-layered architecture. In some implementations, each layer may include multiple nodes. Each node may have a corresponding value, for example, between 0 and 1. The values ​​of the multiple nodes may be referred to as the activations of the corresponding layers.

[0040] In some implementations, layers can be connected to each other. For example, adjacent layers can be connected to each other. Nodes in a layer can be connected to nodes in adjacent layers on a one-to-one, one-to-many, or many-to-one basis. In processing operations, the activation of a layer can be determined based on the activation of connected layers (e.g., the previous layer). In some implementations, each connection between two nodes, referred to as an edge, can be assigned a value, for example, between 0 and 1. The values ​​of multiple connections can be referred to as weights, where weights indicate the importance or priority of each connection. For example, in a case where node activation is determined based on the activation of nodes connected to two nodes and different weights are assigned to these two connections, the resulting activation will be more strongly influenced by the node with the higher weight.

[0041] In some implementations, the binary classification model 231 may include an input layer. The input layer may be configured to segment an input text document into multiple sentences. The input layer may be configured to evaluate a measure of the relative importance of multiple sub-features in the text document (e.g., each word or each sentence in the text document). In some implementations, the input layer may use a term frequency-inverse document frequency (TF-IDF) tool. The TF-IDF tool is configured to identify a predefined number of the most frequently used words in the text document and assign a value to each of the identified words based on the determined relative importance of the words.

[0042] Alternatively, in some implementations, the input layer can be configured to preprocess the input image. The input layer can be configured to divide the input image into multiple image regions. The input layer can be configured to use, for example, edge detection tools to evaluate relatively important features in the input image.

[0043] In some implementations, the binary classification model 231 may include an intermediate attention layer. The intermediate attention layer may include multiple nodes corresponding to multiple features in the input data. For example, nodes in the intermediate attention layer may correspond to sentences or words in the input text document. Alternatively, nodes in the intermediate attention layer may correspond to image features in the input image, such as edges. The intermediate attention layer is configured to indicate the relative importance of corresponding features in subsequent layers of the binary classification model 231. The activation of the intermediate attention layer may be referred to as attention weights and represent the relative importance of corresponding features when classifying the input data. In some implementations, the attention weights of the intermediate attention may sum to 1.

[0044] The binary classification model 231 may include a fully connected layer following an intermediate attention layer. The fully connected layer may be referred to as a dense layer. In some implementations, the fully connected layer may include a single node. A node in the fully connected layer may be connected to each node in the preceding layer. In some implementations, the fully connected layer may include, for example, a single output connection to a single output node. The activation of the fully connected layer may represent the output value of the binary classification model 231. The activation of the fully connected layer corresponds to an evaluation of how much input data belongs to that class. The fully connected layer may be configured to generate a sigmoid activation based on the sigmoid function. That is, the activation value of the fully connected layer may be between 0 and 1. The binary output indicating whether the input data belongs to that class may be determined based on the sigmoid activation and a predefined threshold (e.g., 0.5).

[0045] In the initial state of each binary classification model 231, each parameter in the model is set to an initial value. For example, the parameters can be randomized.

[0046] Data processing module 230 is configured to train each binary classification model 231 using training data. For each class, data processing module 230 uses the binary classification model 231 for that class to process the observations in the training data 30 corresponding to that class. Alternatively, in some embodiments, each binary classification model 231 may be configured to generate an output for each observation in the training data 30. Each output may be positive or negative depending on whether the observation belongs to the corresponding class. Data processing module 230 may be configured to compare these outputs with known classes of the observations in the training data 30. Initially, most of the outputs may be incorrect.

[0047] Data processing module 230 is configured to perform an update mechanism based on output comparisons. For example, the update mechanism may include backpropagation. In this way, the parameters of each binary classification model 231 are updated. Repeating the steps of processing observations and updating parameters has the effect of improving the accuracy of each binary classification model 231 over time. This training process can continue for a predefined number of iterations, or until a predefined level of accuracy is reached for some or all of the binary classification models 231.

[0048] According to one implementation, the data processing module 230 is configured to train multiple binary classification models 231 using a first training data corpus. In some examples, the first training data corpus can be manually constructed such that each observation includes highly relevant features corresponding to the class. In some examples, the first training data corpus may include one or more negative observations for a certain class, comprising random data to improve the performance of the binary classification model 231 corresponding to that class.

[0049] After training, the data processing module 230 is configured to extract one or more class recognition features from one or more training data observations. According to one embodiment, the data processing module 230 is configured to extract class recognition features from a second training data corpus different from the first training data corpus. Class recognition features are specific features of the training data 30 that strongly indicate that the observations belong to a particular class. For example, in the context of image classification, a visual feature corresponding to a wheel could be a class recognition feature for a class called "vehicle". In the context of document classification, class recognition features could correspond to words or sentences in multiple training documents.

[0050] For example, in the previous example shown for a document that might be misclassified by a multi-class classification model:

[0051] "I apologize for the long delay in replying. My manager and I reconsidered your offer yesterday. While it seems like a good fit, we currently do not have the budget for it. We will contact you if it becomes a priority in the future. Thank you for your time!"

[0052] It would be beneficial to extract only the parts relevant to the "not interesting" class before adding them to the training data. In this case, the data processing module 230 can extract the following:

[0053] “Although it seems like a good fit, we don’t have the budget for it right now. We’ll contact you if it does become a priority in the future.”

[0054] According to one implementation, a first training data corpus can be manually constructed such that each observation primarily or exclusively includes class-identifying features. The second training data corpus can be real-world data, such as data collected at a later time, which includes class-identifying features interspersed with unrelated additional components as described above.

[0055] The data processing module 230 includes a feature extraction unit 232 configured to extract one or more class-identifying features from training data observations. The feature extraction unit 232 can be configured to extract class-identifying features from a second training data corpus. The feature extraction unit 232 can be configured to determine class-identifying features based on the activation of an intermediate attention layer in a binary classification model 231 for that class. For each class, the feature extraction unit 232 can be configured to select each training data observation identified as belonging to that class. For example, the feature extraction unit 232 can be configured to select each training data observation for which the corresponding binary classification model 231 generates a positive output. For each observation selected in this manner, the feature extraction unit 232 is configured to consider the activation of an intermediate attention layer in the binary classification model 231.

[0056] In some implementations, the second training data corpus may include one or more observations that do not correspond to any of the known classes. Data processing module 230 may be configured to label one or more training data observations that do not belong to any of the known classes. Data processing module 230 may be configured to process the unknown observations of the training data using each of the binary classification models in binary classification model 231. Data processing module 230 may be configured to label observations if each of the binary classification models in binary classification model 231 generates a negative output. Labeling one or more observations in this way can indicate a new data class. In some implementations, data processing module 230 may output one or more labeled observations to suggest the addition of a new class to the user. For example, data processing module 230 may use a connected user interface or a connection to a user device to output one or more suggestions.

[0057] As described above, nodes in the intermediate attention layer of the trained binary classification model 231 are mapped to features in the training data 30. For example, a word or sentence in the training data 30 may result in a higher activation weight for the corresponding node in the intermediate activation layer. By considering the activation of the intermediate attention layer in the binary classification model 231, the feature extraction unit 232 is configured to identify the corresponding features in the training data 30. For example, the feature extraction unit 232 may identify one or more sentences in the selected observations that result in a high attention weight assigned in the intermediate attention layer, which leads to a positive output from the binary classification model 231.

[0058] The feature extraction unit 232 can be configured to extract class recognition features based on whether the attention weights of the intermediate attention layers are higher than a predefined threshold. Alternatively, in some embodiments, the feature extraction unit 232 can be configured to extract a predefined number of class recognition features based on the highest activation in the intermediate attention layers.

[0059] In some embodiments, the feature extraction unit 232 can be configured to verify the extracted class-identifying features. The feature extraction unit 232 can be configured to process each extracted feature using a corresponding binary classification model 231. If the binary classification model 231 generates a positive output for the feature, the feature can be verified. In some embodiments, the feature extraction unit 232 can be configured to concatenate one or more features extracted from the observation data and verify the concatenated features by processing them using the corresponding binary classification model 231.

[0060] Data processing module 230 is configured to send one or more extracted class recognition features to output terminal 240. Output 240 is configured to output improved training data 40 based on the extracted class recognition features. Improved training data 40 can be used to train a multi-class classification model. According to one implementation, improved training data 40 can be used instead of a second training data corpus to train the multi-class classification model. In some examples, a first training data corpus can be used to train the multi-class classification model for the first time. In addition to the first training data corpus, improved training data 40 can also be used to train the multi-class classification model a second time.

[0061] The improved training data 40 includes several observations. Each observation in the improved training data 40 may include one or more class recognition features extracted from the observations of the training data 30. Multiple class recognition features extracted from the observations of the training data 30 may be concatenated to generate observations in the improved training data 40.

[0062] In some implementations, output 240 can be configured to directly output the improved training data 40 to the multi-class classification model. Processing device 200 can be configured to train the multi-class classification model using the improved training data 40, which includes the extracted class recognition features.

[0063] In this way, the processing device 200 is configured to process the input training data 30 and output improved training data 40. The improved training data 40 is generated based on the extracted class recognition. In this way, the relevance of the training data 30 can be improved. Compared to training with unprocessed training data (e.g., a second training data corpus), training a multi-class classification model with the improved training data 40 can provide improved accuracy. The improved training data 40 can also improve the speed and processing efficiency of the training process.

[0064] Figure 3 The architecture of a data processing module 300, including a binary classification model and a feature extraction unit, is shown. According to one embodiment, a binary classification model for document classification is illustrated. The binary classification model is configured to receive input text 50 and generate output text 60. In some embodiments, the input text 50 may correspond to multiple training documents, and the output text may correspond to improved training data. The binary classification model includes a sentence segmenter 310, an embedding unit 320, an attention layer 330, a dot product unit 340, a final dense layer 350, a sigmoid activation unit 360, an extraction unit 370, and a validation unit 380.

[0065] Sentence segmenter 310 is configured to receive input text 50 and segment the input text 50 into multiple sentences.

[0066] Embedding unit 320 is configured to receive multiple sentences from sentence segmenter 310. Embedding unit 320 is configured to convert discrete data (e.g., categorical data) into continuous data. That is, embedding unit 320 can acquire discrete sentences and generate numerical / vector values ​​representing those sentences. Embedding unit 320 is a layer comprising multiple nodes corresponding to multiple embeddings, wherein the activation of the layer corresponds to the numerical value representing each sentence. Embedding unit 320 can map each sentence to a unique vector, making each sentence recognizable and processed by subsequent layers. In some implementations, embedding unit 320 can perform embeddings at the sentence level using one or more pre-trained models.

[0067] Attention layer 330 includes multiple nodes corresponding to multiple embeddings. That is, the nodes of attention layer 330 correspond to sentences in the input text 50. Attention layer 330 is configured to indicate the relative importance of sentences when classifying the input text 50. Activation of attention layer 330 is referred to as attention weights 331. In some embodiments, the sum of attention weights 331 may be equal to 1. Attention layer 330 is configured to output multiple attention weights 331 and provide the attention weights 331 to dot product unit 340.

[0068] Dot product unit 340 is configured to perform a dot product operation between multiple embeddings and multiple attention weights 331. Dot product unit 340 is a layer comprising multiple nodes corresponding to the multiple embeddings. Each node is connected to a corresponding node in embedding unit 320 and a corresponding node in attention layer 330. Activation of dot product unit 340 is generated as a dot product between the corresponding embedding and attention weight 331. That is, dot product unit 340 is configured to apply attention weights 331 to the corresponding sentence in vector form. Dot product unit 340 is configured to output multiple embeddings weighted by the corresponding attention weights 331. The operation of dot product unit 340 can produce a single vector representing the entire input text 50.

[0069] The final dense layer 350 can be referred to as a fully connected layer. As described above, the final dense layer 350 comprises a single node. The nodes of the final dense layer 350 are connected to each node of the dot product unit 340. The activation of the final dense layer 350 corresponds to an evaluation of how much of the input text 50 belongs to that class.

[0070] The sigmoid activation unit 360 is configured to map the output of the final dense layer 350 onto a sigmoid function. The value of the mapped output is in the range of 0 to 1. The value of the mapped output represents the expected value for the input text 50 belonging to that class. The sigmoid activation unit 360 can also be configured to apply a threshold to generate a prediction 361 for whether the input text 50 belongs to that class. For example, if the value of the mapped output exceeds the threshold (e.g., exceeds 0.5), a positive prediction can be generated.

[0071] Extraction unit 370 is configured to extract text from input text 50. Extraction unit 370 can be configured to extract sentences from input text 50 if sigmoid activation unit 360 generates a prediction 361 indicating that input text 50 belongs to that class. For example, extraction unit 370 can be configured to extract text when sigmoid activation unit 360 generates a positive output.

[0072] Extraction unit 370 is configured to extract one or more sentences from input text 50 based on activation weights 331 generated by attention layer 330. As described above, nodes in attention layer 330 are mapped to sentences in input text 50. For example, a sentence in input text 50 may result in a higher activation weight for the corresponding node in attention layer 330. By considering the activation of attention layer 330, extraction unit 370 is configured to identify the corresponding sentence in input text 50. Extraction unit 370 can be configured to extract sentences based on whether the activation weights of attention layer 330 are higher than a predefined threshold.

[0073] Verification unit 380 is configured to send each extracted sentence to the binary classification model for processing. Verification unit 380 sends the extracted sentences to embedding unit 320. If sigmoid activation unit 360 generates a positive output for the sentence, then the sentence can be verified. Extraction unit 370 is configured to send the verified sentences as output text 60.

[0074] Figure 4 A block diagram of a classification system 1 according to an embodiment of the present invention is shown. The classification system 1 includes a document classifier 100 and a training data improvement module.

[0075] Document classifier 100 is basically as follows: Figure 1 The described, and training data improvement module includes, essentially, as about Figure 2The processing apparatus 200 is described above. Initially, a first training data corpus 31 can be used to train the document classifier 100. The first training data corpus 31 can be manually constructed such that each observation is highly correlated with its corresponding class. In some embodiments, the first training data corpus 31 can initially be used to train a binary classification model of the processing apparatus 200.

[0076] After training, the document classifier 100 is configured to receive input text 10 and assign it to one of a plurality of document classes 20. In some implementations, an update mechanism can improve the classification system as the classification is confirmed to be correct or incorrect. For example, user / customer feedback or supervision can confirm whether a document classification is correct or incorrect. The correct class can be determined, and new training data observations including text documents and the correct class can be formed. A second training data corpus 32 can be collected in this manner. In some implementations, each new training data observation can be processed by the training data improvement module 200 to extract one or more class recognition features and generate improved training data.

[0077] In operation, training data is input into the training data improvement module 200. For example, a second training data corpus 32 can be input into the training data improvement module 200. As described above, the training data may include multiple text documents, each associated with a certain class. The training data improvement module 200 is configured to output improved training data, which is provided to the document classifier 100. The improved training data can be used to train or retrain the document classifier 100.

[0078] The document classifier 100 can be trained or retrained essentially as described above, wherein the improved training data replaces the second training data corpus 32. The document classifier 100 can be retrained using a combination of the first training data corpus 31 and improved training data based on the second training data corpus 32. Training a multi-classification model using the improved training data can provide improved accuracy compared to training with the unprocessed second training data corpus 32. The improved training data can also improve the speed and processing efficiency of the training process.

[0079] The attached diagram Figure 5 A flowchart illustrating the classification method according to the implementation scheme is shown. The method begins at step S01.

[0080] In step S02, the processor receives training data corresponding to multiple classes. The training data can be used to train a multi-class classification model. The multi-class classification model can be a text classification model, and the training data can include multiple text documents. Each of the multiple classes can correspond to the intended purpose of a text document. In some examples, the training data can include a first training data corpus and a second training data corpus. The first and second corpora can be received at different times.

[0081] In step S03, the processor trains a binary classification model for each of the multiple classes. Each binary classification model is configured to determine whether a training data observation belongs to that class. The binary classification model may include an intermediate attention layer followed by a fully connected layer. For each class, training the binary classification model for that class involves using the binary classification model for that class to process multiple observations in the training data corresponding to that class. In some examples, a first training data corpus may be used to train the binary classification model. The first training data corpus can be manually constructed such that each training data observation is highly correlated with its corresponding class.

[0082] In step S04, one or more class recognition features are extracted from the training data observations based on the activation of the intermediate attention layer in the binary classification model. In some implementations, a binary classification model is used to process multiple observations from a second training data corpus. The second training data corpus may include collected real-world data, such as multiple real-world text documents. Each class recognition feature can be extracted based on whether the activation weights of the intermediate layer are higher than a predefined threshold.

[0083] Extracting class recognition features may involve dividing each training data text document into multiple sentences. These class recognition features may correspond to sentences extracted from multiple training documents in a second training data corpus. The processor may label one or more training data observations that do not belong to any of the known classes. Labeling one or more observations in this way can indicate a new data class.

[0084] In step S05, the processor trains a multi-class classification model using class recognition features extracted for each of the multiple classes in the second training data corpus. The processor may output improved training data for training the multi-class classification model, which includes the class recognition features. In some embodiments, the improved training data can be used instead of the second training data corpus to train or retrain the multi-class classification model. A combination of the first training data corpus and improved training data based on the second training data corpus can be used to train the multi-class classification model.

[0085] In this way, the processing device is configured to process a second training data corpus and output improved training data. Improved training data is generated based on class recognition extracted from the second training data corpus. This improves the relevance of the training data. Training a multi-class classification model using the improved training data provides improved accuracy compared to training with the unprocessed second corpus. The improved training data also increases the speed and processing efficiency of the training process.

[0086] The method ends at step S06.

[0087] These and other objects and features of the invention will be apparent in this disclosure, which includes the foregoing and ongoing written description.

[0088] The foregoing description details certain embodiments of the invention. However, it will be understood that the invention can be practiced in many ways, regardless of the detail presented herein. As also stated above, it should be noted that the use of particular terms in describing certain features or aspects of the invention should not be construed as implying that such terms are redefined herein as limited to any specific characteristic of the invention's features or aspects associated with that term.

[0089] This invention is not limited to the specific embodiments shown in the accompanying drawings and described in detail above. Those skilled in the art will recognize that other arrangements are conceived. The invention encompasses every possible combination of various features of each disclosed embodiment. Where useful for a particular application, one or more of the elements described herein with respect to the various embodiments may be implemented in a more dispersed or integrated manner than explicitly described, or even removed or rendered inoperable in certain circumstances. While the invention has been described with reference to specific illustrative embodiments, modifications and variations of the invention may be constructed without departing from the spirit and scope of the invention as set forth in the following claims.

Claims

1. A computer-implemented method for training a multi-class classification model (230), comprising: The processor receives training data (30) corresponding to multiple classes; For each of the plurality of classes (20): The processor trains a binary classification model (231), which is configured to determine whether the training data (30) observations belong to the class (20); For each training data (30) observation identified as belonging to the class (20), one or more class identification features are extracted from the training data (30) observation based on the activation of the intermediate attention layer in the binary classification model (231), wherein each class identification feature is extracted based on whether the activation weight of the intermediate layer is higher than a predefined threshold, wherein extracting the class identification feature includes using the corresponding binary classification model (231) to verify each feature; The training process continues for a predefined number of iterations until a predefined level of accuracy is achieved for some or all of the binary classification models (231); and The processor trains a multi-class classification model (230) using the class recognition features extracted for each of the plurality of classes (20). The binary classification model comprises multiple layers, and each layer includes multiple nodes. Adjacent layers in the plurality of layers are interconnected, and the plurality of nodes in the adjacent layers are also interconnected. Each connection between two nodes in the plurality of nodes has a value between 0 and 1, which is used as a weight to indicate the importance or priority of each connection.

2. The method according to claim 1, wherein the multi-class classification model (230) is a text classification model, and the training data includes multiple text documents.

3. The method of claim 2, wherein each of the plurality of classes (20) corresponds to the intended purpose of the text document.

4. The method of claim 2, wherein the class recognition features correspond to sentences extracted from a plurality of training documents.

5. The method of claim 2, wherein extracting the class recognition features includes dividing each of the training data text documents into multiple sentences.

6. The method of claim 5, wherein extracting the class identification features includes evaluating a relative importance measure for each sentence in the plurality of text documents.

7. The method according to claim 1, wherein the binary classification model (231) includes an intermediate attention layer followed by a fully connected layer.

8. The method of claim 1 further includes one or more training data (30) observations that do not belong to any one of a plurality of known classes (20).

9. A processing apparatus comprising a processor configured to perform a method comprising the following steps: The processor receives training data (30) corresponding to a plurality of classes (20); and for each of the plurality of classes (20): The processor trains a binary classification model (231), which is configured to determine whether the training data (30) observations belong to the class (20); For each training data (30) observation identified as belonging to the class (20), one or more class identification features are extracted from the training data (30) observation based on the activation of the intermediate attention layer in the binary classification model (231), wherein each class identification feature is extracted based on whether the activation weight of the intermediate layer is higher than a predefined threshold, wherein extracting the class identification feature includes using the corresponding binary classification model (231) to verify each feature; The training process continues for a predefined number of iterations until a predefined level of accuracy is achieved for some or all of the binary classification models (231); and The processor trains a multi-class classification model (230) using the class recognition features extracted for each of the plurality of classes (20). in, The binary classification model comprises multiple layers, and each of the multiple layers comprises multiple nodes. Adjacent layers in the plurality of layers are interconnected, and the plurality of nodes in the adjacent layers are also interconnected. Each connection between two nodes in the plurality of nodes has a value between 0 and 1, which is used as a weight to indicate the importance or priority of each connection.

10. The processing apparatus according to claim 9, wherein the multi-class classification model (230) is a text classification model, and the training data (30) includes a plurality of text documents.

11. The processing apparatus of claim 10, wherein each of the plurality of classes (20) corresponds to the intended purpose of the text document.

12. The processing apparatus of claim 10, wherein the class recognition features correspond to sentences extracted from a plurality of training documents.

13. The processing apparatus of claim 10, wherein extracting the class recognition features comprises dividing each of the training data text documents into a plurality of sentences, and wherein extracting the class recognition features comprises evaluating a relative importance metric for each sentence in the plurality of text documents.

14. A non-transitory computer-readable medium configured to store instructions, which, when executed by a processor, cause the processor to perform a method comprising the following steps: The processor receives training data (30) corresponding to multiple classes (20); And, for each of the plurality of classes (20): The processor trains a binary classification model (231), which is configured to determine whether the training data (30) observations belong to the class (20); For each training data (30) observation identified as belonging to the class (20), one or more class identification features are extracted from the training data (30) observation based on the activation of the intermediate attention layer in the binary classification model (231), wherein each class identification feature is extracted based on whether the activation weight of the intermediate layer is higher than a predefined threshold, wherein extracting the class identification feature includes using the corresponding binary classification model (231) to verify each feature; The training process continues for a predefined number of iterations until a predefined level of accuracy is achieved for some or all of the binary classification models (231); and The processor trains a multi-class classification model (230) using the class recognition features extracted for each of the plurality of classes (20). in, The binary classification model comprises multiple layers, and each of the multiple layers comprises multiple nodes. Adjacent layers in the plurality of layers are interconnected, and the plurality of nodes in the adjacent layers are also interconnected. Each connection between two nodes in the plurality of nodes has a value between 0 and 1, which is used as a weight to indicate the importance or priority of each connection.

15. The non-transitory computer-readable medium of claim 14, wherein the binary classification model (231) includes an intermediate attention layer followed by a fully connected layer.

16. The non-transitory computer-readable medium of claim 14, wherein the method further comprises one or more training data (30) observations that do not belong to any one of a plurality of known classes (20).

Images