Deep learning-based automatic ontology extraction for detecting new domain knowledge

Abstract

The present invention relates to deep learning-based automatic ontology extraction for detecting new domain knowledge. A system processes unstructured data to identify multiple subsets of text in a text collection within the unstructured data. For a subset from the multiple subsets, a probability is determined based on the subset's position in the text collection, the part of speech (POS) of each word in the subset, and the POS of one or more words on the left and right sides of the subset, where the number of the one or more words is selected based on the length of the text collection. The system generates a feature vector for the subset, the feature vector including the probability and additional features of the subset; and uses a classifier to classify the subset into one of multiple classes based on the feature vector of the subset, the multiple classes representing the ontology of the domain of knowledge.

Description

Technical Field

[0001] introduction

[0002] The information provided in this section is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors, to the extent it is described in this section, and in all aspects of that description that might not otherwise be considered prior art at the time of filing, is not admitted, either explicitly or implicitly, to be prior art to the present disclosure.

[0003] The present disclosure relates generally to natural language processing, and more particularly to automatic ontology extraction based on deep learning (where the deep learning is based on transfer learning) to detect new domain knowledge. Background Art

[0004] Word2vec (W2V) is a natural language processing technique that uses a neural network model to learn word associations from large text corpora. The trained model can detect synonyms or suggest additional words for parts of sentences. W2V represents each distinct word as a vector. The vectors are chosen so that mathematical functions can indicate the level of semantic similarity between the words represented by these vectors. W2V accepts a large text corpus as input and produces a multidimensional vector space. Each unique word in the corpus is assigned a corresponding vector in the vector space. The word vectors are positioned in the vector space so that words that share a common context in the corpus are close to each other in the vector space. Summary of the Invention

[0005] A system includes a processor and a memory storing instructions, wherein the instructions, when executed by the processor, configure the processor to: process unstructured data to identify multiple subsets of text in a text collection in the unstructured data. The instructions configure the processor to: determine a probability for a subset from the multiple subsets based on the position of the subset in the text collection, the part of speech (POS) of each word in the subset, and the POS of one or more words on the left and right sides of the subset, wherein the number of the one or more words is selected based on the length of the text collection. The instructions configure the processor to: generate a feature vector for the subset, the feature vector including the probability and additional features of the subset. The instructions configure the processor to: use a classifier to classify the subset into one of a plurality of classes based on the feature vector of the subset, the plurality of classes representing an ontology of a domain of knowledge.

[0006] In other features, the processor is configured to: train a model using a first set of manually labeled feature vectors generated from unstructured data; and automatically label a second set of feature vectors generated from the unstructured data using the trained model, wherein the second set of feature vectors is one or more orders of magnitude larger than the first set of feature vectors.

[0007] Among other features, the classifier includes a transfer learning-based classifier. The processor is configured to: train the transfer learning-based classifier using the larger set of automatically labeled second vectors to train a lower level layer of the transfer learning-based classifier; and then retrain the top level layer of the transfer learning-based classifier using the set of manually labeled first feature vectors having higher quality but a smaller number than the larger set of automatically labeled second vectors to improve an F1 score of the transfer learning-based classifier to be used to classify feature vectors representing additional unstructured data into the plurality of classes.

[0008] In another feature, the model comprises a random forest based model, and wherein the classifier comprises a transfer learning based deep neural network.

[0009] Among other features, the probability comprises a Bayesian probability of finding the subset at a predetermined position in a text set of a predetermined length in the unstructured data given the POS of the one or more words on the left and right hand sides of the subset.

[0010] Additional features of the feature vector include, among other features: the starting and stopping positions of the subset in the text set, the length of the text set, and the word2vec of each word in the subset.

[0011] In other features, the processor is configured to: determine the length of the text set; and assign a POS to each word in the text set. In other features, the processing of the unstructured data includes cleaning the unstructured data by one or more of: removing extra spaces, adding spaces between words, splitting incorrectly connected words, joining incorrectly split words, and correcting spelling.

[0012] In other features, the unstructured data includes vehicle service reports received from dealers, and the plurality of classes include part, symptom, action, and other non-technical data.

[0013] In other features, the processor is configured to: process the data portion classified by the classifier; and assess a quality of at least one of a manufactured vehicle, a serviced vehicle, and a part used in the vehicle based on the processing of the data portion classified by the classifier.

[0014] In yet other features, a method includes processing unstructured data to identify multiple subsets of text in a text collection within the unstructured data. The method includes determining, for a subset from the multiple subsets, a probability based on the position of the subset in the text collection, a part of speech (POS) of each word in the subset, and the POS of one or more words on the left and right sides of the subset, the number of the one or more words being selected based on the length of the text collection. The method includes generating a feature vector for the subset, the feature vector including the probability and additional features of the subset. The method includes using a classifier to classify the subset into one of a plurality of classes based on the feature vector of the subset, the plurality of classes representing an ontology of a domain of knowledge.

[0015] In other features, the method further includes: training a model using a first set of manually labeled feature vectors generated from the unstructured data; and automatically labeling a second set of feature vectors generated from the unstructured data using the trained model, wherein the second set of feature vectors is one or more orders of magnitude larger than the first set of feature vectors.

[0016] In other features, the method further includes: training the classifier using the automatically labeled second set of feature vectors to initially train a lower level layer of the classifier; and then training the top level layer of the classifier using the manually labeled first set of feature vectors having a higher quality than the automatically labeled second set of feature vectors to classify feature vectors representing the additional unstructured data into the multiple classes.

[0017] In another feature, the model comprises a random forest based model, and wherein the classifier comprises a deep neural network.

[0018] Among other features, the probability comprises a Bayesian probability of finding the subset at a predetermined position in a text set of a predetermined length in the unstructured data given the POS of the one or more words on the left and right hand sides of the subset.

[0019] Additional features of the feature vector include, among other features: the starting and stopping positions of the subset in the text set, the length of the text set, and the word2vec of each word in the subset.

[0020] In other features, the method further comprises: determining a length of the text corpus; and assigning a POS to each word in the text corpus.

[0021] Among other features, the processing of the unstructured data includes cleaning the unstructured data by one or more of removing extra spaces, adding spaces between words, splitting incorrectly joined words, joining incorrectly split words, and correcting spelling.

[0022] In other features, the unstructured data includes vehicle service reports received from a dealer, and the plurality of classes include parts, symptoms, actions, and other non-technical data.

[0023] In other features, the method further includes processing the portion of data classified by the classifier; and assessing a quality of at least one of manufacturing the vehicle, servicing the vehicle, and a part used in the vehicle based on the processing of the portion of data classified by the classifier.

[0024] The present invention also provides the following technical solutions:

[0025] 1. A system comprising:

[0026] processor; and

[0027] a memory storing instructions that, when executed by the processor, configure the processor to:

[0028] processing unstructured data to identify a plurality of subsets of text in a set of text in the unstructured data;

[0029] determining, for a subset from the plurality of subsets, a probability based on a position of the subset in the corpus of text, a part of speech (POS) of each word in the subset, and POS of one or more words on a left-hand side and a right-hand side of the subset, the number of the one or more words being selected based on a length of the corpus of text;

[0030] generating a feature vector for the subset, the feature vector comprising the probability and additional features of the subset; and

[0031] The subset is classified into one of a plurality of classes based on the feature vector of the subset using a classifier, the plurality of classes representing an ontology of a domain of knowledge.

[0032] 2. The system according to technical solution 1, wherein the processor is configured to:

[0033] training a model using a first set of manually labeled feature vectors generated from the unstructured data; and

[0034] automatically labeling a second set of feature vectors generated from the unstructured data using the trained model,

[0035] The second feature vector set is one or more orders of magnitude larger than the first feature vector set.

[0036] 3. A system according to technical solution 2, wherein the classifier includes a transfer learning-based classifier, and the processor is configured to: train the transfer learning-based classifier using a larger automatically labeled second vector set to train a lower level layer of the transfer learning-based classifier; and then retrain the top level layer of the transfer learning-based classifier using the manually labeled first feature vector set with higher quality but smaller number than the larger automatically labeled second vector set to improve the F1 score of the transfer learning-based classifier, which is to be used to classify feature vectors representing additional unstructured data into the multiple classes.

[0037] 4. A system according to technical solution 2, wherein the model includes a random forest-based model, and wherein the classifier includes a deep neural network based on transfer learning.

[0038] 5. A system according to technical solution 1, wherein the probability includes a Bayesian probability of finding the subset at a predetermined position in a text set of a predetermined length in the unstructured data, given the POS of the one or more words on the left and right sides of the subset.

[0039] 6. A system according to technical solution 1, wherein the additional features of the feature vector include: the start and stop positions of the subset in the text set, the length of the text set, and the word2vec of each word in the subset.

[0040] 7. The system according to technical solution 1, wherein the processor is configured to:

[0041] determining the length of the text set; and

[0042] A POS is assigned to each word in the corpus.

[0043] 8. A system according to technical solution 1, wherein the processing of the unstructured data includes cleaning the unstructured data by one or more of the following: removing redundant spaces, adding spaces between words, splitting incorrectly connected words, connecting incorrectly split words, and correcting spelling.

[0044] 9. The system according to claim 1, wherein:

[0045] The unstructured data includes vehicle service reports received from dealers; and

[0046] The multiple classes include parts, symptoms, actions, and other non-technical data.

[0047] 10. The system according to technical solution 9, wherein the processor is configured to:

[0048] processing the portion of data classified by the classifier; and

[0049] The quality of at least one of a manufactured vehicle, a service vehicle, and a part used in a vehicle is assessed based on the processing of the portion of the data classified by the classifier.

[0050] 11. A method comprising:

[0051] processing unstructured data to identify a plurality of subsets of text in a set of text in the unstructured data;

[0052] determining, for a subset from the plurality of subsets, a probability based on a position of the subset in the corpus of text, a part of speech (POS) of each word in the subset, and POS of one or more words on a left-hand side and a right-hand side of the subset, the number of the one or more words being selected based on a length of the corpus of text;

[0053] generating a feature vector for the subset, the feature vector comprising the probability and additional features of the subset; and

[0054] The subset is classified into one of a plurality of classes based on the feature vector of the subset using a classifier, the plurality of classes representing an ontology of a domain of knowledge.

[0055] 12. The method according to technical solution 11 further comprises:

[0056] training a model using a first set of manually labeled feature vectors generated from the unstructured data; and

[0057] automatically labeling a second set of feature vectors generated from the unstructured data using the trained model,

[0058] The second feature vector set is one or more orders of magnitude larger than the first feature vector set.

[0059] 13. The method according to technical solution 12 further includes: using the automatically labeled second vector set to train the classifier to initially train the low-level layer of the classifier; and then, using the manually labeled first feature vector set with higher quality than the automatically labeled second feature vector set to train the high-level layer of the classifier to classify the feature vectors representing additional unstructured data into the multiple classes.

[0060] 14. The method according to technical solution 12, wherein the model includes a random forest-based model, and wherein the classifier includes a deep neural network.

[0061] 15. A method according to technical solution 11, wherein the probability includes a Bayesian probability of finding the subset at a predetermined position in a text set of a predetermined length in the unstructured data in the case of the POS of the one or more words on the left-hand side and the right-hand side of the subset.

[0062] 16. The method according to technical solution 11, wherein the additional features of the feature vector include: the start and stop positions of the subset in the text set, the length of the text set, and the word2vec of each word in the subset.

[0063] 17. The method according to technical solution 11 further comprises:

[0064] determining the length of the text set; and

[0065] A POS is assigned to each word in the corpus.

[0066] 18. A method according to technical solution 11, wherein the processing of the unstructured data includes cleaning the unstructured data by one or more of the following: removing redundant spaces, adding spaces between words, splitting incorrectly connected words, connecting incorrectly split words, and correcting spelling.

[0067] 19. The method according to claim 11, wherein:

[0068] The unstructured data includes vehicle service reports received from dealers; and

[0069] The multiple classes include parts, symptoms, actions, and other non-technical data.

[0070] 20. The method according to technical solution 19 further comprises:

[0071] processing the portion of data classified by the classifier; and

[0072] The quality of at least one of a manufactured vehicle, a service vehicle, and a part used in a vehicle is assessed based on the processing of the portion of the data classified by the classifier.

[0073] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims, and accompanying drawings.The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The present disclosure will become more fully understood from the detailed description and accompanying drawings, in which:

[0075] Figure 1 A system for performing deep learning-based (deep learning-based transfer learning) ontology extraction according to the present disclosure is shown;

[0076] Figure 2 Shown for use Figure 1 A flowchart of a method for performing deep learning-based ontology extraction using a system;

[0077] Figure 3 Shown for use Figure 1 A flowchart of a method for generating a vector by a system;

[0078] Figure 4 A flow chart showing a method for using Figure 1 systems to train machine learning models to automatically generate large amounts of labeled training data to train classifiers (such as deep learning neural networks); and

[0079] Figure 5 The steps performed by the encoding module 106 during the encoding process to encode the n-gram "intake rocker arm" in the verbatim text are shown.

[0080] Among the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0081] A novel encoding scheme is proposed for training deep learning models based on transfer learning. The encoding includes various elements such as part-of-speech (POS) based on a distributed Bayesian probability model, Word2vec (w2v), the position of technical terms within the document, the length of the verbatim text, word frequency, the inverse document frequency of the term, and POS. In particular, the POS based on the distributed Bayesian probability model is constructed based on POS associated with key technical terms and their position within verbatim text of a specific length, capturing adaptive contextual information about the technical terms. The Bayesian probability of the POS associated with the technical term at a specific position in the verbatim text of a specific length is converted into a vector. This vector serves as one of the elements of the novel encoding scheme.

[0082] An ontology is a set of concepts, their instances, and categories in a subject area or domain that shows their properties and the relationships between them. Ontology learning (which is an unsolved research problem, particularly in the automotive industry) involves identifying and extracting concepts captured in text data and classifying them into specific classes. In industries such as vehicle manufacturing, such concepts can be used to perform a variety of tasks (e.g., emerging safety issue identification, quality control, supply chain, and diagnosis and prognosis). The present disclosure provides a natural language processing (NLP) system that includes data encoding and a deep learning model based on transfer learning. The deep learning model uses contextual information, positional features (e.g., start and end positions, relative positions, and length of word-by-word text), syntactic information, and a novel distribution probability model based on parts of speech to extract and classify new concepts from unstructured text data (e.g., extracting ngrams from vehicle warranty / service data and classifying these ngrams into their specific classes, such as parts, symptoms, and actions). This model significantly improves the accuracy of term extraction and outperforms traditional machine learning models.

[0083] Automated ontology extraction models and extracted concepts provide semantically-enabled domain models for diverse text algorithms in the manufacturing industry. Newly discovered parts, symptoms, and actions provide improved coverage for constructing new fault representations for fault detection. This system overcomes the limitations of manually curated ontologies, which are time-consuming and impractical due to the scale of the data (e.g., warranty / service data collected from vehicle dealerships worldwide).

[0084] In the system of the present disclosure, data collected from various sources is encoded by combining context, position, and syntactic information, which allows the data to be transformed into a hyperdimensional vector space. Although the word2vec model provides a static context vector built from surface-level words (i.e., actual words or phrases, such as engine control module), the word2vec model fails to consider the different positions of such words in different verbatim texts of different lengths when constructing its surrounding information. More importantly, the w2vec model constructs context vectors by using actual words (e.g., engine control module), but if the words present in the data have not been observed in historical data, the w2vec model ends up constructing an empty vector. This greatly limits the ability to build vectorized representations from data related to new technology spaces (e.g., electric vehicles and autonomous vehicles). The system also employs a random forest model to automatically and accurately perform industrial-scale data labeling.

[0085] Typically, the word2vec (w2v) model provides corpus-level vectorization for each word or phrase. However, the w2v model provides the same vector for a phrase regardless of its occurrence at different positions in different verbatim texts throughout the corpus. Moreover, the w2v model relies heavily on surface phrases (e.g., vehicles) to construct vectors. In contrast, the vectorization in the system of the present disclosure uses the part of speech (POS) assigned to the phrase, and then calculates its probability based on the part of speech of the surrounding phrases along with their positions in different verbatim texts of different lengths. The system also extends the w2v model by transforming it into POS2vec. This vector is combined with w2v as one of the encoders for the phrase.

[0086] Typically, supervised classifiers in industrial settings exhibit limited applicability due to the limited availability of labeled data. In the system disclosed herein, a random forest model is trained on a small amount of high-quality, manually labeled data. The trained model is then used to correct labeling errors and then employed to automatically label large amounts of data. The trained model is not only accurate but also saves a significant amount of time and effort that would otherwise be spent manually labeling large amounts of data from numerous sources in an industrial setting.

[0087] Specifically, the disclosed system extracts and classifies business-critical technical phrases from unstructured warranty text data using data encoding and deep learning models, such as parts, symptoms, and actions (i.e., types of repairs or services) in the context of vehicle warranty. The newly extracted terms can be used to construct new fault representations, which in turn provide higher coverage for fault detection.

[0088] The system generally improves the field of natural language processing technology, and in particular solves problems associated with ontology extraction by overcoming the above limitations of the word2vec model. The system extends the word2vec model by accommodating contextual information at the corpus level. The system constructs dynamic context vectors by utilizing the part-of-speech and position information of surrounding terms. The dynamic context vector provides more comprehensive context information than w2v. The distribution probability model provides adaptive context, where the part-of-speech of a phrase based on its position in different word-by-word texts is used together with the part-of-speech of the surrounding terms. The distribution probability model dynamically estimates probabilities across the entire corpus, which takes into account changes in the context of a particular term. These and other features of the system of the present disclosure are described in detail below.

[0089] Many vehicle manufacturers manufacture and sell vehicles internationally. Whenever a vehicle receives service, dealers around the world prepare service reports. These reports typically consist of short, unstructured text, including customer complaints, fault analyses performed, and parts identified as defective and replaced. Manufacturers typically maintain a centralized database of these reports. The content and format of the text in these reports can vary widely. For example, the text is not only free-form (i.e., non-standardized), but can also be detailed or abbreviated, potentially containing local terminology and abbreviations. The volume of data generated is enormous.

[0090] If extracted from these reports, certain information (such as parts, symptoms, and actions) can be used by different groups for different purposes (for example, identifying emerging safety issues, quality control, supply chain, and diagnosis and prognosis). Given the scale and variability of the data, manually curating a large number of these reports and extracting parts, symptoms, and actions from them is impractical. Instead, a deep learning model based on transfer learning can be trained to automatically extract this information from these reports. This model can learn ontologies and therefore adapt to new and emerging technologies (for example, electric and autonomous vehicles).

[0091] Figure 1 This article provides an overview of a system 100 for performing transfer learning-based deep learning ontology extraction according to the present disclosure. Although this system is described throughout this disclosure in the context of warranty data (which is used as an illustrative example only), the system has broad applicability in various other engineering and manufacturing industries (e.g., aerospace, heavy equipment manufacturing, mining). One or more components of system 100 can be distributed in the cloud.

[0092] System 100 includes an input module 102, a cleaning module 104, an encoding module 106, a labeling module 108, and a classifier module 110. System 100 is described in detail below. Briefly, input module 102 receives service reports from dealerships. Cleaning module 104 cleans the data in the service reports. Encoding module 106 encodes the cleaned data and generates vectors. Labeling module 108, used during the training phase, labels these vectors using an ML model to train a transfer learning deep neural network in classifier module 110 to classify the vectors. Classifier module 110 includes a transfer learning deep neural network trained using the labeled vectors, which classifies the vectors generated by encoding module 106 during normal operation.

[0093] For example only, the input to system 100 includes numerous records of service reports received from dealers around the world. Input module 102 receives these records containing service reports. For example, these service reports may be received via the Internet or via an intranet maintained by a manufacturing company. If the service reports are not in English and text format, they are scanned and translated (if necessary).

[0094] Different dealerships may use different terminology to describe similar issues. For example, one report may state, "Vehicle stalled, ECM replaced," while another report may describe the same issue as, "Vehicle suddenly stopped, control module malfunction," etc. In addition to these differences, the text in the reports may also contain other inconsistencies, such as spelling errors, abbreviations, missing spaces between words, excessive spaces between words, missing punctuation, etc. The cleaning module 104 cleans the text data in the reports to remove these inconsistencies.

[0095] In computational linguistics, an n-gram is a continuous sequence of n items from a given text or speech sample. These items can be phonemes, syllables, letters, words, or phrases, depending on the application. The encoding module 106 includes a parsing module 120, a part-of-speech (POS) module 122, a probability module 124, and a vector module 126. The encoding module 106 is described in further detail below. In short, the parsing module 120 collects n-grams from a text or speech corpus (such as the cleaned text data output by the cleaning module 104). The POS module 122 assigns POS to phrases in the n-gram. The probability module 124 generates a distribution probability for the n-gram based on the POS, as explained in detail below. For each n-gram, the vector module 126 generates a multidimensional vector that includes the distribution probability and various other features describing the n-gram. For example, these features include n-gram start, n-gram end, start index, end index, length of word-by-word text, distribution probability (explained below), w2v, POS to vector, length of word-by-word text, term frequency, inverse document frequency, term frequency inverse document frequency, etc.

[0096] The labeling module 108 labels the encoded data (i.e., vectors). It uses a hybrid approach to label the encoded data. A small number of manually labeled (and therefore high-quality) n-grams and their vectors are used to train a machine learning model, such as a random forest model. Specifically, a label is assigned to each vector so that each word in the n-gram text can be classified as a part, symptom, action, or other text (i.e., not a part, symptom, or action) when it is detected as associated with a label. The trained random forest model is tested for accuracy and any labeling errors are corrected.

[0097] Thereafter, the labeling module 108 uses the trained random forest model to automatically label a large number of vectors received from the encoding module 106. The large number of labeled vectors from the labeling module 108 is then used to train the transfer learning deep neural network in the classifier module 110, as described below. The hybrid approach used by the labeling module 108 to generate a large amount of labeled training data (i.e., initially using a small amount of high-quality labeled data to train the ML model, and then using the trained model to automatically label a large amount of training data) is a major improvement in the field of deep learning technology, because large training data sets are typically manually labeled to ensure the accuracy of deep learning. In contrast, due to the hybrid approach, the labeling module 108 not only automatically generates a large amount of labeled training data, but the labeled training data is also highly accurate. This greatly improves the performance of deep learning models in actual industrial settings, enabling them to be trained on a large scale to further improve model accuracy.

[0098] The transfer learning deep neural network used in the classifier module 110 is then trained. This classifier module uses the vectors and the associated tags generated by the tagging module 108 as input to operate as a four-class classifier. During normal operation, the tagging module 108 is no longer used. When the input module 102 receives new reports, the cleaning module 104 cleans the text in these reports. The encoding module 106 encodes the cleaned text and generates vectors for the n-grams in the cleaned text. These vectors are then directly input to train the transfer learning deep neural network in the classifier module 110, which classifies (i.e., tags) the n-grams into one of four classes: part, symptom, action, or other text. A trained model with a satisfactory F1 score (see definition below) is saved in the classifier module 110 for use in learning new ontologies or augmenting existing ontologies from raw input data (e.g., service reports). The trained model can be leveraged for various purposes (e.g., quality control, safety analysis, etc.). New ontologies can be learned from new raw data. Existing ontologies can be augmented with ontologies learned from raw data. The F1 score (also known as the F score or F measure) is a measure of test accuracy. The F1 score is defined as the weighted harmonic mean of test precision and recall.

[0099] Encoding module 106 will now be described in further detail. Notably, encoding module 106 utilizes not only the size of n-grams or their positions within the verbatim text (such as their start and end positions (e.g., as determined by parsing module 120)), but also contextual and syntactic information about the n-grams within and across verbatim texts. Furthermore, in w2v, if a term (e.g., "vehicle") appears in one position in one verbatim text and in another position in another verbatim text, the vector for that term is the same (i.e., static). That is, in w2v, regardless of where the term appears in the verbatim text, the term has the same vector. Therefore, while the vectors in w2v capture the context surrounding the term, they do not have adaptive context. Furthermore, in w2v, the vectors rely on the surface terms (i.e., the written words) in the verbatim text. Therefore, in w2v, when the input vocabulary or terms change due to technological change (e.g., from fossil fuel vehicles to alternative energy vehicles) or are historically unobserved, the vectors will be empty.

[0100] In contrast, encoding module 106 additionally utilizes the part of speech associated with a term (e.g., determined by POS module 122). Encoding module 106 converts n-grams into their corresponding part of speech. When a part of speech is associated with a particular part, symptom, or action of a certain size (e.g., unigram, bigram, trigram, or tetragram) within a certain length of verbatim text (e.g., determined by parsing module 120), encoding module 106 determines the surrounding words and their part of speech. Thus, encoding module 106 does not utilize the surface terms (i.e., the words themselves), but rather their part of speech, which is based on the terms' association with the part, symptom, or action and their location within the certain length of verbatim text.

[0101] Based on this information, encoding module 106 constructs a Bayesian probability model from the entire corpus. Distribution probabilities are estimated to build a dynamic, adaptive context (e.g., performed by probability module 124). For example, if a part of speech is a proper noun (NN) associated with a part (e.g., vehicle) that appears in a short verbatim text, then one term on the left and one term on the right of the proper noun, along with the corresponding parts of speech of these neighboring terms, is selected from all short verbatim texts, where the NN appears at the position where the part (e.g., vehicle) POS is observed, and the distribution probabilities are estimated. If the verbatim text is of medium length, two terms are selected on either side; if the verbatim text is of long length, three terms are selected on either side.

[0102] The following are used Figure 1An example of a processing scheme for generating conditional probabilities of n-grams in a cleaned verbatim text by the system 100. For example, assume that the encoding module 106 receives the following cleaned verbatim text from the cleaning module 104: "The vehicle was turned off, the intake rocker arm was disassembled, and the valve spring retainer was replaced." Figure 5 1 and 2 are used to illustrate and describe an example of steps performed by the encoding module 106 during the encoding process to encode the n-gram "intake rocker arm" in the verbatim text above.

[0103] The encoding module 106 parses the raw verbatim text (e.g., using the parsing module 120 ), associates a part of speech (POS) with each portion of the raw verbatim text (i.e., the POS module 122 tags n-gram tokens in the verbatim text with the part of speech), and extracts POS assignments for the verbatim text (e.g., using the probability module 124 ).

[0104] Examples of POS abbreviations used to represent various parts of speech are shown in Table 1 below. Figure 1 The system 100 can use any of these POS to generate conditional probabilities of n-grams in the cleaned verbatim text using the processing scheme described below.

[0105] serial number mark describe 1. CC Coordinating conjunctions 2. CD Cardinal Numbers 3. DT Determiners 4. EX Existential Sentences 5. FW foreign words 6. IN Preposition or subordinating conjunction 7. JJ adjective 8. JJR comparative adjectives 9. JJS Superlative adjective 10. LS List item markers 11. MD Modal auxiliary verbs 12. NN Common noun, singular form 13. NNS Common nouns, plural form 14. NNP Proper noun, singular form 15. NNPS proper noun, plural 16. PDT Predeterminer 17. POS possessive ending words 18. PRP Personal pronouns 19. PRP$ possessive pronouns 20. RB adverb 21. RBR comparative adverbs 22. RBS Adverb superlative 23. RP Particle 24. SYM symbol 25. TO to as a preposition or infinitive 26. UH interjection 27. VB Basic form of verb 28. VBD past tense of verbs 29. VBG Gerunds and present participles 30. VBN past participle 31. VBP Verbs not in the third person singular 32. VBZ third person singular verb 33. WDT Wh-determiner 34. WP Wh-pronouns 35. WP$ possessive pronouns 36. WRB interrogative pronouns

[0106] Table 1

[0107] Note that w2v constructs a vector for each word in the verbatim text. In contrast, the encoding module 106 uses the POS assignments in the verbatim text and constructs vectors based on the POS assignments in the verbatim text as follows. This feature provides another improvement in the field of natural language processing technology, as explained below.

[0108] For example, as shown in the example above, encoding module 106 can construct a vector (e.g., using vector module 126) for a selected token or phrase "intake rocker arm" (which is an n-gram with n=3) as follows. The part of speech (POS) associated with the token "intake rocker arm" is "NN NN NN." For the selected phrase and the selected verbatim text, encoding module 106 determines the length of the verbatim text and the position at which the selected n-gram token occurs in the verbatim text. Position is important because if an n-gram token appears at different positions in the verbatim text, it is possible that not only its own POS may change, but also the POS of its surrounding terms in the verbatim text may change. Therefore, because the vector for the phrase is generated based on the position-dependent POS of the phrase, the vector has adaptive context, in contrast to w2v, which is the same regardless of position and is based only on the words themselves, not their POS.

[0109] Encoding module 106 extracts the POS of a phrase based on its position in the verbatim text and also determines the length of the verbatim text. Depending on whether the verbatim text is short, medium, or long (a standard that can be predetermined), encoding module 106 selects the POS of one, two, or three terms on each side (left and right) of the phrase. If the verbatim text is short, the part of speech of one term on the left and one term on the right are selected. If the verbatim text is medium, two terms on either side are selected; if the verbatim text is long, three terms on either side are selected.

[0110] The encoding module 106 then estimates the Bayesian conditional probability of the POS of the selected phrase (e.g., using the probability module 124). For example, if the length of the verbatim text is short, the probability module 124 traverses each short verbatim text in the corpus and determines whether the POS of the selected phrase "NN NN NN" is reported at the same position in each short verbatim text, with one selected POS on the left and one on the right (in this example, VBN and VBD). If the length of the verbatim text is medium, the probability module 124 traverses each medium verbatim text in the corpus and determines whether the POS of the selected phrase "NN NN NN" is reported at the same position in each medium verbatim text, with two selected POS on the left and two on the right (in this example, VBN VBZ on the first and second positions on the left, and VBD RB on the first and second positions on the right). If the length of the verbatim text is long, the probability module 124 traverses each long verbatim text in the corpus and determines whether the POS "NN NN NN" of the selected phrase is reported at the same position in each long verbatim text, with three selected POS on the left and three on the right (in the example, VBN VBZ NNP on the first, second, and third positions on the left; and VBD RB VBD on the first, second, and third positions on the right).

[0111] Therefore, the probability module 124 determines the conditional probability of the POS of a selected phrase not at the verbatim level but at the corpus level (i.e., the probability that the POS of the phrase appears at a particular position in a verbatim text of a particular length). The probability is additionally conditioned on the particular POS having one, two, or three terms on each side of the phrase, depending on the length of the verbatim text (short, medium, or long).

[0112] Therefore, the conditional probability is the probability that the POS of an n-gram appears at a specific position in a verbatim text of a specific length (given that there are one, two, or three POS on the left and right sides of the n-gram). If the length of the verbatim text is short, the probability is the probability that the POS of a phrase (NN NN NN in the example) appears at a specific position in a short verbatim text (given one POS each on the left, right, and top (VBN and VBD in the example)). If the length of the verbatim text is medium, the conditional probability is the probability that the POS of a phrase (NN NN NN in the example) appears at a specific position in a medium verbatim text (given two POS each on the left, right, and top (in the example, VBN VBZ on the first and second positions on the left, and VBD RB on the first and second positions on the right)). If the length of the verbatim text is long, the conditional probability is the probability that the POS of a phrase (NN NN NN in the example) appears at a specific position in a long verbatim text (given three POS each on the left, right, and top (in the example, VBN VBZ NNP on the first, second, and third positions on the left; and VBD RB VBD on the first, second, and third positions on the right).

[0113] An example of a conditional probability is a Bayesian probability. The encoding module 106 (e.g., the vector module 126) then encodes the conditional probability of the phrase into a vector of the phrase (i.e., included as a feature in the vector). The conditional probability feature of the vector generated by the encoding module 106 distinguishes the vector based on POS and position from the conventional word-based w2v and is a significant improvement in the field of natural language processing technology. An example of calculating the Bayesian probability in the case of a POS (such as NN) in a short verbatim text (where a term on the left has a POS VBN) is given by equations (1) and (2). Calculations for other cases can be implemented based on the same idea.

[0114] The tagging module 108 will now be described in further detail. For each verbatim text received from the cleaning module 104, the encoding module 106 breaks the verbatim text into n-grams (e.g., 1-grams, 2-grams, 3-grams, and 4-grams). For each n-gram, the encoding module 106 generates a vector, as described above. To train the random forest model used by the tagging module 108, a limited number of n-grams and associated vectors are initially manually labeled (e.g., whether the n-gram is a part, symptom, action, or non-technical information).

[0115] Once a random forest model has been trained on a limited number of manually labeled vectors to identify specific vectors as specific tokens (e.g., whether the vector represents a part, symptom, or action), the random forest model is tested on unlabeled n-grams using an additional validation dataset containing the vectors. Errors can occur (e.g., the random forest model may mislabel a vector) because the manual labeling used to train the random forest model can be inconsistent due to subjective judgment (e.g., one labeler may label an n-gram as a part when it is not, etc.). These labeling errors are corrected by validating the vector labeling performed by the random forest model until the random forest model can label vectors with a predetermined F1 accuracy score, which is empirically determined for the application domain (e.g., F1 = 0.91).

[0116] Thereafter, the labeling module 108 uses the trained random forest model to label the large amount of cleaned data encoded by the encoding module 106. The encoding module 106 generates vectors for the large amount of cleaned data, and the labeling module 108 labels these vectors using the trained random forest model. The vectors generated by the encoding module 106 and the corresponding labels generated by the labeling module 108 are then used to train the transfer learning-based deep neural network used in the classifier module 110. It is noteworthy that the labels used to train the transfer learning-based deep neural network are not manually generated as is conventional, but are instead automatically generated by the trained random forest model, which is a significant improvement in the field of natural language processing technology.

[0117] Once the transfer learning-based deep neural network in classifier module 110 is trained, system 100 is ready for normal operation (i.e., production use). In use, input module 102 receives new service reports. Cleansing module 104 cleans these new reports. Encoding module 106 encodes the new, cleansed data and generates new vectors that include the conditional probabilities and other features described above. Labeling module 108 is no longer used. The vectors generated by encoding module 106 are input to the trained neural network in classifier module 110. The transfer learning-based deep neural network classifies (i.e., labels) these vectors.

[0118] In deep neural network classifiers based on transfer learning, a two-stage approach is used. Initially, the entire network (i.e., all layers of the network) is trained using a large labeled dataset generated by a random forest model. During the transfer learning phase, the lower layers of the network are frozen (i.e., the algorithm does not update the weights of the neurons associated with the lower layers), and then only the top layers of the network are trained using a high-quality but low-volume manually labeled dataset. This approach is particularly important when a large amount of data is available (but in some cases not directly relevant to the application domain) to train the model but has a relatively low F1 score. High-quality data can then be used to further improve the F1 score by retraining the top layers. Transfer learning methods have significantly improved F1 scores on real-world data.

[0119] Figure 2-4 A flow chart of a method performed by the system 100 is shown. Figure 2 The overall method performed by the system 100 is shown. Figure 3 The method performed by the encoding module 106 to generate the vector is shown in further detail. Figure 4 A hybrid approach is further detailed that is used by the labeling module 108 to automatically generate a large amount of training data for training the classifier. In the following description, the term control is used to refer to an operation performed by one or more modules of the system 100. Since these methods and operations of the modules of the system 100 have been described in detail above, the following description is kept brief for the sake of brevity.

[0120] Figure 2 A method 200 for deep learning-based ontology extraction according to the present disclosure is shown. At 202, the control receives raw data (e.g., a vehicle's service report) from various sources. At 204, the control cleans the raw data. At 206, the control generates a vector (in the form of a lexical vector) based on the parts of speech and conditional probabilities of n-grams in the cleaned data. Figure 3 Detailed description of the method).

[0121] At 208, some of these vectors are manually labeled. At 210, the control trains an ML model (e.g., a random forest model) using the manually labeled vectors. At 212, Figure 4 After testing and correcting the trained ML model using the method shown in , the control uses the trained ML model to automatically label a large number of vectors (generated based on a large amount of cleaned raw data).

[0122] At 214, the control trains a classifier (e.g., a deep learning neural network) using the large number of vectors automatically labeled by the trained ML model. At 216, the control uses the trained classifier to classify vectors from new raw data (after cleaning and generating vectors at 204, 206).

[0123] Figure 3 A method 240 for generating a vector at 206 in method 200 is shown. At 242, the control parses the cleaned verbatim text into n-grams. At 244, the control determines the position and POS of the n-gram, as well as the length of the verbatim text. At 246, the control generates a conditional probability for the n-gram based on the position and POS of the n-gram and the POS of surrounding terms that depend on the length of the verbatim text. At 248, the control determines additional features (e.g., the start and end position of the n-gram, the start and end index of the n-gram, w2v, etc.). At 250, the control generates a vector for each n-gram including its conditional probability and additional features.

[0124] Figure 4 A method 260 for labeling a large amount of training data using a hybrid approach according to the present disclosure is shown. At 262, control trains an ML model (e.g., a random forest model) using a limited number of vectors and corresponding manual labels. At 264, control labels the vectors using the trained ML model. At 266, control determines whether the ML model labels the vectors with a predetermined accuracy (i.e., whether the ML model is trained). If the ML model does not label the vectors with the predetermined accuracy, control returns to 262 (i.e., continues training the ML model). If the ML model labels the vectors with the predetermined accuracy (i.e., if the ML model is trained), control proceeds to 268. At 268, control uses the trained ML model to automatically label a large amount of training data to train a classifier (e.g., a deep learning neural network).

[0125] The foregoing description is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses. The broad teachings of the present disclosure can be implemented in various forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be so limited, as other modifications will become apparent upon studying the drawings, the description, and the following claims. It should be understood that one or more steps within the method can be performed in a different order (or simultaneously) without changing the principles of the present disclosure. Further, although each of the embodiments is described above as having certain features, any one or more of those features described with respect to any embodiment of the present disclosure may be implemented and / or combined with the features of any of the other embodiments, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the arrangement of one or more embodiments with each other is still within the scope of the present disclosure.

[0126] Spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "immediately adjacent," "on top of," "above," "below," and "disposed." Unless explicitly described as "directly," when describing a relationship between a first element and a second element in the above disclosure, the relationship can be a direct relationship with no other intervening elements between the first element and the second element, but can also be an indirect relationship with one or more intervening elements (either spatially or functionally) between the first element and the second element. As used herein, the phrase at least one of A, B, and C should be interpreted to mean a logical (A OR B OR C) using a non-exclusive logical OR and should not be interpreted to mean "at least one of A, at least one of B, and at least one of C."

[0127] In the accompanying drawings, the direction of the arrows, as indicated by arrows, generally demonstrates the flow of information (such as data or instructions) of interest. For example, when component A and component B exchange various types of information, but the information transmitted from component A to component B is relevant to the illustration, an arrow may point from component A to component B. This unidirectional arrow does not imply that no other information is transmitted from component B to component A. Furthermore, for information transmitted from component A to component B, component B may send a request for the information or an acknowledgment of receipt to component A.

[0128] In this application (including the definitions below), the term "module" or the term "controller" may be replaced with the term "circuit". The term "module" may refer to, be part of, or include: an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0129] A module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of the present disclosure may be distributed among multiple modules connected via the interface circuits. For example, multiple modules may allow for load balancing. In further examples, a server (also referred to as a remote or cloud) module may perform certain functions on behalf of a client module.

[0130] The term code, as used above, may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term shared processor circuit encompasses a single processor circuit that executes some or all code from multiple modules. The term group processor circuit encompasses a processor circuit that is combined with additional processor circuits to execute some or all code from one or more modules. Reference to multiple processor circuits encompasses multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term shared memory circuit encompasses a single memory circuit that stores some or all code from multiple modules. The term group memory circuit encompasses a memory circuit that is combined with additional memory to store some or all code from one or more modules.

[0131] The term memory circuit is a subset of the term computer-readable medium. As used herein, the term computer-readable medium does not encompass transient electrical or electromagnetic signals propagating through a medium (such as a carrier wave); thus, the term computer-readable medium may be considered tangible and non-transitory. Non-limiting examples of non-transitory, tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0132] The apparatus and method described in this application can be implemented in part or in whole by a special-purpose computer, which is created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The functional blocks, flow chart components, and other elements described above serve as software specifications, which can be converted into computer programs through routine work by a skilled technician or programmer.

[0133] A computer program includes processor-executable instructions stored on at least one non-transitory, tangible computer-readable medium. A computer program may also include or rely on stored data. A computer program may include a basic input / output system (BIOS) that interacts with the hardware of a special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, and the like.

[0134] A computer program may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated by a compiler from source code, (iv) source code executed by an interpreter, (v) source code compiled and executed by a just-in-time compiler, etc. By way of example only, source code may be written using a syntax according to languages ​​including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language Version 5), Ada, ASP (Active Server Pages), PHP (Hypertext Preprocessor Language), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®.

Claims

1. A system comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the processor to: processing unstructured data to identify a plurality of subsets of text in a set of text in the unstructured data; determining, for a subset from the plurality of subsets, a probability based on a position of the subset within the corpus of text, a part of speech of each word in the subset, and parts of speech of one or more words on a left-hand side and a right-hand side of the subset, the number of the one or more words being selected based on a length of the corpus of text; generating a feature vector for the subset, the feature vector comprising the probability and additional features of the subset; as well as classifying the subset into one of a plurality of classes based on the feature vector of the subset using a classifier, the plurality of classes representing an ontology of a domain of knowledge; Wherein, the processor is configured to: training a model using a first set of manually labeled feature vectors generated from the unstructured data; and automatically labeling a second set of feature vectors generated from the unstructured data using the trained model, wherein the second feature vector set is one or more orders of magnitude larger than the first feature vector set; wherein the classifier comprises a transfer learning-based classifier, and the processor is configured to: train the transfer learning-based classifier using a larger set of automatically labeled second feature vectors to train a lower level layer of the transfer learning-based classifier; and then retrain a top level layer of the transfer learning-based classifier using the manually labeled first feature vector set having a higher quality but a smaller number than the larger set of automatically labeled second feature vectors to improve an F1 score of the transfer learning-based classifier, the F1 score to be used to classify feature vectors representing additional unstructured data into the plurality of classes; The transfer learning-based classifier uses contextual information, positional features, syntactic information, and a novel distribution probability model based on part of speech to extract and classify new concepts from unstructured text data.

2. The system according to claim 1, wherein: The model comprises a random forest based model, and wherein the classifier comprises a deep neural network based on transfer learning.

3. The system according to claim 1, wherein: The probability comprises a Bayesian probability of finding the subset at a predetermined position in a text set of a predetermined length in the unstructured data given the parts of speech of the one or more words on the left-hand side and the right-hand side of the subset.

4. The system according to claim 1, wherein: The additional features of the feature vector include: the start and stop positions of the subset in the text set, the length of the text set, and word2vec of each word in the subset.

5. The system according to claim 1, wherein The processor is configured to: determining the length of the text set; and Assign a part of speech to each word in the corpus of text.

6. The system according to claim 1, wherein: The processing of the unstructured data includes cleaning the unstructured data by one or more of: removing extra spaces, adding spaces between words, splitting incorrectly joined words, joining incorrectly split words, and correcting spelling.

7. The system of claim 1 , wherein: The unstructured data includes vehicle service reports received from dealers; and The multiple classes include parts, symptoms, actions, and other non-technical data.

8. The system according to claim 7, wherein: The processor is configured to: processing the portion of data classified by the classifier; and The quality of at least one of a manufactured vehicle, a service vehicle, and a part used in a vehicle is assessed based on the processing of the portion of the data classified by the classifier.

9. A method comprising: processing unstructured data to identify a plurality of subsets of text in a set of text in the unstructured data; determining, for a subset from the plurality of subsets, a probability based on a position of the subset within the corpus of text, a part of speech of each word in the subset, and parts of speech of one or more words on a left-hand side and a right-hand side of the subset, the number of the one or more words being selected based on a length of the corpus of text; generating a feature vector for the subset, the feature vector comprising the probability and additional features of the subset; as well as classifying the subset into one of a plurality of classes based on the feature vector of the subset using a classifier, the plurality of classes representing an ontology of a domain of knowledge; The method further comprises: training a model using a first set of manually labeled feature vectors generated from the unstructured data; and automatically labeling a second set of feature vectors generated from the unstructured data using the trained model; wherein the second feature vector set is one or more orders of magnitude larger than the first feature vector set; The method further includes: training the classifier using the automatically labeled second set of feature vectors to initially train a low-level layer of the classifier; and then training a high-level layer of the classifier using the manually labeled first set of feature vectors having a higher quality than the automatically labeled second set of feature vectors to classify feature vectors representing additional unstructured data into the plurality of classes; The classifier uses contextual information, positional features, syntactic information, and a novel distribution probability model based on part of speech to extract and classify new concepts from unstructured text data.

10. The method according to claim 9, wherein: The model comprises a random forest based model, and wherein the classifier comprises a deep neural network.

11. The method according to claim 9, wherein: The probability comprises a Bayesian probability of finding the subset at a predetermined position in a text set of a predetermined length in the unstructured data given the parts of speech of the one or more words on the left-hand side and the right-hand side of the subset.

12. The method according to claim 9, wherein The additional features of the feature vector include: the start and stop positions of the subset in the text set, the length of the text set, and word2vec of each word in the subset.

13. The method of claim 9, further comprising: determining the length of the text set; as well as Assign a part of speech to each word in the corpus of text.

14. The method according to claim 9, wherein The processing of the unstructured data includes cleaning the unstructured data by one or more of: removing extra spaces, adding spaces between words, splitting incorrectly joined words, joining incorrectly split words, and correcting spelling.

15. The method of claim 9, wherein: The unstructured data includes vehicle service reports received from dealers; and The multiple classes include parts, symptoms, actions, and other non-technical data.

16. The method according to claim 15, further comprising: processing the portion of data classified by the classifier; as well as The quality of at least one of a manufactured vehicle, a service vehicle, and a part used in a vehicle is assessed based on the processing of the portion of the data classified by the classifier.

Images