System and method for building an evolving ontology from user-generated content

Abstract

A method and system for constructing an evolving ontology database. The method includes: receiving a plurality of data entries; calculating a semantic similarity score between any two data entries; clustering the data entries into a plurality of current topics based on the semantic similarity score; selecting a new concept from the current topic by comparing the current topic with a plurality of previous topics prepared using previous data entries; and updating the evolving ontology database using the new concept. The semantic similarity score between any two data entries is calculated by the following formula, #imgabs0##imgabs1#where s i is the weight of the feature source, f j is the feature similarity between two data items, w j Yes j The weight of , j, k and n are positive integers.

Description

[0001] Cross-references

[0002] Several references, including patents, patent applications, and various publications, are cited and discussed in the description of the present disclosure. Citation and / or discussion of such references is provided solely to clarify the description of the present disclosure and is not an admission that any such reference is "prior art" to the disclosure described herein. All references cited and discussed in this specification are incorporated herein by reference in their entirety to the same extent as if each reference were individually incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to building evolving ontologies from complex and dynamic data, and more particularly, to systems and methods for building evolving ontologies from user-generated content on e-commerce websites. Background Art

[0004] The background description provided herein is for the purpose of presenting the content of the present disclosure in general. To the extent described in this background section, the work of the inventors currently named, as well as certain aspects of the specification that may not be considered as prior art at the time of application, are not explicitly or implicitly admitted that the present disclosure is prior art.

[0005] Computer-mediated communication is becoming the most convenient and important way to share and exchange information in today's society. People can submit their feedback directly to specific businesses or manufacturers and conduct online research before making many routine consumer purchase decisions by reading other users' reviews. However, it is difficult to effectively utilize the large amount and diversity of user-generated content on the web by only checking a single review score or multiple positive or negative reviews.

[0006] Therefore, there exists an unresolved need in the art to address the aforementioned deficiencies and inadequacies. Summary of the invention

[0007] In some aspects, the present disclosure relates to a method for constructing an evolving ontology database. In some embodiments, the method includes:

[0008] The computing device receives a plurality of data entries;

[0009] The computing device calculates a semantic similarity score between any two data entries based on feature sources and feature similarities of the data entries;

[0010] The computing device clusters the data items into a plurality of current topics based on the semantic similarity scores;

[0011] The computing device selects new concepts from the current topic by comparing the current topic with a plurality of previous topics prepared using previous data entries; and

[0012] The computing device updates the evolving ontology database with the new concepts.

[0013] In some embodiments, the semantic score between any two data items is calculated by the following formula:

[0014] where s i is the weight of the feature source, f j is one of the feature similarities between two data items, w j Yes j The weight of , j, k and n are positive integers.

[0015] In some embodiments, the data entries are user-generated feedback, and the step of calculating the semantic similarity score includes: predicting a sentiment similarity value by a sentiment analyzer, the sentiment similarity value indicating the similarity between the two data entries regarding positive feedback, negative feedback, neutral feedback, very negative feedback, and Internet abuse; predicting a text similarity value by a similarity calculator, the text similarity value indicating the similarity between the semantics of text extracted from the two data entries; predicting a syntactic similarity value by a neutral language parser, the syntactic similarity value indicating the syntactic complexity of the text of the two data entries.

[0016] In some embodiments, the step of clustering the data entries further includes: calculating a semantic similarity score of the two data entries using the sentiment similarity value, the textual similarity value, and the syntactic similarity value.

[0017] In some embodiments, the step of selecting new concepts from the current topic includes: retrieving the current topic and previous topics; identifying nearly duplicate topics from the current topic and previous topics; removing nearly duplicate topics from the current topic to obtain non-duplicate topics; comparing non-duplicate topics with concepts in an ontology database to obtain novel concept candidates, wherein the novel concept candidates are non-duplicate topics with low similarity to any concept in the ontology database; and verifying the novel concept candidates according to instructions of an ontology database administrator to obtain new concepts.

[0018] In some embodiments, the step of updating the evolving ontology database includes: detecting the most relevant parent concept by comparing at least one verified concept with concepts in the ontology; calculating the similarity between at least one verified concept and a sibling concept to obtain the most similar sibling concept, wherein the sibling concept is a child concept of the most relevant parent concept; proposing an ontology adjustment based on the most relevant parent concept and the most similar sibling concept; and updating the ontology using the best adjustment among the proposed ontology adjustments.

[0019] In some embodiments, the proposed adjustment includes an insertion adjustment, and in the insertion adjustment, the new concept is defined as a child node of the most relevant parent concept.

[0020] In some embodiments, the proposed adjustment includes a boosting adjustment, and in the boosting adjustment, the new concept is defined as a sibling node of the most relevant parent concept.

[0021] In some embodiments, the proposed adjustment includes a shift adjustment, and in the shift adjustment, the new concept is defined as a child node of the most similar sibling concept.

[0022] In some embodiments, the proposed adjustment includes a merge adjustment, and in the merge adjustment, the new topic is combined with the most similar sibling concept to form a combined concept, the combined concept is defined as a child node of the most relevant parent concept, and the new topic and the most similar sibling concept are defined as child nodes of the combined concept.

[0023] In certain embodiments, each concept in the ontology database is defined by a classification model, which includes a logistic recursive model and a gradient boosting classifier.

[0024] In some embodiments, the method further includes adjusting the classification model according to the updated ontology.

[0025] In some embodiments, the method further comprises: cleaning and tokenizing the data entries prior to the step of calculating the semantic similarity score.

[0026] In some aspects, the present disclosure relates to a system for constructing an evolving ontology database. In some embodiments, the system includes a computing device. The computing device has a processor and a storage device storing computer executable code. When the computer executable code is executed at the processor, the computer executable code is configured to perform the above method.

[0027] In some aspects, the present disclosure relates to a non-transitory computer-readable medium storing computer executable code. When the computer executable code is executed at a processor of a computing device, the computer executable code is configured to perform the above method.

[0028] These and other aspects of the disclosure will become apparent from the following description of the preferred embodiments taken in conjunction with the accompanying drawings and captions thereof, although variations and modifications may be made without departing from the spirit and scope of the novel concepts of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings illustrate one or more embodiments of the present disclosure and, together with the written description, serve to explain the principles of the present disclosure. Throughout the drawings, the same reference numerals are used wherever possible to refer to the same or similar elements of the embodiments.

[0030] Figure 1 An evolving ontology system according to some embodiments of the present disclosure is schematically illustrated.

[0031] Figure 2A An emerging topic detector according to certain embodiments of the present disclosure is schematically illustrated.

[0032] Figure 2B A new proof-of-concept according to certain embodiments of the present disclosure is schematically illustrated.

[0033] Figure 2C A body adjustment module according to certain embodiments of the present disclosure is schematically illustrated.

[0034] Figure 2D The figure schematically illustrates an entity update module according to some embodiments of the present disclosure.

[0035] Figure 3A The current body (part) according to some embodiments of the present disclosure is schematically shown.

[0036] Figure 3B The lifting operation of the adjustment body according to some embodiments of the present disclosure is schematically illustrated.

[0037] Figure 3C The diagram schematically illustrates a displacement operation of an adjustment body according to certain embodiments of the present disclosure.

[0038] Figure 3D The merging operation of adjusting entities according to some embodiments of the present disclosure is schematically illustrated.

[0039] Figure 4 A flowchart of establishing and updating an evolving ontology from user-generated content according to some embodiments of the present disclosure is schematically shown.

[0040] Figure 5 A method for detecting emerging topics according to certain embodiments of the present disclosure is schematically illustrated.

[0041] Figure 6 A method for verifying a new subject to obtain a new concept according to some embodiments of the present disclosure is schematically illustrated.

[0042] Figure 7 A method for proposing ontology adjustments based on validated new concepts and updating the ontology using the best adjustments according to certain embodiments of the present disclosure is schematically illustrated. DETAILED DESCRIPTION

[0043] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, as many modifications and variations therein are apparent to those skilled in the art. Various embodiments of the present disclosure are now described in detail. With reference to the accompanying drawings, the same numerals represent the same components throughout the views. As used in this specification and throughout the claims that follow, the meanings of "one", "a kind" and "" include plural forms unless the context clearly indicates otherwise. Moreover, as used in this specification and throughout the claims that follow, the meaning of "in..." includes "in..." and "on..." unless the context clearly indicates otherwise. Moreover, for the convenience of the reader, titles or subtitles may be used in the specification, which will not affect the scope of the present disclosure. In addition, some of the terms used in this specification are defined in more detail below.

[0044] In this specification, in the context of the present disclosure and in the specific context in which each term is used, the terms used in this specification generally have their ordinary meaning in the art. Certain terms used to describe the present disclosure are discussed below or elsewhere in the specification to provide practitioners with additional guidance on the description of the present disclosure. It should be understood that the same thing can be said in more than one way. Therefore, alternative language and synonyms can be used for any one or more terms discussed herein, and there is no special meaning as to whether the terms are elaborated or discussed herein. Synonyms for certain terms are provided. The enumeration of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any term discussed herein, is illustrative only and in no way limits the scope and meaning of the present disclosure or any exemplary term. Similarly, the present disclosure is not limited to the various embodiments given in this specification.

[0045] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the same meaning as in the context of the relevant art and the present disclosure, and should not be interpreted as ideal or overly formal meanings unless explicitly defined as such herein.

[0046] As used herein, "about", "approximately", "substantially" or "approximately" shall generally mean within 20 percent, preferably within 10 percent, and more preferably within 5 percent of a given value or range. The numerical values ​​given herein are approximate, meaning that if not explicitly stated, the term "about", "approximately", "substantially" or "approximately" can be inferred.

[0047] As used herein, "plurality" means two or more.

[0048] As used herein, the terms "containing", "including", "carrying", "having", "including", "involving", etc. should be understood as open-ended, ie, meaning including but not limited to.

[0049] As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A or B or C), using a non-exclusive logical OR. It should be appreciated that one or more steps within a method can be performed in a different order (or concurrently) without altering the principles of the present disclosure. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0050] As used herein, the term "module" may refer to a portion of or include: an application specific integrated circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or combined) that executes code; other suitable hardware components that provide the above functions; or a combination of some or all of the above, such as in a single chip system. The term "module" may include a memory (shared, dedicated, or combined) that stores code executed by a processor.

[0051] As used herein, the term "code" may include software, firmware and / or microcode, and may refer to a program, routine, function, class and / or object. As used above, the term "shared" means that a single (shared) processor may be used to execute some or all of the code from multiple modules. In addition, some or all of the code from multiple modules may be stored by a single (shared) memory. As used above, the term "combined" means that a group of processors may be used to execute some or all of the code from a single module. In addition, a group of memories may be used to store some or all of the code from a single module.

[0052] As used herein, the term "interface" generally refers to a communication tool or device used to perform data communication between components at an interaction point between components. Generally, interfaces can be applicable in both hardware and software, and can be unidirectional or bidirectional interfaces. Examples of physical hardware interfaces can include: electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. The components that communicate with the interface can be, for example, multiple components or peripheral devices of a computer system.

[0053] The present disclosure relates to computer systems. As shown in the figure, computer components may include physical hardware components and virtual software components, where the physical hardware components are shown as solid boxes and the virtual software components are shown as dashed boxes. It should be understood by those skilled in the art that, unless otherwise indicated, these computer components may be implemented in the form of, but not limited to, software, firmware, or hardware components or a combination thereof.

[0054] The devices, systems, and methods described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer programs may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage, and optical storage.

[0055] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the present disclosure are shown. However, the present disclosure may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will be comprehensive and complete and will fully convey the scope of the present disclosure to those skilled in the art.

[0056] In some aspects, in order to utilize the large amount and diverse user-generated content on the Internet, the present disclosure provides an ontology structure for such data sets, thereby improving the efficiency of many downstream semantic analysis tasks. The challenge of building an ontology on such data can stem from two characteristics of user-generated content. First, domain knowledge is very limited or difficult to obtain. It is inefficient and useless to manually summarize concepts and semantic relationships from such a large amount of data. Second, the basic structure of such an ontology evolves over time. Emerging topics continue to occur. Therefore, new concepts need to be created, and relevant ontology structures are needed to be adjusted and optimized accordingly.

[0057] In some embodiments, the hierarchical structure is manually organized and a hierarchical machine learning classifier is trained for semantic analysis. However, this approach relies heavily on human effort to understand the content and label the training data, and the approach cannot automatically track changes in the data.

[0058] In some embodiments, the data stream is segmented into time segments, semantic analysis is applied to each segment, and then emerging topics are identified within the segments. Although this approach can detect hot events and novel concepts, it cannot capture the hierarchical structure between concepts.

[0059] In some embodiments, entities are extracted from the data set and linked to a well-established general knowledge graph. Further semantic analysis and inference can be performed based on the knowledge graph. The limitation of this method is that the general knowledge graph is stable and therefore cannot keep up with the rapid changes in the semantic structure of user-generated data. Moreover, it is costly because the general knowledge graph needs to be maintained by a large number of experts. In addition, this method cannot discover concepts that do not exist in the existing knowledge graph.

[0060] To overcome these challenges and shortcomings, in some aspects, the present disclosure provides a semantic analysis framework to detect emerging topics from large-scale, evolving data streams, and further provides a set of methods for validating new concepts and optimizing related ontology structures. In some embodiments, the present disclosure provides a system that uses natural language processing, active learning, semi-supervised learning techniques, and principled human-computer interaction.

[0061] In certain embodiments, the system consists of two parts: 1) a real-time semantic analysis pipeline that automatically mines and detects emerging topics and new concepts from user-generated data; and 2) a management interface that displays the analysis results and facilitates system administrators to search, verify, and adjust the ontology structure.

[0062] The semantic analysis pipeline consists of three modules:

[0063] 1. The semantic analyzer clusters items in the data stream that belong to the same topic. In the first step of the entire pipeline, we use the natural language parser to extract the factual part of the text, calculate the semantic similarity score between two items based on word embedding and sentence embedding, and predict the sentiment polarity of a given text.

[0064] 2. The temporal analysis module is responsible for predicting whether the discovered emerging topics are about known topics or new concepts.

[0065] 3. An ontology optimization module is designed to maintain and adjust the semantic relationships between concepts, and the training process of the machine learning model is started based on the analysis results and verification.

[0066] The management interface provides the following uses:

[0067] 1. The visualization interface demonstrates the detected topics, along with relevant statistics, generated summaries, sentiment distributions, and suggested semantic relations with existing concepts.

[0068] 2. The management interface verifies the validity of detected concepts, edits semantic relationships within the ontology structure, controls the training process of machine models, and supervises model prediction results.

[0069] Figure 1 The following schematically shows an evolutionary ontology system according to some embodiments of the present disclosure. Figure 1As shown, the system 100 includes a computing device 110. In some embodiments, the computing device 110 can be a server computer, a cluster, a cloud computer, a general-purpose computer, a mobile device, a tablet computer, or a dedicated computer, which constructs an ontology based on historical data or / and current data, and updates the ontology based on new input of data, so that the ontology model automatically evolves with the updated data under minimal supervision. In some embodiments, the computing device 110 can communicate with other computing devices or services to obtain user-generated data from those computing devices to update the ontology, and provide the ontology to those computing devices. In some embodiments, communication is performed via a network, which can be a wired or wireless network and can be in various forms, such as a public network and a private network.

[0070] like Figure 1 As shown, computing device 110 may include, but is not limited to, processor 112, memory 114, and storage device 116. In some embodiments, computing device 110 may include other hardware components and software components (not shown) to perform its corresponding tasks. Examples of these hardware and software components may include, but are not limited to, other required memories, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices.

[0071] Processor 112 may be a central processing unit (CPU) configured to control the operation of computing device 110. Processor 112 may execute an operating system (OS) or other applications of computing device 110. In some embodiments, computing device 110 may have more than one CPU as a processor, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. Memory 114 may be a volatile memory, such as a random access memory (RAM), for storing data and information during operation of computing device 110. In some embodiments, memory 114 may be a volatile memory array. In some embodiments, computing device 110 may run on more than one memory 114. Storage device 116 is a non-volatile data storage medium for storing an OS (not shown) and other applications of computing device 110. Examples of storage device 116 may include non-volatile memory, such as flash memory, memory card, USB drive, hard drive, floppy disk, optical drive, solid state drive (SSD), or any other type of data storage device. In some embodiments, storage device 116 may be local storage, remote storage, or cloud storage. In some embodiments, computing device 110 may have multiple storage devices 116, which may be the same storage device or different types of storage devices, and applications of computing device 110 may be stored in one or more storage devices 116 of computing device 110. In some embodiments, computing device 110 is a cloud computer, and processor 112, memory 114, and storage device 116 are shared resources provided on demand over the Internet.

[0072] like Figure 1 As shown, storage device 116 includes ontology application 118, and at least one of user generated data 190, training data 192, new topic database 194, and ontology 196. Ontology application 118 is configured to construct an ontology and to update the ontology using the data.

[0073] The ontology application 118 further includes: an emerging topic detector 120, a new concept validator 140, an ontology adjustment module 160, an ontology update module 170, a regulation module 180, and a management interface 185. In some embodiments, the ontology application 118 may include other applications or modules necessary for the operation of the ontology application 118. It should be noted that each module is implemented by a computer executable code or instruction, or a data table or database that together form an application. In some embodiments, each module may also include sub-modules. Alternatively, some modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable code. In some embodiments, some or all modules of the ontology application 118 may be located at a remote computing device or distributed in the cloud.

[0074] Emerging topic detector 120 is configured to score semantic distances between each pair of data entries and cluster entries based on topics when receiving or retrieving data entries from user-generated data 190, so as to generate topics for user-generated data 190. Emerging topic detector 120 may retrieve data entries within a specified time range, such as last week, last month, or last quarter, or may be a certain number of the most recent data entries, such as the last 1,000 data entries, the last 10,000 data entries, or the last 100,000 data entries. In one example, emerging topic detector 120 retrieves data entries for the last week, which is referred to as week 0. Reference Figure 2A , the emerging topic detector 120 includes a data cleansing and tagging unit 122 , a sentiment analyzer 124 , a similarity calculator 126 , a natural language parser (NLP) 128 , a semantic scorer 130 , and a cluster classifier 132 .

[0075] Data entries from user-generated data 190 (e.g., feedback on an e-commerce platform) may include noise. Data cleaning and tagging 122 is configured to retrieve data entries from user-generated data 190, clean and tokenize those data entries, and send those tokenized data entries to sentiment analyzer 124, similarity calculator 126, and NLP 128. In some embodiments, the cleaning process refers to removing certain symbols or words that are irrelevant to downstream work. In some embodiments, scikit-learn can be used to perform cleaning. Specifically, data cleaning and tagging 122 can use the classes listed in this link: http: / / scikit-learn.org / stable / modules / generated / sklearn.feature_extraction.text.TfidfVectorizer.html. For example, one of the attributes of the above class is "stop words stop_words", and data cleaning and tagging 122 provides a list of stop words, and in the operation of stop_words, the listed stop words are removed from the data entries. In an example, user 1 submitted a feedback, "This color of this Under Armor T-shirt is cool", and the words "this", "of", and "this" are included in the stop word list, then the output can be: identifier "User 1, feedback number 123", cleaned text "color, Under, Armor, T-shirt, cool".

[0076] The data cleaning and tokenizer 122 is also configured to tokenize the cleaned data entries based on the dictionary after cleaning the data entries. For example, if the mapping between the string and its ID is tokenized as {Armor: 0; Color: 1, Cool: 2, T-shirt: 3, Under: 4}, then the cleaned text "Color, Under, Armor, T-shirt, Cool" provides a list output with tokenized IDs [1, 4, 0, 3, 2] after tokenization. Through data cleaning and tokenization, each tokenized data entry is represented by a user identifier, a feedback identifier, and a tokenized cleaned text. The data cleaning and tokenization module 122 is also configured to send the tokenized data entries to the sentiment analyzer 124, the similarity calculator 126, and the NLP 128 after cleaning and tokenization.

[0077] The sentiment analyzer 124 is configured to predict the sentiment polarity of the cleaned text of each data entry upon receiving the tokenized data entry. In some embodiments, the sentiment is represented by a vector, and each dimension of the vector defines a sentiment. In some embodiments, five different sentiments are defined: positive, neutral, negative, very negative, and internet abuse. The correlation between the tokenized data entry and the sentiment is represented by a number between 0 and 1. 1 indicates a high correlation and 0 indicates no correlation. In some embodiments, the representation value of the sentiment is normalized so that the sum of the representation values ​​of all sentiments is 1. In one example, the result of analyzing the tokenized data entry by the sentiment analyzer 124 is [0.7, 0.2, 0.1, 0.0, 0.0], that is, positive 0.7, neutral 0.2, negative 0.1, very negative 0.0, and internet abuse 0.0. Therefore, the data entry is very likely to be positive feedback, may be neutral, and the possibility of being negative is very small. In some embodiments, the sentiment analyzer 124 uses certain techniques described by Pang, Bo et al. (Pang, Bo and Lee, Lillian, Opinion mining and sentiment analysis, Foundations and Trends in Information Retrieval, 2008, Vol. 2: No. 1-2, pp 1-135), the entire contents of which are incorporated herein by reference. In some embodiments, the sentiment analyzer 124 is a convolutional neural network classifier. In some embodiments, the sentiment analyzer 124 is pre-trained using a training data set included in the training data 192, wherein each data entry in the training data set includes a tokenized value and labels them with its corresponding sentiment attribute. In some embodiments, the labels of the training data entries may be that one of the emotions is 1 and the other emotions are 0. However, after training, the sentiment analyzer 124 may assign a number between 0 and 1 to one or more emotions of each data entry so as to more accurately represent the data entries about five different emotions. The sentiment analyzer 124 is also configured to send the sentiment vector to the semantic scorer 130 after obtaining the sentiment vector of the data entry.

[0078] Similarity calculator 126 is configured to determine the similarity between each pair of cleaned texts based on sentence embedding when receiving tokenized data entries. Here, the cleaned texts from any two data entries form a pair. The cleaned text of each data entry is represented by a vector. In some embodiments, the word representation in the vector space uses the method described by Mikolov, Thomas et al. (Mikolove, Tomas et al, efficient estimation of word representation in vector space, 2013, arxiv: 1301.3781v3), the entire contents of which are incorporated herein by reference. Through word embedding, the words in the cleaned text are mapped to real number vectors. Compare the vector of one data entry text in the pair with the vector of the other data entry text in the pair to determine the similarity or distance between them. In some embodiments, similarity calculator 126 uses the method described in Kusner et al. (Kusner et al. From word embedding to document distances, Proceedings of Machine Learning Research, 2015, V37, pp. 957-966), the entire contents of which are incorporated herein by reference. In some embodiments, the similarity score between each pair of data entries is normalized to 0-1, where 0 indicates no similarity, and 1 indicates substantially the same. In one example, the similarity between the cleaning texts from two data entries is 0.7, indicating a high degree of similarity between the two data entries or a close distance between the two data entries in the vector space. Similarity calculator 126 is also configured to, after obtaining the similarity score between any two data entries, send the similarity score to semantic scorer 130.

[0079] NLP 128 is configured to determine the syntactic structure of the text by analyzing the constituent words of the text based on the basic syntax when receiving the tokenized data entry. In some embodiments, the syntactic feature is a part-of-speech tag. In some embodiments, a pre-trained model is used, for example, Stanford parser (https: / / nlp.stanford.edu / software / lex-parser.shtml), the entire contents of which are incorporated herein by reference. In some embodiments, NLP 128 is also configured to process the initial parser output to provide a certain statistical result. For example, after syntactic parsing, NLP parser 128 can further calculate the number of nouns and verbs in the output. When the result of the data entry is 3 and 1, the text of the data entry includes 3 nouns and 1 verb. This simple and novel feature of the text is useful for accurate ontology construction and updating below. In some embodiments, NLP 128 is also configured to evaluate the syntactic or grammatical complexity of the data entry, and express the complexity as a real number. In some embodiments, NLP 128 is configured to calculate complexity using the number of unique words, the number of verbs, the number of nouns, the number of verb phrases, and the number of noun phrases. For example, assume that the maximum number of unique words in the text in all data sets (e.g., all training data sets) is C0 (e.g., 100), the maximum number of verb phrases in the text in all data sets is V0 (e.g., 10), and the maximum number of noun phrases in the text in all data sets is N0 (e.g., 20). For a given text t, it contains c1 unique words (e.g., 20), v1 verb phrases (e.g., 2), and n1 noun phrases (e.g., 5), the complexity of the text t can be calculated using the following formula: ((c1+1)×(v1+1)×(n1+1)) / ((C0+1)×(V0+1)×(N0+1)). In some embodiments, the value of the complexity, i.e., a real number, is used as the result of NLP 128. NLP 128 is also configured to send the results to semantic scorer 130 after obtaining the results.

[0080] The semantic scorer 130 is configured to calculate the semantic similarity score between each pair (any two) of texts when receiving different aspects of semantic information from the sentiment analyzer 124, the similarity calculator 126, and the NLP parser 128, i.e., the sentiment vector of the text of the data entry, the similarity score between each pair of texts of the data entry, and the parsing result of the text of the data entry. The semantic similarity score between each pair of texts is calculated by the following formula:

[0081]

[0082] n corresponds to the main type or source of features. Here n is 2, where i=0, 1, or 2 corresponds to sentiment features from sentiment analyzer 124, text similarity features from similarity calculator 126, and syntactic features from NLP parser 128, respectively. i is the weight of the feature source. In some embodiments, s i ∈[0,1.0] and In other words, each s i is equal to a number between 0 and 1 (including 0 and 1), and all s i The sum of is 1. In some embodiments, s for sentiment features, text similarity features, and syntactic features i They are 0.10, 0.85 and 0.05 respectively. j is a feature function that measures the similarity between two data items. Each of the features source, sentiment, text similarity, and syntax may include one or more feature functions, and the total number of feature functions is k. For example, f sentiment can be the cross entropy of the sentiment distribution of two items, f fact can be the cosine similarity of the noun phrase Tf-idf vectors of the two entries, and the method is extensible, where new feature functions can be added by adding new f j and the corresponding new w j can be easily incorporated into the above formula. j Yes j The higher the semantic similarity score between two data entries, the more similar the two data entries are. In some embodiments, the weight w j can be set to any value and will be automatically optimized during training. After obtaining the semantic similarity score between each pair of data entries, semantic scorer 130 is configured to send the semantic similarity score to cluster classifier 132. In some embodiments, parameter s is learned using training data entries retrieved from training data 192. i 、f j and w j , where the semantic score of each pair of training data entries is recorded to represent the relationship between the two training data entries. In other words, the feature function is an unsupervised model and is trained using the training data. In some embodiments, the training data is labeled with the corresponding features before training the model. In other embodiments, some models can be trained without adding any labels to the data. In some embodiments, feature functions such as sentiment prediction models require manual annotation for training. In some embodiments, feature functions such as text similarity models do not require labeled data for training.

[0083] The cluster classifier 132 is configured to classify those data entries into clusters when receiving the semantic similarity score between the texts of each pair of two data entries. In some embodiments, for the set of data entries {e1, e2, ..., em} belonging to a cluster, the semantic score of any pair of data entries in the given set is greater than a predetermined threshold t. The threshold can be selected according to system requirements. In some embodiments, if the system needs a high recall rate on novel topic detection, it can use a smaller number (such as 2) as a threshold. Then, most possible novel topics will be detected. On the contrary, if the system needs high precision, a relatively large number can be used. For example, assuming that the average number of topic sizes in a week in the historical record is 50, 60 can be used as a threshold. Then all detected clusters are very likely topics. After clustering, each cluster is defined as a new topic (concept candidate) and the cluster is stored in a new topic database 194. In some embodiments, the new topic database 194 stores new topics in batches or time intervals, such as one week, two weeks, one month or one quarter. For example, the system can process a batch of data entries every week and store new topics every week. Thus, we have new topics for the current week, new topics for the week before the current week, new topics for the week before the week before the week before, etc. These stored new topics are accessible to the new concept validator 140. In some embodiments, in addition to storing the new topics, the cluster classifier 132 may also send a message to the new concept validator 140 to notify the new concept validator 140 that a new batch of topics are available in the new topic database 194 so that the new concept validator 140 can immediately verify whether any newly detected topic meets the conditions of a new concept.

[0084] The new concept verifier 140 is configured to retrieve new topics from the new topic database 194 and verify whether any new topic is a new concept. Here, we define a new topic as a recognized topic detected from the recent data stream, that is, a cluster detected by the cluster classifier 132, and a new concept as a verified new topic. In other words, a new topic is a candidate for a new concept, and a new concept is a verified new topic. The verified new topic can then be used to update the ontology. Figure 2B As shown, the new concept validator 140 includes a new topic retrieval module 142 , a near duplicate identification module 144 , a concept comparison module 146 , a concept proposal module 148 and a concept verification module 150 .

[0085] The new topic retrieval module 142 is configured to retrieve new topics from the new topic database 194. The new topic retrieval module 142 can retrieve those new topics at predetermined time intervals (e.g., weekly or monthly), or in response to a message from the cluster classifier 132 that new topics are stored in the new topic database 194, or in response to an instruction from a system administrator of the management system 100. In some embodiments, the topic database 194 stores new topics on a weekly basis, and the new topic retrieval module 142 retrieves new topics for the last four weeks and sends the retrieved new topics to the near duplicate identification module 144. The new topics for the last four weeks include new topics for the current week and new topics for the previous three weeks, which are named Week 0, Week -1, Week -2, and Week -3, respectively.

[0086] When receiving a new topic from the new topic retrieval module 142, the near duplicate identification module 144 removes duplicate topics from the retrieved topics, thereby obtaining the most representative new topic. In some embodiments, when comparing whether the first topic and the second topic are duplicated: the near duplicate identification module 144 compares each data entry in the first topic with each data entry in the second topic to calculate a semantic similarity score; uses the semantic similarity score to determine whether the data entry in the first topic belongs to the second topic; then calculates the percentage of the data entries in the first topic that belong to the second topic; and determines whether the first topic and the second topic are duplicated based on the percentage. The near duplicate identification module 144 can calculate the semantic similarity score as described in relation to the semantic scorer 130, or call the semantic scorer 130 to calculate the semantic score. The near duplicate identification module 144 can use the average semantic similarity score between each data entry in the first topic and the data entry in the second topic to determine whether the data entry in the first topic belongs to the second topic. The threshold value of the average semantic score can be set in the range of 0.6-1.0, or preferably higher than 0.7, or more preferably higher than 0.8 or 0.9. When the percentage of data entries in the first topic that belong to the second topic is greater than a predetermined threshold, the near duplicate identification module 144 can determine that the first topic is a duplicate of the second topic. In some embodiments, the threshold is set to about 0.6, preferably about 0.7, and more preferably about 0.8 or 0.9.

[0087] In some embodiments, the near-duplicate identification module 144 is configured to compare the topics in the current week with the topics in the previous week to determine the repetitions by the above method. In one example, the current week 0 includes multiple T0 topics, the week -1 includes multiple T1 topics, the week -2 includes multiple T2 topics, and the week -3 includes multiple T3 topics. Each topic in the T0 topic is compared with the topics in the T1, T2, and T3 topics, and the repeated topics in the T0 topic are defined as T0 repeated topics. The near-duplicate identification module 144 removes the T0 repeated topics from the topic T0, and obtains non-repeating topics, i.e., T0-non-repeating (T0-non-repeating=T0-T0 repeating) by removing or deleting those repeated topics. In some embodiments, the T0, T1, T2, and T3 topics are combined together, and these topics are compared with each other to determine and remove repeated topics; the T0 topic or the non-repeating topics in all T0, T1, T2, and T3 topics are used for further processing. In yet another embodiment, the T1, T2, and T3 topics are combined, the T0 topic is compared with the combined topic, and the repeated topics between the T0 topic and the combined topic are removed from the T0 topic. In some embodiments, the new topic can be directly added to the ontology as a new concept to initialize the ontology. In some embodiments, the initial ontology can also be defined manually. After obtaining non-repeated topics, the near-duplicate identification module 144 is also configured to send those representative new topics to the concept comparison module 146.

[0088] The concept comparison module 146 is configured to calculate the probability of whether the representative new topic belongs to an existing concept when a non-repeated topic is received. In some embodiments, the concept comparison module 146 uses the classification model of the existing concepts in the ontology to determine when the new topic belongs to the concept. In some embodiments, for each concept in the ontology, a binary text classifier is provided and trained. In other words, each concept in the ontology 196 has its text classifier model. In some embodiments, the machine learning models of these classifiers can be binary classifiers, such as logical recursion, gradient boosting classifiers, and convolutional neural networks. (In some embodiments, when a concept is created and added to the ontology, text documents are collected and can be verified, for example, by a system administrator. These documents are semantically similar and are used as positive samples for the concept model. Some other documents that can be randomly selected from other existing categories or concepts are used as negative samples. Then, the corresponding text classifier of the new concept will be trained based on the combination of positive and negative samples. When the concept comparison module 146 performs the prediction of whether a representative new topic (i.e., a non-repeated topic) belongs to a concept in the ontology, the concept comparison module 146 inputs each text content of the data entry in a representative new topic into the binary text classifier and obtains The Boolean value of the text content is obtained. The Boolean value indicates whether a given feedback (data entry) belongs to the concept. When determining whether all text contents in a representative new topic belong to the concept, the percentage of text contents belonging to the concept indicates the probability that the representative new topic belongs to the concept. For example, if the representative new topic T contains 100 data entries, and the binary text classifier predicts that 90% of the data entries belong to concept C, the probability that the representative new topic T belongs to concept C is 90%. After calculating the probability and obtaining the probability that each representative new topic belongs to an available concept, the concept comparison module 146 is also configured to send the probability to the concept proposal module 148.

[0089] The concept proposal module 148 is configured to determine whether the representative new topic is a new concept candidate upon receiving the probability of each representative new topic belonging to one of the available concepts. For example, if the representative new topic T contains 100 data entries, and the binary text classifier predicts that 90% of the data entries belong to concept C, the probability that the representative new topic T belongs to concept C is 90%. The probability that the representative new topic T belongs to each concept is determined, and the highest probability that the representative new topic T belongs to one of the concepts is considered to be the probability that the representative new topic T belongs to the concept in the ontology. If the highest probability of a concept is greater than a predetermined number, such as approximately 90%, it is determined that the new topic T belongs to an existing concept. In one example, for the top 5 concepts C1, C2, C3, C4 and C5 to which the new topic T is most likely to belong, the probabilities may be 91%, 85%, 81%, 80% and 70% respectively, and then, because the highest percentage 91% is greater than the threshold 90%, it is determined that the new topic T belongs to C1. In another example, for the top five concepts C1', C2', C3', C4' and C5' that the new topic T' is most likely to belong to, the probabilities are 89%, 83%, 69%, 69% and 65% respectively, then the new topic T does not belong to the existing concept because the highest probability 89% is lower than the predetermined threshold 90%. In some embodiments, the threshold can vary based on the characteristics of the data entry and the purpose of the project. After selecting the new concept candidate based on the probability, the concept proposal module 148 is also configured to send the new concept candidate to the concept verification module 150.

[0090] The concept verification module 150 is configured to verify the new concept candidate to obtain a verified concept when receiving the new concept candidate. In some embodiments, the concept verification module 150 automatically verifies the new concept candidate based on certain criteria. In some embodiments, the concept verification module 150 provides an interface to display the new concept candidate to the system administrator, and verifies the new concept candidate according to the instruction from the system administrator via the interface. After verification, the concept verification module 150 discards the new concept candidate that has not passed the verification, and sends the verified new concept to the ontology adjustment module 160. The verified new concept is also referred to as the verified concept for short.

[0091] The ontology adjustment module 160 is configured to propose adjustments to the ontology upon receiving a validated concept. Figure 2C The ontology adjustment module 160 includes an ontology and new concept retrieval module 162 , a parent concept detection module 164 , a sibling concept similarity module 166 and an adjustment proposal module 168 .

[0092] The ontology and new concept retrieval module 162 is configured to retrieve ontologies from the ontology 196 and retrieve or receive validated concepts from the concept validation module 150 of the new concept validator 140 , and send the retrieved or received ontologies and validated concepts to the parent concept detection module 164 .

[0093] The parent concept detection module 164 is configured to detect the parent concept in the ontology for each verified concept when receiving the ontology and the verified concept. In some embodiments, this determination is similar to the functions of the concept comparison module 146 and the concept proposal module 148. Specifically, for each verified concept, the parent concept detection module 164 inputs each text content in the verified concept into the classifier of a concept of the ontology, and obtains the value of the input text content. Once the value of all text contents of the verified concept relative to a concept of the ontology is obtained, the probability of whether the concept of the ontology is the parent concept of the verified concept can be obtained. When the probability of the verified concept relative to each concept in the ontology is calculated, the concept in the ontology with the highest probability value is determined as the parent concept of the verified concept. The parent concept detection module 164 is also configured to send the verified concept and its parent concept to the brother concept similarity module 166 after obtaining the corresponding relationship between the verified concept and its parent concept in the ontology. In some embodiments, the parent concept detection module 164 is also configured to analyze each new concept to obtain their corresponding parent concepts.

[0094] The brother concept similarity module 166 is configured to determine the most closely related brother concept of the verified concept when receiving the verified concept and its parent concept. Specifically, the parent concept in the ontology can include more than one low-level concept or the child concept directly under the parent concept in the ontology. Those child concepts of the parent concept are called the brother concepts of the verified concept. Similar to the function of the parent concept detection module 164, the brother concept similarity module 166 is configured to determine the probability of whether the verified concept belongs to any brother concept. That is, the brother concept similarity module 166 uses the text content in the verified concept as the input to the classifier model of each brother concept, so as to obtain the probability that the verified candidate concept belongs to the brother concept. When all probabilities of each brother concept are obtained, the brother concept with the highest probability of the verified concept is determined as the most similar one among the brother concepts. The brother concept similarity module 166 is also configured to send the most similar one of the parent concept and the brother concept to the adjustment proposal module 168.

[0095] The adjustment proposal module 168 is configured to propose adjustments to the ontology based on the information when receiving the most similar sibling concepts of the parent concept and the verified concept. In some embodiments, the adjustment proposal module 168 is configured to propose adjustments to the ontology by performing insertion, promotion, shifting and merging. Figure 3ASchematically shows the current body (part) according to some embodiments of the present disclosure. Figure 3A As shown, the nodes are concepts of the ontology. Nodes A11, A12 and A13 have a common parent node A1, nodes A111, A112 and A113 have a common parent node A11, nodes A121 and A122 have a common parent node A12, and nodes A131, A132, A133 and A134 have a common parent node A13. When a new verified concept is added, the probability that the new topic belongs to node A11 is the highest, that is, A11 is the parent node of the new topic. When the new topic is compared with sibling nodes A111, A112, A113, the new topic is most similar to sibling node A112. When an insert operation is performed, the new topic is added as a subconcept of A11 and a sibling concept of A111, A112 and A113. In other words, concepts A111, A112, A113 and the new concept are subconcepts of node A11.

[0096] like Figure 3B As shown, the adjustment proposal module 168 is configured to propose an adjustment by performing promotion. Specifically, the node of the new topic points to node A1. In other words, after the adjustment, node A11 and the new topic node have a common parent node A1, and node A11 and the new concept node are sibling nodes. Figure 3C As shown, the adjustment proposal module 168 is configured to propose an adjustment by performing a shift. Specifically, the new concept node points to the node A112. In other words, after the adjustment, the new concept node is a child node of the node A112, and the node A112 is the parent node of the new concept node. Figure 3D As shown, adjustment proposes module 168 and is configured to propose adjustment by performing merging. Specifically, new concept node and A112 node are combined to form new node A112 / new concept, and new node A112 / new concept has parent node A11. Now, node A111, A112 / new concept and A113 are the child nodes of node A11. In addition, two child nodes are defined for new node A112 / new concept, and two child nodes are A112 and new theme respectively. That is, node A112 / new concept is the parent node of node A112 and new concept node. After proposing three types of adjustments, adjustment proposes module 168 and is also configured to send the proposed adjustment to ontology update module 170.

[0097] The ontology updating module 170 is configured to verify the proposed adjustments upon receiving the proposed adjustments from the adjustment proposing module 168 of the ontology adjustment module 160, and select the best proposal to update the ontology. Figure 2D The entity update module 170 includes a modification verification module 172 and an update module 174 .

[0098] The modification verification module 172 is configured to verify which proposed adjustment is the optimal adjustment upon receiving the proposed adjustment from the adjustment proposal module 168, and send the optimal adjustment to the update module 174. In some embodiments, the modification verification module 172 is configured to verify the adjustment by finding the optimal hierarchical adjustment. For a given ontology hierarchy H, there is a hierarchy Q of a series of related adjustments = {H1, H2, ..., H n} and the relevant dataset D (e.g., all text corpora used to train and test the corresponding concept classifier), the optimal hierarchy H opt is the following hierarchy:

[0099]

[0100] p(D|H) indicates the likelihood of data D given the hierarchical structure H. In some embodiments, the modified verification module 172 estimates the likelihood using the classification performance of the hierarchical model. In particular, the modified verification module 172 estimates the conditional likelihood using the macro-average recall of the entire classification system. The macro-average recall of the system is the average of the recalls of all concept classifiers on the test set.

[0101] For example, the hierarchy H includes M concepts. For each concept, there is a training set A i and a test set E i . Modify the verification module 180 in A i Training a binary concept classifier on E i Evaluate it and obtain its recall r i Here, recall = (true positives) / (true positives + false negatives). The macro average recall is By comparing the recall rate of each hierarchy, the optimal hierarchy can be determined. The modification verification module 172 is also configured to send the optimal hierarchy to the update module 174.

[0102] The updating module 174 is configured to update the ontology stored in the ontology 196 using the optimal proposal when the adjusted optimal proposal is received.

[0103] Adjustment module 180 is configured to adjust the classifiers of ontology concepts using the updated ontology and corresponding datasets when the ontology is updated by update module 174. Adjustment can be performed after each ontology update, or at predetermined time intervals (e.g., one month), or at the instruction of a system administrator.

[0104] The management interface 185 is configured to provide an interactive interface for presenting results and parameters to a system administrator, and receiving instructions and revised parameters from the system administrator in operation. The management interface 185 includes the above-mentioned validations and parameters, and in addition, may include threshold parameters for the cluster classifier 132, semantic score thresholds for the near-duplicate identification module 144, thresholds for the predictive concept proposal module 148, new concept validation, proposed adjustment validation, etc.

[0105] The user generated data 190 includes historical user generated data, such as user feedback on an e-commerce platform. The user generated data 190 may be arranged at predetermined time intervals, such as weekly or monthly.

[0106] The training data 192 includes data used to train the classifiers in the system 100. Each data set in the training data 192 may correspond to a specific classifier or other type of model and be labeled with corresponding features. For example, a data set of text items may be labeled with sentiment and used to train the sentiment analyzer 124.

[0107] New topic database 194 stores new topics detected by emerging topic detector 120. In some embodiments, new topics are stored in batches. Each batch of new topics may correspond to new topics detected from data entries such as a week, a month, or a quarter.

[0108] Ontology 196 stores the ontology of the system, which can be updated automatically or with minimal supervision by a system administrator. Among other things, ontology 196 includes concepts, relationships between concepts, and classifiers corresponding to each concept.

[0109] In certain embodiments, a system administrator may manually initialize the ontology 196 and update and expand the initialized ontology 196 after receiving more data and after the functionality of the ontology application 118 has been performed.

[0110] In some embodiments, ontology application 118 may use the first batch of data entries, detect emerging topics using emerging topic detector 120 , and use the classified emerging topics as initial ontology 196 .

[0111] Figure 4 The flowchart of establishing and updating the evolutionary ontology from user-generated content according to some embodiments of the present disclosure is schematically shown. Figure 1 The server computing device 110 shown in FIG. 1 is used to implement building and updating the evolving ontology.

[0112] like Figure 4As shown, user-generated data 190 is provided to or retrieved by emerging topic detector 120. User-generated data 190 may include a large amount of historical data, and emerging topic detector 120 may process only a batch of data at a time, such as user feedback in an e-commerce website in the past week. Then, emerging topic detector 120 processes a batch of user-generated data including many data entries to obtain the relationship between any two data entries. The relationship can be represented by a semantic similarity score, where the higher the score, the more similar the two data entries are. Based on the semantic similarity score, emerging topic detector 120 clusters the data entries into different groups. The data entries in the same group have high semantic similarity scores with each other. These groups are considered emerging topics. In some embodiments, emerging topic detector 120 can also use a threshold to filter the group, and only the group with a number of data entries greater than a threshold number (e.g., 50 or 60) is considered as an emerging topic. In some embodiments, emerging topic detector 120 also compares the detected new topics with the new topics detected in the past time (e.g., in the three weeks before the past week), and only keeps the new topics that were not displayed in the three weeks before that. Then, the emerging topic detector 120 sends the detected new topics to the new concept validator 140 .

[0113] When receiving a new topic, the new concept verifier 140 compares the new topic with the nodes in the ontology, where each node in the ontology represents a concept. The new concept verifier 140 calculates a novelty score for each new topic by comparing the similarity between each new topic and each concept. The novelty score can be calculated using a set of classification models. The new concept verifier 140 defines a new topic with a high novelty score as a verified new concept or a simply verified concept. Then, the new concept verifier 140 sends the verified concept to the ontology adjustment module 160.

[0114] When receiving each verified concept, the ontology adjustment module 160 calculates the similarity between the verified new concept and the node in the ontology, and defines the node with the highest similarity as the parent node of the verified concept. The parent node can have multiple child nodes. Then, the ontology adjustment module 160 compares the similarity between the verified concept and all the child nodes of the parent node (also called the sibling nodes of the verified concept), and determines the sibling node with the highest similarity score with the verified concept among those sibling nodes. The sibling node is called the determined sibling node. With the parent node and the determined sibling node, the ontology adjustment module 160 then proposes several different adjustments. In some embodiments, by performing insertion, the ontology adjustment module 160 inserts the verified concept as a child node of the parent node. In some embodiments, by performing promotion, the ontology adjustment module 160 inserts the verified new concept as a brother node of the parent node. In some embodiments, by performing shifting, the ontology adjustment module 160 inserts the verified new concept as a child node of the determined brother node. In some embodiments, by performing a merge, the ontology adjustment module 160 merges the verified concepts and the determined sibling nodes into a merged node. The merged node is a child node of the parent node, and the merged node is a parent node of the verified concepts and the determined sibling nodes. The ontology adjustment module 160 then sends those proposed adjustments to the ontology update module 170.

[0115] Upon receiving the proposed adjustments, the ontology update module 170 evaluates which adjustment is optimal and updates the ontology using the optimal adjustment proposal.

[0116] In some embodiments, after updating the ontology 196, the adjustment module 180 can further adjust the entire system and maintain relevant models according to the ontology changes. Models with high credibility are kept with high weights or defined with high weights, while models with low credibility will be discarded or defined with low weights.

[0117] In some embodiments, the system also includes an administration interface 185. The administration interface 185 provides an interface, such as a graphical user interface (GUI), to a system administrator so that the administrator can interact with the application during processing. For example, the system administrator can use the administration interface 185 to visualize and demonstrate keywords, novelty thresholds, frequency of occurrence, and summaries of new concepts, adjust novelty score thresholds, validate new concepts, etc.

[0118] In some embodiments, the system may also include an initialization step to build the ontology 196 from scratch. In some embodiments, the initial ontology 196 is manually prepared by a system administrator. In some embodiments, the ontology 196 is automatically built by using a certain amount of user-generated data to detect emerging topics, classifying those emerging topics, and using the detected emerging topics as concepts of the ontology to build the initial ontology 196. In some embodiments, the initialization of the ontology 196 is performed by the system administrator supervising and revising the results of the above-mentioned automatic method.

[0119] Figure 5 A method for detecting emerging topics according to some embodiments of the present disclosure is schematically shown. In some embodiments, the method comprises Figure 1 In some embodiments, Figure 5 The method shown corresponds to the functionality of the emerging topic detector 120. It should be noted that unless otherwise specified in the present disclosure, the steps of the method may be arranged in a different order and are therefore not limited to Figure 5 in the order shown.

[0120] like Figure 5 As shown, at process 502, the data cleaning and tagging unit 122 retrieves or receives a batch of data entries from the user-generated data 190. The batch of data entries may be, for example, user feedback on an e-commerce website last week. The number of data entries may vary, for example, 10,000 data entries.

[0121] After retrieving the data entries, at process 504, the data cleaning and tagging device 122 cleans the data entries and tokenizes the cleaned data entries into numbers. The data entries (e.g., feedback) are typically text. In some embodiments, when images are included in the data entries, the data cleaning and tagging device can remove the images from the data entries or convert the images into text. The data cleaning and tagging device 122 then separates the text into words and cleans the words by removing certain irrelevant symbols or words. After obtaining the cleaned words, the data cleaning and tagging device 122 tokenizes each data entry into a digital representation and sends the tokenized text of the data entry to the sentiment analyzer 124, the similarity calculator 126, and the NLP 128.

[0122] At process 506, the sentiment analyzer 124 predicts the sentiment polarity of each tokenized text upon receiving the tokenized text of the data entry. In some embodiments, the sentiment analyzer 124 defines five emotions and gives five corresponding values ​​for each data entry using a pre-trained model. The five emotions include positive, neutral, negative, very negative, and Internet abuse. In some embodiments, the pre-trained model is a classification model, such as a gradient recursive classifier, and the training data is retrieved from the training data 192. The training data may be a set of data entries with sentiment labels, i.e., positive, neutral, negative, very negative, and Internet abuse features of the data entry. When the target data entry is different, the sentiment label may also be changed accordingly. For example, if the data entry does not contain any Internet abuse data, the label may not need to include the feature. In one example, the result of a data entry analyzed by the sentiment analyzer 124 may be [0.7, 0.2, 0.1, 0.0, 0.0], i.e., positive 0.7, neutral 0.2, negative 0.1, very negative 0.0, and Internet abuse 0.0. Therefore, the data entry is highly likely to be positive feedback, possibly neutral, and has a low probability of being negative. After predicting the sentiment polarity of the batch of data entries, the sentiment analyzer 124 sends the results to the semantic scorer 130 .

[0123] At process 508, similarity calculator 126, after receiving the tokenized data entry, calculates the text similarity between any two tokenized data entries based on sentence embedding. Specifically, similarity calculator 126 represents the words in each text (i.e., each cleaned and tokenized data entry) by n-dimensional vector space, wherein semantically similar or semantically related words are closer according to the training model. After the text is represented by the vector, similarity calculator 126 calculates the similarity between any two texts. In some embodiments, in order to calculate the similarity, similarity calculator 126 not only considers the meaning of the words in the text, but also considers the relationship between the words in the text, especially the order of the words in the text. In some embodiments, the similarity score is represented by a number between 0 and 1, wherein 0 indicates that the two data entries are far apart in the vector space and have no similarity at all, and 1 indicates that the two data entries are close or overlapped in the vector space, and are substantially the same. In one example, if the similarity score is greater than about 0.6-0.8, it is considered that the two texts are very similar, and if the similarity score is less than about 0.6, it is considered that the two texts are not similar. In some embodiments, the comparison between two tokenized texts results in a plurality of scores, each score corresponding to a word or words having similar characteristics. For example, words related to color in the text are selected for comparison so that the result of the comparison includes a similarity score corresponding to the color. After calculating the similarity score between any two of the cleaned and tokenized data entries, the similarity calculator 126 sends the similarity score to the semantic scorer 130.

[0124] At process 510, NLP 128, after receiving the cleaned and tokenized data entry (text), determines the syntactic structure of the text by analyzing the constituent words of the text based on the basic grammar. In some embodiments, NLP 128 uses part-of-speech tagging. In some embodiments, NLP 128 evaluates the syntactic or grammatical complexity of the data entry and represents the complexity as a real number. After obtaining the numbers for each cleaned and tokenized data entry, NLP 128 sends these numbers to semantic scorer 130.

[0125] In some embodiments, processes 506, 508, and 510 are performed in parallel or independently.

[0126] At process 512, semantic scorer 130 calculates a semantic similarity score for each pair of data entries (i.e., any two data entries) upon receiving the sentiment polarity of each data entry from sentiment analyzer 124, the similarity score between any two data entries from similarity calculator 126, and the NLP score of each data entry. Semantic scorer 130 calculates the semantic similarity score based on the above three types of features using the following formula:

[0127]

[0128] Where n corresponds to the main types of features or feature sources: sentiment features, text similarity features, and syntactic features; s i is the weight of the feature source; f j is a feature function that measures the similarity between two data items, and each of the features source, sentiment, text similarity, and syntax may include one or more feature functions; k is the total number of feature functions; w j Yes j In some embodiments, the parameters in the formula can be obtained using a training data set using a training model, or the parameters are predetermined values ​​entered by a system administrator. In some embodiments, the semantic similarity score is a positive number. After obtaining the semantic similarity score between each pair of data entries (cleaned text and tokenized text) using the above formula, the semantic scorer 130 sends the semantic similarity score to the cluster classifier 132.

[0129] At process 514, cluster classifier 132, upon receiving the semantic similarity score between each pair (any two) of data entries, classifies the data entries based on the semantic similarity score. Specifically, cluster classifier 132 groups the data entries into clusters, and the data entries in the same cluster have higher semantic similarity scores. In some embodiments, a threshold is defined for the cluster, which means that the semantic similarity score of any two data entries in the same cluster is greater than the threshold score. The value of the threshold score can be determined based on the subject of the data entry, the required recall rate, and the required precision. In some embodiments, when a high recall rate is required, a small threshold is given. In some embodiments, when high precision is required, a large threshold is given. After obtaining the cluster, cluster classifier 132 stores the cluster in the new subject database 194. In some embodiments, each cluster includes one or more data entries, and cluster classifier 132 can only store clusters with a large number of data entries. The threshold number of data entries in the cluster can be set to about 5-500. In some embodiments, the threshold number is set in the range of 25-120. In some embodiments, the threshold number is set in the range of about 50-60. In one example, the average cluster size in a week is about 50, the threshold number is set to 60, and the stored clusters are likely to be real topics or topics. The stored clusters are also called emerging topics.

[0130] Through the above steps 502-514, the emerging topic detector 120 obtains a certain number of new topics, each of which includes some data entries. These steps can be repeatedly performed in batches at predetermined time intervals (e.g., weekly or monthly). In other words, user-generated entries are collected and stored on a weekly basis, and when data entries are available, the emerging topic detector 120 processes the data entries within a week. Therefore, the new topic database 194 includes different new topic sets, each set corresponding to data entries from a specific week or a specific month.

[0131] Figure 6 A method for verifying a new subject to obtain a new concept according to some embodiments of the present disclosure is schematically shown. The new concept is a verified new subject. In some embodiments, the method is Figure 1 In some embodiments, Figure 6 The method shown corresponds to the functionality of the new concept proof-of-concept 140. It should be noted that unless otherwise specified in the present disclosure, the steps of the method may be arranged in a different order and are therefore not limited to Figure 6 In some embodiments, Figure 6 The process shown in Figure 5 The processes shown are then executed sequentially.

[0132] At process 602, the new topic retrieval module 142 retrieves new topics from the new topic database 190. The retrieved new topics include the current batch of new topics for analysis and several previous batches of new topics that have been analyzed before. For example, the new topic retrieval module 142 retrieves new topics from the most recent week (hereinafter referred to as week 0) and retrieves new topics from the three weeks before the most recent week (hereinafter referred to as week -1, week -2, week -3). The batches of week 0, week -1, week -2, and week -3 include, for example, 120, 130, 11, and 140 new topics, respectively. By Figure 5 The process shown analyzes one week of data entries to obtain each batch of new topics. In the following, the batches and numbers of topics are used only for clarity of description. Figure 6 The process shown in is not intended to limit the scope of the present disclosure. After retrieving a new topic, the new topic retrieval module 142 sends the new topic to the near duplicate identification module 144.

[0133] After retrieving four batches of new topics, at process 604, the near duplicate identification module 144 identifies duplicate topics in the Week 0 topics. Specifically, in order to compare whether a topic in Week 0 is a duplicate of a topic in any topic in Week -1, Week -2, or Week -3 (hereinafter referred to as a target topic), the near duplicate identification module 144 performs the following operations: first, a semantic similarity score is calculated between each data entry in the Week 0 topic and a data entry in the target topic, and based on the semantic similarity score, whether the Week 0 data entry belongs to the target topic is determined; then, the process is repeated and the probability of each Week 0 data entry belonging to the target topic is determined; then, the percentage of Week 0 data entries belonging to the target topic is calculated. If the percentage is higher than a predetermined value, the near duplicate identification module 144 determines that the Week 0 topic is a duplicate of the target topic. If the percentage is lower than a predetermined value, the near duplicate identification module 144 continues to compare the Week 0 topic with all other Week -1, Week -2, and Week -3 topics. If the Week 0 topic is not a duplicate topic of any of the Week -1, Week -2, and Week -3 topics, then the near duplicate identification module 144 determines that the Week 0 topic is a non-duplicate topic. The near duplicate identification module 144 repeats the above process for each Week 0 topic, obtains non-duplicate topics from the Week 0 topics, and sends the non-duplicate topics to the concept comparison module 146. In one example, among the 120 Week 0 new topics, 90 of them have one or more duplicate topics in Week -1, Week -2, or Week -3, and 30 of them are non-duplicate topics.

[0134] At process 606, concept comparison module 146 calculates whether non-repeating topics belong to existing concepts. Specifically, for each concept in the ontology, a binary text classifier is constructed and trained. That is, each concept in the ontology database has its text classifier model. In some embodiments, the classifier model is a logical recursion or gradient boosting classifier. For each topic of non-repeating topics (for example, 30 non-repeating topics), non-repeating topics include many data entries. Each data entry in the non-repeating topic is used as the input of a classifier of a concept (hereinafter referred to as the target concept) to obtain a Boolean value indicating whether the data entry belongs to the target concept. After calculating each data entry in the non-repeating topic to determine whether it belongs to the target concept, the percentage of the data entry belonging to the target concept in the non-repeating topic can be calculated. For example, if the non-repeating topic T contains 100 data entries, and 90 of them belong to a given target concept C, the probability that the non-repeating topic T belongs to the target concept C is 90%. In some embodiments, after the data entry in the non-repeating topic is compared with all concepts, the highest probability corresponding to one of the concepts is recorded. As a result, each of the 30 non-repeating topics is given a probability score relative to one of the concepts (the highest score when compared to all concepts). The concept comparison module 146 then sends these 30 probability scores to the concept proposal module 148, each of which corresponds to one of the concepts.

[0135] At process 608, the concept proposal module 148 ranks the 30 non-duplicate topics based on their probability scores and proposes new topics with low probability scores as proposed concepts. In some embodiments, low probability scores are defined as less than about 0.4. In some embodiments, low probability is defined as less than 0.25. The number of new topics can be 8, and the concept proposal module 148 sends the proposed concepts (e.g., 8 proposed concepts from the 30 non-duplicate new topics) to the concept verification module 150.

[0136] At process 610, upon receiving the proposed concepts, the concept verification module 150 presents the proposed concepts, for example, 8 proposed concepts, to the system administrator, and the system administrator verifies the proposed concepts, for example, 5 of the 8 proposed concepts may be selected as true concept candidates.

[0137] Then at process 612, the concept validation module 150 may further label the 120 week 0 new topics with "duplicate data entry", "unvalidated concept" or "validated concept" in the new topic database 194, and send the 5 validated concepts to the ontology adjustment module 160. In some embodiments, concept validation may not be required, and the concept proposal module 148 directly sends the proposed concepts (e.g., 8 proposed concepts) to the ontology adjustment module 160. In some embodiments, validation may also be performed automatically using certain criteria (e.g., features of the subject terms).

[0138] Figure 7 The method for proposing ontology adjustments based on verified concepts and updating the ontology using the optimal adjustments according to some embodiments of the present disclosure is schematically shown. In some embodiments, the method comprises Figure 1 In some embodiments, Figure 7 The method shown corresponds to the functions of the main body adjustment module 160 and the main body update module 170. It should be particularly noted that unless otherwise specified in the present disclosure, the steps of the method can be arranged in a different order and are therefore not limited to Figure 7 In some embodiments, Figure 7 The process shown in Figure 6 The processes shown are then executed sequentially.

[0139] At process 702, the ontology and new concept retrieval module 162 retrieves the ontology 196 and retrieves (or receives) the validated concepts from the concept validation module 150 and sends the retrieved data to the parent concept detection module 164. The process is described below in relation to one validated concept, and each new validated concept should be similarly processed.

[0140] At process 704, in response to receiving the retrieved data, the parent concept module 164 detects the parent concept for each verified concept from the ontology. In some embodiments, each existing concept from the ontology has a classifier, and the verified concept includes multiple data entries. When the verified concept is compared with the existing concept, the parent concept module 164 inputs each text content of the verified concept into the classifier of the existing concept to obtain a value. This value indicates whether the text of the new concept belongs to the ontology concept. When all data entries are analyzed, the percentage of the data entry belonging to the existing concept is calculated and regarded as the probability of whether the verified concept belongs to the existing concept. The parent concept module 164 compares the data entry of the verified concept with each existing concept (node) in the ontology, and obtains the probability of whether the verified concept belongs to any existing concept. Then, the parent concept module 164 selects the existing concept corresponding to the highest probability as the parent concept of the verified concept. Then, the parent concept module 164 sends the ontology, the selection of the parent concept and the verified concept (or its specific identification) to the brother concept similarity module 166. In some embodiments, the parent concept module 164 may provide not only the most relevant parent concepts, but also a list of relevant parent concepts of the verified concept with corresponding probability values. The results may be presented and selected through the management interface 185 .

[0141] At process 706, upon receiving the ontology, parent concept, and verified concept, the sibling concept similarity module 166 determines all child concepts of the parent concept, also referred to as the verified concept's sibling concepts; uses the sibling concept's classifier to calculate the probability of a data entry in the verified concept belonging to one of the sibling concepts; calculates the percentage of the data entry belonging to the sibling concept; repeats the process to calculate the percentage of the data entry relative to each sibling concept; and selects the sibling concept with the highest percentage. The sibling concept similarity module 166 then sends the parent concept and the most closely related sibling concept (with the highest percentage) to the adjustment proposal module 168. In some embodiments, the sibling concept similarity module 166 can provide not only closely related sibling concepts, but also a list of related sibling concepts of the verified concept with corresponding probability values. In some embodiments, the sibling concept similarity module 166 can include more than one sibling concept list, each list corresponding to a related parent concept, and the system administrator views and selects the parent concept and sibling concept of the verified concept through the management interface 185.

[0142] At process 708, upon receiving the most relevant parent concept and the most closely related sibling concept, the adjustment proposal module 168 proposes several ways to adjust the hierarchy of the ontology. In some embodiments, the adjustment proposal module 168 may insert the new concept candidate as a child node of the parent concept. In some embodiments, the adjustment proposal module 168 may propose the following: FIG. 3B to FIG. 3DThe basic operations of lifting, shifting, and merging are shown. In some embodiments, the adjustment proposal module 168 sends the proposed adjustment to the modification verification module 172.

[0143] At process 710, upon receiving the proposed adjustment, modification verification module 172 verifies the adjustment. Specifically, for data set D, each proposed adjustment has a corresponding hierarchy. The optimal hierarchy of the multiple hierarchies can be determined in the following manner:

[0144]

[0145] The optimal hierarchy is then defined as the validated hierarchy. In some embodiments, the management interface 185 may provide a system administrator with a means to change parameters (e.g., cancel) to change the result of the optimal hierarchy and optimize the result. After validation, the modification validation module 172 presents the validation results through the management interface 185, which may include a list of proposed adjustments and a value indicating whether the proposed adjustments are optimal.

[0146] At process 712, the system administrator may select one of the proposed adjustments through management interface 185 to verify the verified adjustment, and if the verification selection is yes, management interface 185 sends the verification to update module 174. If the system administrator determines that the adjustment is invalid, it may provide instructions to parent concept module 164 via management interface 185 to cause parent concept module 164 to detect a parent concept for another verified concept. In some embodiments, the system administrator may provide instructions to adjustment proposal module 168 via management interface 185 to cause adjustment proposal module 168 to propose a different adjustment of the hierarchy using different parameters. In some embodiments, the verification step is not necessary, and the verified adjustment is sent directly to update module 174.

[0147] At process 714 , upon receiving a valid adjustment or a verified adjustment, the update module 174 updates the ontology using the valid adjustment.

[0148] In some embodiments, the method further includes an adjustment mechanism, wherein the adjustment module 180 analyzes the updated ontology and retrains the relevant models based on the updated ontology.

[0149] In summary, certain embodiments of the present disclosure provide a semantic analysis pipeline to automatically mine and detect emerging topics and new concepts from user-generated data. In addition, a management interface is provided to present detected topics as well as statistics, generated summaries, sentiment distributions, and receive instructions from system administrators to adjust system parameters.

[0150] The foregoing description of exemplary embodiments of the present disclosure has been presented for purposes of illustration and description only and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed.Many modifications and variations are possible in light of the above teachings.

[0151] The embodiments are selected and described in order to explain the principles of the present disclosure and its practical application, so as to enable other persons skilled in the art to utilize the present disclosure and the various embodiments and make various modifications to suit the specific purpose intended. Alternative embodiments will become apparent to those skilled in the art to which the present disclosure belongs without departing from the spirit and scope of the present invention. Therefore, the scope of the present disclosure is limited by the appended claims rather than the foregoing description and the exemplary embodiments described therein.

[0152] References:

[0153] 1. Tomas Mikolov, Ilya Sutskever et al, Distributed representations of words and phrases and their compositionality, 2013, arXiv:1310.4546[cs.CL].

[0154] 2. Quoc Le, Tomas Mikolov, Distributed representations of sentences and documents, Proceedings of the 31st International Conference on Machine Learning, Beijing, China, 2014, JMLR 32(2):1188-1196.

[0155] 3. Yoon Kim, Convolutional Neural Networks for Sentence Classification, arXiv:1408.5882[cs.CL].

[0156] 4. Lei Tang, Jianping Zhang, Huan Liu, Automatically adjusting contenttaxonomies for hierarchical classification, Proceedings, 2006.

[0157] 5、Bo Pang and Lillian Lee,Opinion mining and sentiment analysis,Foundations and Trends in Information Retrieval,2008V2(1-2):1–135。

[0158] 6、David M Blei,Probabilistic topic models,Communications of the ACM,2012,V55(4):77–84。

[0159] 7、Kunal Punera,Suju Rajan,Joydeep Ghosh,Automatically learningdocument taxonomies for hierarchical classification,Special interest tracksand posters of the 14th international conference on World Wide Web,2005,pp.1010-1011。

[0160] 8、Mikolov,Tomas;et al.Efficient estimation of word representations invector space,2013,CoRR,abs / 1301.3781。

[0161] 9、Sanjeev Arora,Yingyu Liang,Tengyu Ma,A simple but tough-to-beatbaseline for sentence embeddings,ICLR 2017。

[0162] 10、Yiming Yang,Thomas Ault,Thomas Pierce and Charles W Lattimer,Improving text categorization methods for event tracking,SIGIR'00 Proceedingsof the 23rd annual international ACM SIGIR conference on Research anddevelopment in information retrieval,2000,pp.65-72。

[0163] 11、Jian Zhang,Zoubin Ghahramani and Yiming Yang,A probabilistic modelfor online document clustering with application to novelty detection,NIPS'04Proceedings of the 17th International Conference on Neural InformationProcessing Systems,2004,pp1617-1624。

[0164] 12、Matt J.Kusner,Yu Sun,Nicholas I.Kolkin,Kilian Q.Weinberger,Fromword embeddings to document distances,Proceedings of the 32nd InternationalConference on Machine Learning,Lille,France,JMLR:W&CP,2015,v37:857-966。

[0165] 13、Dingquan Wang,Weinan Zhang,Gui-Rong Xue,and Yong Yu,Deepclassifier for large scale hierarchical text classification,Proceedings,2009。

[0166] 14、David M.Blei,John D.Lafferty,Dynamic topic models,Proceedings ofthe 23rd International Conference on Machine Learning,Pittsburgh,PA,2006,pp113-120。

[0167] 15、Wilas Chamlertwat,Pattarasinee Bhattarakosol,Tippakorn Rungkasiri,Discovering consumer insight from twitter via sentiment analysis,Journal ofUniversal Computer Science,2012,V18(8):973-992。

[0168] 16、https: / / github.com / dmlc / xgboost

[0169] 17、Nagaraju Bandaru,Eric D.Moyer,Shrisha Radhakrishna,Method andsystem for analyzing user-generated content,U.S.Published Patent ApplicationNo.2008 / 0133488 A1,2008。

[0170] 18、Rui Cai,Qiang Hao,Changhu Wang,Rong Xiao,Lei Zhang,Mining topic-related aspects from user generated content,U.S.Patent No.8,458,115 B2,2013。

[0171] 19、Rajeev Dadia,Vidya Sagar,Anisingaraju,Prashanth Talanki,Systemsand methods for analyzing consumer sentiment with social perspective insight,U.S.Published Patent Application No.2016 / 0196564A1,2016。

Claims

1. A method for constructing an evolutionary ontology database, comprising: The computing device receives a plurality of data entries; The computing device calculates a semantic similarity score between any two of the data items based on feature sources and feature similarities of the data items; The computing device clusters the data items into a plurality of current topics based on the semantic similarity scores; the computing device selecting a new concept from the current topic by comparing the current topic to a plurality of previous topics prepared using previous data entries; The computing device updates the evolving ontology database using the new concept; Wherein, the step of updating the evolution ontology database comprises: detecting the most relevant parent concept corresponding to the highest probability value by comparing the at least one verified concept with the concepts in the ontology to obtain probability values; Calculating the similarity between the at least one verified concept and a sibling concept to obtain a sibling concept with the greatest similarity, wherein the sibling concept is a child concept of the most relevant parent concept corresponding to the highest probability value; Proposing ontology adjustments based on the most relevant parent concept corresponding to the highest probability value and the sibling concept of the greatest similarity; and The ontology is updated using the best adjustment among the proposed ontology adjustments.

2. The method according to claim 1, wherein: The semantic similarity score between any two data entries in the data entries is calculated by the following formula: Semantic similarity score = ,in, is the weight of the feature source, is one of the feature similarities between the two data entries in the data entries, yes , and j, k and n are positive integers.

3. The method according to claim 2, wherein: The data entry is user-generated feedback, and the step of calculating the semantic similarity score comprises: predicting a sentiment similarity value by a sentiment analyzer, the sentiment similarity value indicating the similarity between the two data entries in terms of positive feedback, negative feedback, neutral feedback, very negative feedback, and Internet abuse; predicting, by a similarity calculator, a text similarity value, the text similarity value representing the similarity between the semantics of the text extracted from the two data entries; and A syntactic similarity value is predicted by a neutral language parser, the syntactic similarity value representing the syntactic complexity of the texts of the two data items.

4. The method according to claim 3, wherein: The step of clustering the data items further comprises: A semantic similarity score for the two data items is calculated using the sentiment similarity value, the textual similarity value, and the syntactic similarity value.

5. The method according to claim 2, wherein: The steps for selecting new concepts from the current topic include: Retrieving the current topic and the previous topic; comparing each data entry in the current topic with each data entry in the previous topic to calculate a semantic similarity score to identify near-duplicate topics; Removing the nearly duplicate topics from the current topic to obtain non-duplicate topics; comparing the non-repeated topics with concepts in the ontology database to obtain novel concept candidates, wherein the novel concept candidates are non-repeated topics having low similarity scores with any concepts in the ontology database; and The novel concept candidates are verified according to instructions from an administrator of the ontology database to obtain the new concepts.

6. The method according to claim 1, wherein: The proposed adjustment includes an insertion adjustment in which the new concept is defined as a child node of the most relevant parent concept corresponding to the highest probability value.

7. The method according to claim 1, wherein: The proposed adjustments include a boosting adjustment in which the new concept is defined as a sibling node of the most relevant parent concept corresponding to the highest probability value.

8. The method according to claim 1, wherein: The proposed adjustment includes a shift adjustment in which the new concept is defined as a child node of the sibling concept of the maximum similarity.

9. The method according to claim 1, wherein: The proposed adjustment includes a merging adjustment, in which the new topic is combined with the sibling concept with the maximum similarity to form a combined concept, the combined concept is defined as the child node of the most relevant parent concept corresponding to the highest probability value, and the new topic and the sibling concept with the maximum similarity are defined as child nodes of the combined concept, and the new topic is a candidate for the new concept.

10. The method according to claim 2, wherein: Each concept in the ontology database is defined by a classification model, which includes a logistic recursive model and a gradient boosting classifier.

11. The method according to claim 10, wherein: The step of updating the evolving ontology database includes updating labels of the plurality of data entries, and the method further includes adjusting the classification model using the updated data entries.

12. A system for constructing an evolutionary ontology database, the system comprising a computing device, the computing device comprising a processor and a storage device storing computer executable code, wherein: When executed at the processor, the computer executable code is configured to: receiving a plurality of data entries; calculating a semantic similarity score between any two of the data items based on feature sources and feature similarities of the data items; clustering the data items into a plurality of current topics based on the semantic similarity scores; selecting new concepts from the current topic by comparing the current topic with a plurality of previous topics prepared using previous data entries; as well as Updating the evolving ontology database using the new concept; Wherein, the step of updating the evolution ontology database comprises: detecting the most relevant parent concept corresponding to the highest probability value by comparing the at least one verified concept with the concepts in the ontology to obtain probability values; Calculating the similarity between the at least one verified concept and a sibling concept to obtain a sibling concept with the greatest similarity, wherein the sibling concept is a child concept of the most relevant parent concept corresponding to the highest probability value; Proposing ontology adjustments based on the most relevant parent concept corresponding to the highest probability value and the sibling concept of the greatest similarity; and The ontology is updated using the best adjustment among the proposed ontology adjustments.

13. The system according to claim 12, wherein: The semantic similarity score between any two data entries in the data entries is calculated by the following formula: Semantic similarity score = ,in, is the weight of the feature source, is the feature similarity between the two data entries in the data entries, yes , and j, k and n are positive integers.

14. The system according to claim 13, wherein: The data entry is user-generated feedback, and the computer executable code is configured to calculate a semantic similarity score by: predicting a sentiment similarity value by a sentiment analyzer, the sentiment similarity value indicating the similarity between the two data entries in terms of positive feedback, negative feedback, neutral feedback, very negative feedback, and Internet abuse; predicting a text similarity value by a similarity calculator, the text similarity value representing the similarity between the semantics of the text extracted from the two data entries; as well as A syntactic similarity value is predicted by a neutral language parser, the syntactic similarity value representing the syntactic complexity of the texts of the two data items.

15. The system of claim 13, wherein: The computer executable code is configured to select a new concept from the current topic by: Retrieving the current topic and the previous topic; comparing each data entry in the current topic with each data entry in the previous topic to calculate a semantic similarity score to identify near-duplicate topics; Removing the nearly duplicate topics from the current topic to obtain non-duplicate topics; comparing the non-repeated topics with concepts in the ontology database to obtain novel concept candidates, wherein the novel concept candidates are non-repeated topics having low similarity scores with any concepts in the ontology database; and The novel concept candidates are verified according to instructions from an administrator of the ontology database to obtain the new concepts.

16. The system of claim 12, wherein: The proposed adjustment includes insertion adjustment, lifting adjustment, shift adjustment and merging adjustment, and the computer executable code is configured to: In the insertion adjustment, the new concept is defined as a child node of the most relevant parent concept corresponding to the highest probability value; In the lifting adjustment, the new concept is defined as a sibling node of the most relevant parent concept corresponding to the highest probability value; In the shift adjustment, the new concept is defined as a child node of the sibling concept with the maximum similarity; as well as In the merge adjustment, the new topic is combined with the sibling concept with the maximum similarity to form a combined concept, the combined concept is defined as the child node of the most relevant parent concept corresponding to the highest probability value, and the new topic and the sibling concept with the maximum similarity are defined as child nodes of the combined concept.

17. A non-transitory computer readable medium storing computer executable code, wherein: When executed at a processor of a computing device, the computer executable code is configured to: receiving a plurality of data entries; Calculate a semantic similarity score between any two of the data entries; clustering the data items into a plurality of current topics based on the semantic similarity scores; selecting new concepts from the current topic by comparing the current topic with a plurality of previous topics prepared using previous data entries; as well as updating the evolving ontology database using the new concepts; The step of updating the evolution ontology database includes: detecting the most relevant parent concept corresponding to the highest probability value by comparing the at least one verified concept with the concepts in the ontology to obtain probability values; Calculating the similarity between the at least one verified concept and a sibling concept to obtain a sibling concept with the greatest similarity, wherein the sibling concept is a child concept of the most relevant parent concept corresponding to the highest probability value; Proposing ontology adjustments based on the most relevant parent concept corresponding to the highest probability value and the sibling concept of the greatest similarity; and The ontology is updated using the best adjustment among the proposed ontology adjustments.

18. The non-transitory computer readable medium of claim 17, wherein: The semantic similarity score between any two data entries in the data entries is calculated by the following formula: Semantic similarity score = ,in, is the weight of the feature source, is the feature similarity between the two data entries in the data entries, yes , and j, k and n are positive integers.

Images