Identifying taxonomic hierarchies using a trained machine learning pipeline

Abstract

Techniques are disclosed for identifying a category associated with a target data item using a trained machine learning (ML) pipeline, even if the identified category does not already exist in the hierarchy. The ML pipeline may include, among other things, a trained cluster-based machine learning model and a trained classification-based machine learning model. If the results of the cluster-based machine learning model and the classification-based machine learning model are the same, the target data item is assigned to a hierarchical classification that matches the same results of the machine learning model. The assigned hierarchical classification may be verified by operation of a subsequent trained ML model that determines whether parent and child categories in the identified classification are properly associated with each other.

Description

[Technical field]

[0001] RELATED APPLICATIONS, INCORPORATED BY REFERENCE The related application, Application No. 17 / 303,918, filed June 10, 2021, is incorporated herein by reference. Applicants hereby override any disclaimer of claim scope in the parent application or its prosecution history, and advise the USPTO that the claims in this application may be broader than any claims in the parent application.

[0002] Technical Field The present disclosure relates to hierarchical classification of data. In particular, the present disclosure relates to identifying classification hierarchies in data using a trained machine learning pipeline. [Background technology]

[0003] background Descriptions and terminology used in any of a variety of contexts change over time. Also, at any given time, terminology can often vary between different organizations ("entities"), even when referring to the same subject matter. The natural variation in terminology, whether between entities or over time, when describing the same subject matter complicates analysis of data items because the meaning of the terminology can be unclear. Alternatively, a query for a subject matter using a first term may not identify query results that refer to the subject matter but are described using a second, different term. Summary of the Invention

[0004] Applying consistent terminology to a subject area will increase the efficiency and accuracy of analysis.

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued, and thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0006] Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which: It should be noted that references to "an" or "one" embodiment in this disclosure are not necessarily to the same embodiment, and that these references mean at least one embodiment. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates a system according to one or more embodiments. [Diagram 2] FIG. 1 illustrates an example set of operations for converting inconsistent or non-standard terminology into consistent terminology, according to one or more embodiments. [Diagram 3] FIG. 1 illustrates an example set of operations for identifying categories in a target dataset, according to one or more embodiments. [Figure 4] FIG. 1 illustrates an example set of operations for validating categories identified in a target dataset, according to one or more embodiments. [Diagram 5] FIG. 1 is a block diagram illustrating a computer system according to one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Detailed Description In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in different embodiments. In some instances, well-known structures and devices are described with reference to block diagram form in order to avoid unnecessarily obscuring the present invention.

[0009] 1. Overview 2. System Architecture 3. Category Identification Using ML Pipeline 4. Category Verification 5. Example of embodiment 6. Computer Networks and Cloud Networks 7. Various extensions 8. Hardware Overview 1. Overview One or more embodiments apply multiple independently trained and executed machine learning models to assign a hierarchical classification to the target data item. The system applies both a cluster-based machine learning model and a classification-based machine learning model to determine a candidate hierarchical classification of the target data item. If both candidate hierarchical classifications are the same, the system assigns the candidate hierarchical classification to the target data item. Some embodiments include additional models, each of which may also contribute an independently derived hierarchical classification to the analysis. In these embodiments, if a majority of the models agree (e.g., two-thirds, three-quarters), the hierarchical classification on which the majority of the models agree is associated with the target data item. In some examples, if none of the models agree, the most accurate hierarchical classification that is estimated to be the most accurate is associated with the target data item.

[0010] One or more embodiments described and / or claimed herein may not be included in this Summary section.

[0011] 2. Architecture Overview The embodiments of the system described below are configured to extract categories and / or hierarchical sets of categories (or simply "hierarchies") from target data items using one or more trained machine learning models. In some cases, the system can identify elements of the hierarchy, such as categories at any hierarchical level not previously identified (e.g., parent ("second level") categories, child ("first level") categories). At a high level, the system can achieve these objectives using a series of machine learning models, each trained specifically to perform a particular analysis. The individual trained machine learning models are arranged as "pipelines," such that some of the individual trained machine learning models further process the analysis product output of the previous machine learning model in the pipeline. In some aspects, results from various trained ML models are compared, and a result is extracted based on the comparison.

[0012] Figure 1 illustrates a system 100 according to one or more embodiments. As shown in Figure 1, the system 100 includes clients 102A, 102B, a machine learning application 104, a data repository 122, and an external resource 126. In one or more embodiments, the system 100 may include more or fewer components than those illustrated in Figure 1.

[0013] The components shown in Figure 1 may be local or remote from each other. The components shown in Figure 1 may be implemented in software and / or hardware. Each component may be distributed across multiple applications and / or machines. Multiple components may be combined into one application and / or machine. Alternatively, operations described with respect to one component may be performed by another component.

[0014] The clients 102A, 102B may be web browsers, mobile applications, other software applications that are communicatively coupled to a network (e.g., via a computing device). The clients 102A, 102B may interact with other elements of the system 100 directly or through a cloud service using one or more communication protocols, such as HTTP and / or other communication protocols of the Internet Protocol (IP) suite.

[0015] In some examples, one or more of the clients 102A, 102B are configured to receive and / or generate data items. The clients 102A, 102B can transmit the data items to the ML application 104 for analysis. The ML application 104 can analyze the transmitted data items by applying one or more trained ML models to the transmitted data items, thereby extracting a hierarchy from the data items.

[0016] The clients 102A, 102B may also include user devices configured to render a graphic user interface (GUI) generated by the ML application 104. The GUI may present an interface through which a user triggers the execution of computing transactions, thereby generating data items. In some examples, the GUI may include features that allow a user to view training data, classify the training data, and instruct the ML application 104 to extract a hierarchy from a set of data items, as well as other features of the embodiments described herein. Additionally, the clients 102A, 102B may be configured to allow a user to provide user feedback via the GUI regarding the accuracy of the ML application 104 analysis. That is, a user may use the GUI to label an analysis generated by the ML application 104 as accurate or inaccurate, thereby further revising or validating the training data. In some examples, a user may use the GUI to label a machine learning analysis of subject data generated by the ML application 104, thereby revising aspects of the hierarchy extracted from the set of data items. This latter feature allows a user to label subject data analyzed by the ML application 104, thereby enabling the ML application 104 to update the training of the ML application.

[0017] The ML application 104 of the system 100 may be configured to train one or more ML models using training data, prepare subject data prior to ML analysis, analyze the subject data, and thereby extract a hierarchy from the prepared subject data. As described herein, the ML application 104 may not only extract a hierarchy from the subject data, but may also identify any hierarchical levels of categories and / or subcategories not previously associated with a category.

[0018] The machine learning application 104 includes a feature extractor 108 , a machine learning engine 110 , a front-end interface 118 , and an action interface 120 .

[0019] The feature extractor 108 may be configured to identify characteristics associated with a data item. The feature extractor 108 may generate a corresponding feature vector that represents the identified characteristics. For example, the feature extractor 108 may identify attributes in the training data and / or the "subject" data that the trained ML model is directed to analyze. Once identified, the feature extractor 108 may extract characteristics from one or both of the training data and the subject data.

[0020] The feature extractor 108 can tokenize some data item characteristics into tokens. The feature extractor 108 can then generate a feature vector that includes a set of values, with each value representing a different characteristic token. The feature extractor 108 can use a document-to-vector (colloquially referred to as "doc-to-vec") model to tokenize the characteristics (e.g., as extracted from human-readable text) and generate feature vectors that correspond to one or both of the training data and the subject data. The example doc-to-vec model is provided solely for illustrative purposes. Other types of models may be used to tokenize the characteristics.

[0021] The feature extractor 108 can add other features to the generated feature vector. In one example, the feature vector can be represented as [f1, f2, f3, f4], where f1, f2, f3 correspond to characteristic tokens and f4 is a non-characteristic feature. Examples of non-characteristic features can include, but are not limited to, a label that quantifies a weight (or weights) to be assigned to one or more features of the set of features described by the feature vector. In some examples, the label can indicate one or more classifications associated with the corresponding feature.

[0022] As mentioned above, the system can use the labeled data to train, retrain, and apply its analysis to new (subject) data.

[0023] A feature extractor 108 may optionally be applied to the subject data to generate feature vectors from the subject data that may facilitate analysis of the subject data by other ML models, as described below.

[0024] The machine learning engine 110 of the ML application 104 includes training logic 112 and parsing logic 114. The parsing logic 114 further includes a terminology normalizer 115 and a machine learning pipeline 116.

[0025] In some examples, the training logic 112 receives a set of data items as input (i.e., a training corpus or training dataset). Examples of data items include, but are not limited to, electronically rendered documents and electronic communications. Examples of electronic communications include, but are not limited to, emails, SMS or MMS text messages, electronically transmitted transactions, electronic communications communicated via social media channels, clickstream data, electronic documents, and / or electronically stored text. In one example, a type of electronic document may include a text file in any format (e.g., .txt, .doc, .PDF) that describes requirements such as a job posting or a candidate's work history. In some examples, the data items may be in the form of structured data (e.g., submitted in a browser or computing application format, including PDF format) or unstructured text (e.g., free text documents such as .txt, .doc, .PDF, or other "blob of text" formats).

[0026] In some examples, the training data used by the training logic 112 to train the machine learning engine 110 includes feature vectors of data items generated by the feature extractor 108, as described above.

[0027] The training logic 112 can be in communication with user systems, such as clients 102A, 102B. The clients 102A, 102B can include an interface used by a user to label an electronically stored training data set.

[0028] A machine learning (ML) engine 110 is configured to automatically learn a hierarchical classification (sometimes referred to as an "extracted taxonomy" or "categories") of data items using training logic 112. The trained ML engine 110 can be applied to subject data to analyze one or more characteristics of the subject data. These characteristics can be used according to the techniques described below in the context of FIGS. 2, 4, and 4.

[0029] Types of ML models that may be associated with one or both of the ML engine 110 and / or ML application 104 include, but are not limited to, linear regression, logistic regression, linear discriminant analysis, classification and regression trees, naive Bayes, k-nearest neighbors, learning vector quantization, support vector machines, bagging and random forests, boosting, backpropagation, neural networks, and / or clustering.

[0030] The analysis logic 114 applies the trained machine learning engine 110 to analyze the subject data. The analysis logic 114 can analyze the data items to predict a category of the subject data item based on one or more attributes associated with the subject data item. The analysis logic 114 can use the one or more trained ML models to predict the category of the subject data item, as well as predict one or more categories not currently present in the hierarchical classification.

[0031] 1 as including a normalizer 115 and an ML pipeline 116. Other configurations of the parsing logic 114 may include additional or fewer elements.

[0032] Normalizer 115 functions to associate user-specific terminology with standard terminology, or at least a uniform set of terminology, related to a particular field or subject matter. Applying normalizer 115 to user data may, in some embodiments, improve the accuracy and precision of the results of system 100, and may also enable system 100 to be applied to any of a variety of industries, or even to a variety of separate entities within a particular industry.

[0033] The normalizer 115 first (1) receives (e.g., at the user's command) or independently determines (e.g., by applying a trained ML model) a subject or field of interest to the user, and (2) identifies a corresponding standardized library of terms related to the indicated field of interest. The normalizer 115 can then create associations or "maps" that connect colloquial terms related to the user data to the standardized library. The normalizer 115 can operate on the subject data as well as the training data, so that data items that are relevant to the user but have varying terminology can be analyzed consistently. For example, an entity may post a job using colloquial terms that are specific to the entity, whereas each job seeker may use terminology in their job application that is either (a) different from that of the entity and / or (b) different from each other. The normalizer 115 enables consistent analysis of these data by finding common terminology that is a means of comparing the job requirements in the job postings and the skills of the job seekers.

[0034] In some embodiments, the normalizer 115 may include a trained ML model that generates feature vectors from input data items. In the case of text documents, the trained ML model may be a "doc-to-vec" model that generates vectors from text-based electronic documents and / or files. In some examples, a pre-trained, commercially available doc-to-vec model may be employed (e.g., ORACLE® TALEO®).

[0035] Once the input data item is represented as a feature vector, the normalizer 115 can identify a library of standard terms that correspond to the subject matter of the data item. For example, the system can perform a comparison using a cosine similarity function between one or more feature vectors corresponding to the input data item and portions of the standard terminology library. For example, the system 100 can be in communication with a data store that includes one or more standard terminology / standard taxonomy libraries (e.g., library 124 in data repository 122). In some examples, these subject matter libraries, once represented as feature vectors, can each have a summary or synopsis that can be efficiently compared by the system to the input data vector to select the most similar library. In one particular example, a standard library applicable to human resource applications (talent acquisition, human capital management) is produced by ONET® and may be referred to as the ONET® Standard Occupation Classification (SOC®) system. This particular standard library includes approximately 16,000 different occupational names. Similar standard libraries applicable to various subject fields exist and may be used depending on the particular application.

[0036] Having identified a subject library containing standard terms, the system may then "normalize" the terminology in the subject data item by identifying the terminology in the subject data item and then identifying the standard terminology that corresponds to the terminology. The system may then generate a "normalized" version of the subject data item in which colloquial terms present in the subject data item are replaced with corresponding terms from the standard terminology library. Details of this normalization process are described below in the context of FIG. 2.

[0037] Once the system generates a vector representation of the data items using standard nomenclature, the analysis logic 114 of the system 100 continues to process the data items using one or more trained machine learning algorithms in the ML pipeline 116.

[0038] The ML pipeline 116 may be arranged such that one or more trained machine learning models process the output of a previous trained machine learning model, thereby subjecting the data item to sequential processing steps. In some examples, the ML pipeline 116 may include multiple trained machine learning models that process the data item or the output of a previous machine learning item in series, in parallel, or in a combination thereof. In some embodiments described below, the "voting" operation may select from the output of parallel machine learning model processes that may operate on the same version of the data item and output different analyses.

[0039] In some examples, the ML pipeline 116 can include one or both of supervised and unsupervised machine learning algorithms. In various examples, these different types of machine learning algorithms can be arranged in series (e.g., one model further processes the output of a previous model), in parallel (e.g., two or more different models further process the output of a previous model), or both. As described above, in the case of a parallel processing configuration, the ML pipeline 116 can include a criterion by which to select from the output of parallel branches in the pipeline. In some examples, selected outputs of the segments of the ML pipeline 116 can be further processed by additional serial or parallel ML model configurations. In other examples, selected outputs of the segments of the ML pipeline 116 can be used to provide analytical conclusions (e.g., predictions, recommendations, predicted categories).

[0040] An example method of how an ML pipeline can be used to identify additional, previously unidentified categories within a set of data items is described in the context of Figures 2-4.

[0041] The front-end interface 118 manages the interaction between the clients 102A, 102B and the ML application 104. In one or more embodiments, the front-end interface 118 refers to hardware and / or software configured to facilitate communication between a user and the clients 102A, 102B and / or the machine learning application 104. In some embodiments, the front-end interface 118 is a presentation layer in a multi-tier application. The front-end interface 118 can process requests received from clients and convert results from other application layers into a format that can be understood or processed by the client.

[0042] For example, one or both of the clients 102A, 102B can submit requests to the ML application 104 via the front-end interface 118 to perform various functions, such as labeling training data and / or analyzing subject data. In some examples, one or both of the clients 102A, 102B can submit requests to the ML application 104 via the front-end interface 118 to view a graphical user interface related to the analysis of subject data items against one or more playlists. In yet other examples, the front-end interface 118 can receive user input that reorders individual interface elements.

[0043] The front-end interface 118 refers to hardware and / or software that may be configured to render user interface elements and receive input via user interface elements. For example, the front-end interface 118 may generate web pages and / or other graphical user interface (GUI) objects. A client application, such as a web browser, may access and render an interactive display according to the protocols of the Internet Protocol (IP) suite. Additionally or alternatively, the front-end interface 118 may provide other types of user interfaces that comprise hardware and / or software configured to facilitate communication between a user and an application. Examples of interfaces include, but are not limited to, GUIs, web interfaces, command line interfaces (CLIs), tactile interfaces, and voice command interfaces. Examples of user interface elements include, but are not limited to, check boxes, radio buttons, drop-down lists, list boxes, buttons, toggles, text fields, date and time selectors, command lines, sliders, pages, and forms.

[0044] In one embodiment, the various components of the front-end interface 118 are specified in various languages. The behavior of the user interface elements is specified in a dynamic programming language, such as JavaScript. The content of the user interface elements is specified in a markup language, such as HyperText Markup Language (HTML) or XML User Interface Language (XUL). The layout of the user interface elements is specified in a style sheet language, such as Cascading Style Sheets (CSS). Alternatively, the front-end interface 118 is specified in one or more other languages, e.g., Java, C, C++.

[0045] The action interface 120 may include an API, CLI, or other interface for invoking functions to perform actions. One or more of these functions may be provided by a cloud service or other application that may be external to the machine learning application 104. For example, one or more components of the machine learning application 104 may invoke an API to access information stored in the data repository 122 to use as a training corpus for the machine learning engine 104. It will be understood that the actions performed may vary depending on the implementation.

[0046] In some embodiments, the machine learning application 104 can access external resources 126, such as cloud services. Examples of cloud services can include, but are not limited to, social media platforms, email services, short message services, enterprise management systems, and other cloud applications. The action interface 120 can act as an API endpoint for invoking the cloud service. For example, the action interface 120 can generate outbound requests that follow a protocol that can be consumed by the external resource.

[0047] Additional embodiments and / or examples related to computer networks are described below in Section 6 entitled "Computer Networks and Cloud Networks."

[0048] The action interface 120 can process and transform inbound requests to enable further processing by other components of the machine learning application 104. The action interface 120 can store, negotiate, and / or otherwise manage credentials for accessing external resources. Examples of credentials can include, but are not limited to, digital certificates, encryption keys, usernames, and passwords. The action interface 120 can include the credentials in requests to invoke functions provided by the external resources.

[0049] In one or more embodiments, the data repository 122 is any type of storage unit and / or device for storing data (e.g., a file system, a database, a collection of tables, or any other storage mechanism). Furthermore, the data repository 122 can include many different storage units and / or devices. The many different storage units and / or devices may or may not be of the same type, or may or may not be located at the same physical site. Furthermore, the data repository 122 may be implemented on or run on the same computing system as the ML application 104. Alternatively or additionally, the data repository 122 may be implemented on or run on a computing system separate from the ML application 104. The data repository 122 may be communicatively coupled to the ML application 104 by a direct connection or over a network.

[0050] 1, the data repository 122 includes a standard terminology / taxonomy library 124. As described above, the standard terminology / taxonomy library 124 allows the system 100 to associate colloquial terms from any source (even multiple diverse sources) to a single "standard" term. This conversion of diverse colloquial terms to a single term allows the system to directly compare data items, regardless of the terminology they use, to describe aspects that are captured by the corresponding term in the standard terminology / taxonomy library 124.

[0051] Information relating to the subject data items and training data may be implemented across any of the components in system 100. However, this information may be stored in data repository 122 for clarity and explanation.

[0052] In one embodiment, system 100 is implemented on one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device can refer to a physical device that runs an application or a virtual machine. Examples of digital devices include computers, tablets, laptops, desktops, netbooks, servers, web servers, network policy servers, proxy servers, general purpose machines, special function hardware devices, hardware routers, hardware switches, hardware firewalls, hardware firewalls, hardware network address translators (NATs), hardware load balancers, mainframes, televisions, content receivers, set-top boxes, printers, mobile phones, smartphones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication managers, routers, switches, controllers, access points, and / or client devices.

[0053] 3. Identify hierarchical categories using an ML pipeline FIG. 2 illustrates an example of a set of operations collectively referred to as method 200 for preparing data for subsequent hierarchical classification analysis according to one or more embodiments. Method 200 can be optionally applied to data items to map colloquial or idiosyncratic attributes (e.g., attribute names, attribute values) or other descriptions used by a particular entity to equivalent attributes. This conversion of attributes from idiosyncratic to "standard" attributes optionally allows ML models used in subsequent methods (e.g., methods 300 and 400) to be trained using larger data sets pooled from other entities or data sources regardless of the attribute names used. A larger training data set will thereby improve the accuracy of the model. Using method 200 also allows for more accurate and consistent analysis of target data items.

[0054] One or more of the operations depicted in Figure 2 may be modified, rearranged, or omitted altogether. Thus, the particular sequence of operations depicted in Figure 2 should not be construed as limiting the scope of one or more embodiments. Although method 200 is presented in the context of target data items (as a preliminary step to analysis methods 200 and 300), it will be understood that method 200 may be equally applied to training data.

[0055] Method 200 may begin with receiving one or more target data items that use entity-specific terminology (act 204). Examples of entity-specific terminology include terminology used in data item labels, data item descriptions, attribute names, or attribute values. In other examples, the entity-specific terminology may include electronic document content.

[0056] In a particular example, an organization may generate a job posting that lists several job duties and required skills. The natural language used by the entity to describe the job title, job duties, interactions with other job functions in related departments, minimum qualifications, and required skills may be specific (e.g., idiosyncratic) to that particular entity. Any one or more of the words and phrases used by the entity in the job posting may differ from those used by other entities and / or from more commonly used terminology (e.g., "industry standard").

[0057] Moreover, in this example, many job seekers applying for a job may each use different terminology. This complicates the challenge of identifying candidates for the job, as dozens, hundreds, or thousands of job seekers may use dozens, hundreds, or thousands of different permutations of terminology in response, with few or no permutations directly applicable to the terminology used by the entity offering the job. As will be appreciated, in this particular example, method 200 can be applied to both the job offers themselves and the application data from job seekers, such that all sources of data use terminology that is consistent and conveniently compared.

[0058] Once the target data item is received, the system can access a library to normalize the terminology (operation 208). The library can be a library of industry standard terminology. In some examples, the terminology library can be published by an academic institution, a professional association, or an industry trade association. Continuing with the specific example of the job posting introduced in operation 204, public domain job name libraries are generated by various human resource professional groups, academic institutions, and companies.

[0059] Regardless of source or subject matter, the system can access such libraries as a precursor to converting subject item content terminology (e.g., free text), attribute names and / or attribute values ​​associated with subject data items into uniform "normalized" equivalents.

[0060] The system may then identify normalized terms (e.g., attribute names, attribute values, content) in the library that correspond to the entity-specific terms used for the subject data item (operation 212). In some examples, the library may be represented in vector form to facilitate comparison with the subject data, as described below in the context of operation 224. Operation 212, in one example, may include three operations.

[0061] The system may optionally identify entity-specific terms within the target data item (operation 216), which may be accomplished using a trained ML model to perform a cosine similarity analysis on vector representations of terms and / or attributes within the target data item versus library terms.

[0062] The system may generate a feature vector for the target data item (operation 220). In one example, the feature vector may be generated from any of the identified entity-specific terms optionally identified in operation 216. In another example, the system may generate the feature vector based on individual terms and / or permutations of terms within the target data item. In one example, the system uses a "doc-to-vec" trained machine learning model to generate the feature vector. In one example, the system may use a pre-trained doc-to-vec trained machine learning model such as "Taleo"®.

[0063] The system can train the doc-to-vec machine learning model using commercially available or public training datasets and optionally supplement the training by using training data specific to the entity and / or subject matter that will ultimately be analyzed. For example, the system can train the doc-to-vec using generic (i.e., non-subject-specific) training datasets and / or commercially available or public training datasets that are subject-specific (e.g., human resources, physical sciences, finance). In one example, to improve the accuracy of the model applied to the target data items for a particular entity, a supplemental training dataset that is specific to the terminology used by the entity is used to supplement the generic training dataset. In another example, the supplemental training dataset may be specific to a particular subject matter field that is specific to the entity (e.g., the terminology used by the entity in the human resources, finance industry).

[0064] In some examples, a data item can be represented as a vector containing tokens of most words and / or phrases in the target data item, or as a set of vectors, each of which corresponds to a word and phrase (e.g., a group of two or more words). In some examples, the phrase vectors and / or tokens can include any number of permutations of words in the target data item. In some examples, the word permutations can be delimited by the system recognizing portions of speech or formatting that indicate separation of thoughts. For example, transition words such as "and" and "or" and formatting such as semicolons, periods, bullets, etc. can prevent words separated by these features from being combined into tokens or vectors. This can reduce the number of comparisons performed by the system, thereby increasing the overall analysis efficiency of the system without removing substantial content. In some examples, the system can omit definite articles, indefinite articles, and other portions of speech that are useful in written or oral communication, but may not be useful when performing feature vector analysis as described above.

[0065] The system can compare the vector representation of the normalized terms in the library with the vector representation of the entity-specific terms in the target data item in operation 224. Operation 224 can identify the entity-specific terms and their corresponding synonyms in the library of normalized terms. In some examples, this can be accomplished when the system applies a cosine similarity analysis. The system can identify terms in the target data and the library as similar when the value generated by the cosine analysis is above a threshold (e.g., greater than 0.5, greater than 0.75, greater than 0.8). In other examples, the system can apply a K-nearest neighbor trained machine learning model to identify similar terms. Regardless of the comparison algorithm used, the system can generate a version of the vector representing the data item in which colloquial terms (words / phrases) are replaced with terms from a standard terminology library.

[0066] Upon identifying similar terms between the subject data and the library, the system generates a mapping between the entity-specific terms and the normalized terms (operation 228). Although the term "mapping" is used, it will be understood that this simply refers to a measure or other indication of correspondence between different but similar terms.

[0067] The system may then apply the mapping to the target data item, thereby converting the entity-specific terms to normalized terms (operation 232). The system may generate a version of the feature vector that represents the target data item, except that the feature values ​​correspond to the normalized terms instead of the entity-specific terms.

[0068] Upon completing method 200 (which may not be necessary in all circumstances), the system may then execute method 300, the operations of which are illustrated in Figure 3. One or more of the operations illustrated in Figure 3 may be modified, rearranged, or omitted altogether, and thus the particular sequence of operations illustrated in Figure 3 should not be construed as limiting the scope of one or more embodiments.

[0069] Applying method 300, the system may determine or otherwise identify hierarchical categories within one or more (normalized) target data items. In some examples, the system may use method 300 to identify new hierarchical categories presented by one or more target data items, even when such categories are not already part of an existing hierarchy. In various examples, applying method 300 may identify any one or more of categories at any hierarchical level, whether at the leaf node level (i.e., the "first" level), at the direct parent level to the leaf node level (i.e., the "second" level "above" (i.e., more general) than the first level), or at a higher level.

[0070] The method 300 includes receiving a target data item for analysis (operation 302). Examples of target data items include electronic documents in any number of formats. In one example of the variety of data types accommodated by the system, examples of these various formats include unstructured data (operation 304) or structured data (operation 306). Examples of unstructured data 304 include data items that include free text with few constraints on permissible words, values, formats, syntax, and / or punctuation. A particular type of free text analysis that the system can employ includes the so-called "bag of words" or, as used in the Python programming language, "blob of words." The ability of the system to process unstructured text is particularly useful, for example, when receiving documents or data items that are generated separately from individual sources. Continuing with the job search example, the system can process unstructured text-based resumes that describe job seekers' experiences using different terminology as well as different formats, different document organization schemes, and the like. As noted above, these unstructured data items may be submitted via unstructured web or computing application forms, emails, social media posts, SMS or MMS text messages, text edited documents, and the like.

[0071] Examples of structured data (act 306) include data submitted via structured web or computing application forms, PDF® “fillable” forms, etc. In some examples, the system can identify fields within the structured data items and / or structured data item metadata by field name. These identifying features, individually or separately, can instruct the system as to expected values ​​for each field, the type of ML processing to be applied, etc.

[0072] Regardless of the format in which the target data item is received, the system can process the target data item into a "normalized" form using method 200. The system can convert the target data item into a vector representation to facilitate further analysis by the system. The system can perform the normalization conversion process according to method 200 before, during, or after converting the data item into a corresponding feature vector.

[0073] Method 300 includes training one or more machine learning (ML) models for use by the system to progressively analyze one or more target data items (operation 308). In the example of Figure 3, the system employs three machine learning models, although any number of one or more trained machine learning models may be used to process data items in accordance with the present disclosure.

[0074] In some examples, the first trained ML model (operation 312) can include a trained machine learning model trained to identify categories based solely on the text of the subject data item. In some examples, the first trained ML model is configured to identify a broader set of potential categories within the subject data item than those present in a narrower but confirmed accurate whitelist. Whitelists are described below in the context of operation 324. In this manner, the first trained ML model can detect new categories within one or more subject data items that are not present in the whitelist.

[0075] In some examples, the first trained ML model may be a "named entity recognizer" or NER trained ML model. In some examples, the system runs any kind of NER model, one example being the Stanford NER model. The NER model analyzes individual words and phrases (i.e., permutations of words) in a document. Based on its training, the NER model can determine whether any of the individual words and / or phrases (or other detectable attributes) are associated with a corresponding category. In some examples, the first trained ML model may be trained using a list of manually selected and labeled categories or manually selected and labeled data items. In other examples, the NER model may be trained using a trained neural network that provides the NER model with category and contextual data.

[0076] In some examples, the second trained ML model is a classifier model (operation 314). In other examples, the second trained ML model is a neural network or a "deep learning" model. In either case, the second trained ML model is trained to identify categories and parent categories associated with data item attributes, individual tokens and / or permutations of tokens (e.g., generated from words / text by a doc-to-vec model) within the target data item. Regardless of the type of model, the second trained ML model analyzes the output received from the first trained ML model and uses its own classification analysis to determine whether the categories identified by the first trained ML model were correctly identified.

[0077] The second trained ML model may be trained using supervised learning techniques. For example, the second trained ML model may be trained using manually labeled data, i.e., data items where words and phrases are designated as categories by labeling them with parent categories. For example, a whitelist (described below), along with the identified categories and parent category labels of the whitelist, may be used to train the second trained ML model to identify correct associations between data item attributes and categories. Similarly, attributes used to generate the whitelist but that are not associated with a category may be labeled to indicate that there is no category associated with such attribute. This provides negative training examples for the second ML model. In another example, similar to that described above, the second trained ML model may be trained using a neural network trained to identify categories.

[0078] In some examples, the third trained ML model (operation 316) is an unsupervised machine learning model, such as a clustering model. In one embodiment, the third trained ML model can be a K-means clustering model. In one embodiment, the third trained ML model can be trained using a whitelist (described below). The third trained ML model can use the whitelist data to generate clusters of vectors that represent known correct categories. In other embodiments, the third trained ML model can use unlabeled training data to generate multiple clusters that represent categories. In some examples, the third trained ML model can use any one or more of cosine similarity, K-means, or K-nearest neighbor algorithms to identify clusters in the training data.

[0079] Prior to applying any of the trained ML models, the system identifies categories associated with the target data item by referencing a "whitelist" of known correct categories and their associated attributes (operation 324). The whitelist may include a list of vectors representing categories known to be valid. In some examples, the whitelist may also include associated attributes (e.g., words / phrases, other attributes, and / or tokens thereof). However, as will be understood in light of this disclosure, the whitelist may not include all correct categories actually present in one or more data items. The following operations of method 300 (and method 400) are configured to identify additional categories that are present in the data but not reflected on the whitelist. Additionally, method 300 (and method 400) also includes an operation that preserves the integrity of the identified categories by reducing the likelihood that erroneous or incorrect categories are predicted from the data.

[0080] In some examples, the whitelist of categories may be specific to an entity or subject field. In some examples, the whitelist may be prepared manually or generated by trained machine learning model analysis (e.g., doc-to-vec, neural networks) accessed from a third party entity (e.g., industry trade associations, professional associations, academic institutions, corporate entities). As discussed above, the whitelist includes categories known to be correct that may inappropriately exclude categories that should be on the whitelist.

[0081] In one example, the whitelist may be generated by parsing a dataset of data items and labeling each category within each data item of the dataset. In some examples, the category labels are binary numbers that indicate whether an attribute value (or feature vector token) is categorical or non-categorical. As noted above, in one example, the category labels may be applied to individual tokens and / or individual feature vectors that represent corresponding permutations of words and / or attributes within the data items.

[0082] In other examples, the category labels are not binary, but instead label the identified categories with the associated more general parent (or second level) category. Labeling the child (equivalently leaf or first) level categories identified in the data items to be included on the whitelist with the corresponding second level categories has the advantage of associating hierarchical information with the identified first level categories by the labels themselves. This process can be repeated for each first level category identified in the training documents. This differs from labeling the entire training documents with a single label. The identified first level categories and their labels are extracted from the training data and compiled to collectively form the whitelist.

[0083] For example, continuing with the job listing example, the identified first level category may be identified as the job skills of a "hiring manager." Examples of training documents used to identify the first level category and the corresponding second level category may include reference documents used by the entity (e.g., commercially available or provided by an industry trade association), a set of resumes used for machine learning training purposes, an entity-specific job skill list or set of job skill lists, etc. The label associated with the first level category (job skills) may indicate a second level category of "human resources operations" (an organizational function). This association of "hiring manager" and "human resources operations" in the data label provides hierarchical information to the system. Additionally, training documents (e.g., resumes, job requirements lists) may include many skills, each of which is labeled. For example, the same training document that includes proficiencies in various computer programs may be identified as a first level category and labeled with a corresponding second level category ("computer skills"). Similarly, the same training document may include accounting, and the system would identify accounting as the first level category and label it with a second level category ("financial work") that corresponds to accounting.

[0084] The system may apply the first trained machine learning model as part of a process to predict a hierarchical classification of the target data items (operation 328). In one example, the first trained ML model may identify categories by associating attributes, such as words, phrases, and / or permutations of words (or more precisely, their corresponding feature vectors / tokens) in the target data items with categories. These attributes and permutations of attributes are analyzed to determine whether the attributes and / or permutations are associated with a category.

[0085] In some examples, the NER model output is the identified category, the parent category label associated with the category, and part of the context of the data item in which the category is identified. By detecting candidate categories and passing the category and the context of the data item in which the candidate category was detected to a subsequent model, the overall accuracy of the subsequent analysis can be improved.

[0086] The approach using the first trained ML model, particularly the NER model, can cause the first trained ML model to generate "false positive" category identifications. That is, the first trained ML model may erroneously identify an aspect in one or more data items as a category, but the aspect is not a category. Incorrect identification of a category is operationally problematic because it generates an incorrect hierarchical classification. If an error is introduced into the hierarchical classification, the error may grow over time as more data items are analyzed. This may cause time-consuming manual correction. To prevent or reduce the possibility that the system identifies an incorrect category, the analysis output of the first trained ML model may be subsequently processed to remove these "false positive" categories by the collective operation of the second trained ML model and the third trained ML model. These operations are described below.

[0087] The second trained ML model (operation 314) and the third trained ML model (operation 316) can together determine whether the candidate category identified by the first trained ML model is a false positive or a correct result. The system analyzes the results from the first trained ML model (operation 328) with both the second trained ML model at operation 332 and the third trained ML model at operation 336.

[0088] In one example, the system applies the second trained ML model to the output of the first trained ML model, where the output may include the category, the parent category label, and the corresponding context (operation 332). The system may use analysis of the second trained ML model to determine, in part, the accuracy of the first ML model.

[0089] In some examples, the second trained ML model receives target data items previously analyzed by the first trained ML model and classifies words and / or phrases (e.g., permutations of words) according to categories and parent categories. In some examples, the second trained ML model analyzes phrases of words as a way to place noun words in context, thereby better determining the meaning and / or importance attached to particular attributes (e.g., words or phrases). In some examples, the second trained ML model can increase the computational efficiency of the model by omitting certain words and / or portions of speech that are unlikely to be associated with substantive content, as described above.

[0090] In some embodiments, the second trained ML model determines whether the category is properly associated with the parent category based on the training of the model. For example, the second trained ML model can use the model's classifier-based algorithm to determine a hierarchical classification of the data items based on the data item attributes. If the identified category and the parent category match each other, the second trained ML model labels the category and the parent category with a label indicating that the category is correct (i.e., "1"). Additionally or alternatively, the second ML model can determine whether the hierarchical classification generated by the second ML model matches that generated by the first ML model. The result also generates a label indicating a match. If the identified category and the parent category do not match each other or do not match the result of the first trained ML model, the second trained ML model labels the category and the parent with a label indicating that the category is incorrect (i.e., "0").

[0091] The results of these analyses of the related category / parent category data can then be passed to subsequent stages of the ML pipeline for analysis in combination with the results of a third trained ML model, described below.

[0092] The system may apply a third trained ML model (operation 336). For embodiments in which the third trained ML model is a clustering model, the model may identify the centroid of each cluster and calculate a variability (or noise) value for one or more dimensions that define each cluster. In some embodiments, the model calculates cluster variability using a silhouette coefficient to quantify the variability value of each cluster.

[0093] The third trained ML model may then evaluate the output of the first trained ML model, in some examples, by associating the vector representation of the category generated by the first trained ML model with one or more clusters. Once assigned, the third trained ML model calculates the Silhouette Coefficient of the cluster with the newly added vector. If the Silhouette Coefficient increases upon addition of the output vector of the first trained ML model or is otherwise above a threshold, thereby indicating an increase in cluster variability, the system determines that the output vector should not be associated with that cluster. However, if the Silhouette Coefficient decreases upon addition of the output vector of the first trained ML model or is otherwise below a threshold, thereby indicating a decrease in cluster variability, the system determines that the output vector is properly associated with that cluster. In this way, the third trained ML model independently determines the relevance (i.e., hierarchical classification) of parent categories to categories identified by the first trained ML model even if the category is not a previously identified category (e.g., on a whitelist). This process may be repeated with each cluster for each output vector of the first trained ML model.

[0094] In another example, the third trained ML model can perform a cosine similarity analysis to determine whether the newly added vector is properly associated with vectors representing data items in the cluster. If the cosine value of the vectors comparison is above a threshold, the newly added vector (representing a data item) is properly associated with the cluster. If the cosine value of the vectors comparison is below the threshold, the newly added vector (representing a data item) is not properly associated with the cluster.

[0095] The system then determines whether one or more of the categories identified by the first trained ML model and analyzed separately by the second trained ML model and the third trained ML model are potentially valid categories (operation 340). In one example, the system detects whether any category and corresponding parent category identified by the first trained ML model are predicted by both the second trained ML model and the third trained ML model. An equivalent description of this process is that both the second trained ML model and the third trained ML model "vote" for a particular predicted category (and predicted parent category) based on their own respective analyses. In one example, if both the second trained ML model and the third trained ML model (a) identify the same category, and (b) identify the category as being appropriately related to the identified parent category (e.g., by cosine similarity analysis, clustering analysis, neural network analysis, etc.), the category and parent category are identified as potentially valid.

[0096] If both the second trained ML model and the third trained ML model predict a particular category and corresponding parent category within the target data item, the system passes the predictions to method 400 for validation (operation 344).

[0097] In some examples, the whitelist of categories and corresponding parent categories may be combined with the categories and parent categories identified by the second and third trained ML models. This combination of options is illustrated in FIG. 3 by the dashed arrow connecting operation 324 and operation 344.

[0098] However, if a category identified by the first trained ML model is not identified by both the second trained ML model and the third trained ML model, the category is rejected (operation 348). Rejected categories are not passed to method 400 for validation.

[0099] Validate your categories 4 illustrates an example set of operations, collectively referred to as method 400, for validating the categories identified in method 300 in preparation for updating the hierarchy of the classification with newly identified correct categories, according to one or more embodiments. One or more of the operations illustrated in FIG. 4 may be modified, rearranged, or omitted altogether. Thus, the particular sequence of operations illustrated in FIG. 4 should not be construed as limiting the scope of one or more embodiments.

[0100] Method 400 may begin with receiving a composite set of categories (operation 404), generated upon conclusion of method 300. As described above, the composite set of categories may include categories from the whitelist and corresponding parent categories as identified by both the second trained ML model and the third trained ML model. The composite set of categories may represent the categories as feature vectors, as described above. In some embodiments, each category feature vector may identify a source (e.g., by parameter value or label) from which the vector was generated. These sources include the whitelist, the first trained ML model, the second trained ML model, or the third trained ML model.

[0101] The system determines whether the particular category arises from a whitelist or from a joint analysis of the second and third trained ML models (operation 408). For example, the system may analyze a feature vector associated with the particular category and identify parameter values ​​and / or labels within the feature vector that indicate the source of the feature vector. If the source of the feature vector is a whitelist of categories, the process proceeds to operation 424, where the system generates a final set of categories and parent categories. Details of operation 424 are described in more detail below.

[0102] If the system determines that the source of the category is not on the whitelist, then the system analyzes the category using two trained machine learning models, in some embodiments, one of the models is a classifier-type trained machine learning model and another of the models is a clustering-trained machine learning model.

[0103] The trained classifier ML model may be applied to the categories to determine whether the categories are likely to be valid (operation 412). In some examples, the classifier model used in operation 412 may be trained using a whitelist as described above. In some embodiments, the trained classifier model used in operation 412 may be a multi-class machine learning model capable of identifying categories, parent categories, grandparent categories, etc. In other examples, the trained classifier ML model may be a trained deep learning (or neural network) machine learning model.

[0104] The system can use operation 412 to determine whether the child and parent categories identified for the data item by method 300 are properly associated with each other. In some examples, this can be described as the child and parent categories being "related" to each other. In some examples, the multi-class classifier model can perform a cosine similarity analysis to determine whether the parent and child classifications have a similarity above a threshold. If the identified classifications (or categories) are above the threshold, the system determines that the parent and child categories are properly associated with each other. If the identified classifications (or categories) are below the threshold, the system determines that the parent and child categories are not properly associated with each other.

[0105] The system also analyzes the received categories using the trained cluster-based ML model (operation 416). The trained cluster-based ML model may be a reapplication of the third trained machine learning model 316 in some embodiments. In some embodiments, the trained cluster-based ML model may simply perform an analysis similar to that described above in the context of operation 336. That is, the trained cluster-based ML model may cluster the categories received in operation 404 using a K-means clustering algorithm. The system may provisionally include the newly identified category in the cluster and generate a silhouette coefficient for the cluster that quantifies a measure of the variability and dispersion of the cluster before and after including the newly identified category. As described above, if the silhouette coefficient of the cluster increases (or exceeds a threshold) to indicate additional variability within the cluster upon the addition of the newly identified category, the category is rejected from that cluster. That is, the parent and child categories identified by method 300 to be hierarchically classified together are not properly associated with each other. If the silhouette coefficient of the cluster decreases or remains the same (or is otherwise below a threshold) and represents less or equal variability within the cluster upon addition of the newly identified category, the category is associated with that cluster. In other words, the parent and child categories are appropriately hierarchically related to each other. This process can be repeated for each cluster and each newly identified category until each newly identified category is assigned to a cluster or rejected by the trained cluster-based ML model.

[0106] Method 400 then proceeds to operation 420, where the aggregated analysis results of the three machine learning models are analyzed in a "voting" process to determine whether to include or not include the identified category in the hierarchy, which may equivalently be referred to as "validating" the category.

[0107] The results of the analysis of the three machine learning models analyzed are the first trained ML model 312 described in operation 328, and the trained ML models described in operations 412 and 416. A category is verified by determining whether any two of the three trained ML models identify a particular category (optionally in relation to a parent category).

[0108] If any two of these trained ML models make the same prediction at operation 420, the category is verified (operation 424). In some examples, the system may add the newly verified category to a whitelist, thereby expanding the list of known correct categories.

[0109] If none of the three models agree with each other in predicting a particular category (i.e., the category is predicted by only one of the three models), the system may resolve this discrepancy (operation 428) by accepting the prediction of the classifier ML model applied in operation 412. The system may select the prediction of the classifier ML model based on an estimation that the classifier ML model is the most accurate of the three models performing this voting process in operation 420.

[0110] At operation 432, the system determines whether the category being discussed was predicted by the classifier model at operation 412. If the category being discussed was predicted by the classifier model, the category is included in the final set of categories according to operation 424. If the category being discussed was not predicted by the classifier model, but instead was predicted by one of the other two models, the category is rejected (operation 436).

[0111] 5. Example of embodiment Detailed examples are described below for clarity. The components and / or operations described below should be understood as one particular example that may not be applicable to a particular embodiment. Thus, the components and / or operations described below should not be construed as limiting the scope of any of the claims.

[0112] In one embodiment, the above system may be applied to human resource management, such as talent acquisition. As noted above, a position by an entity may result in receiving resumes for multiple job seekers. The system may perform a terminology "normalization" operation so that the system can perform subsequent operations on the vector representation of the resume content that uses consistent terminology.

[0113] The system can compare the vector representation of words and / or phrases in the received resume to a whitelist of skills and associated business functions. The skills in this example correspond to child categories and the associated business functions correspond to parent categories. Examples of job skills and corresponding business functions include heating system maintenance and facilities management, accounting and finance, Java programming and engineering, and employee supervision and management, respectively.

[0114] Once any whitelisted skills and business functions are identified, the system applies the named entity recognizer trained ML model to broadly identify candidate skills and associated business functions. As described above, the system runs the named entity recognizer trained ML model to identify any potential skills and corresponding business functions that are not on the whitelist. For example, the system may determine two instances of skills and business functions that are not on the whitelist: (1) loan management / finance and (2) oil rig operations / business operations. In these instances, the identified first tier that is not on the whitelist is a suitable association between skills and business functions, and the identified second tier is not a suitable association.

[0115] The system then applies the classification-based trained ML model and the cluster-based trained ML model to the first identified hierarchy and the second identified hierarchy. In one example, the classification-based trained ML model identifies both identified hierarchies as appropriate, and the cluster-based trained ML model correctly identifies the first hierarchy as appropriate and the second hierarchy as inappropriate. Because the two models agree on the first identified hierarchy and do not agree on the second identified hierarchy, only the first identified hierarchy is passed on to subsequent work in the ML pipeline for validation. The second identified hierarchy is rejected.

[0116] The first identified hierarchy of "Loan Management / Finance" is analyzed using another trained multi-class classifier-based ML model and a clustering-based ML model as described above. Both of these models perform their analysis and both determine that the relevance is appropriate. Additionally, the named entity recognizer has the first identified hierarchy generated initially and also agrees with this analysis. As described above, only two of the three ML models need to agree at this stage of processing to validate the hierarchy.

[0117] In some examples, validation of the hierarchy with the association of the other trained multi-class classifier-based ML model with the clustering-based ML model may be based on independent predictions performed by both models, or may be determined based on similarity analysis and / or silhouette coefficient analysis to ensure that the job skills and business functions are sufficiently similar to each other to justify validation. Once validated, the first identified hierarchy for "Loan Management / Finance" may be added to a whitelist for future use.

[0118] 6. Computer Networks and Cloud Networks In one or more embodiments, a computer network provides connectivity among a set of nodes. The nodes can be local and / or remote from one another. The nodes are connected by a set of links. Examples of links include coaxial cable, unshielded twisted cable, copper cable, optical fiber, and virtual links.

[0119] A subset of nodes implements computer networks. Examples of such nodes include switches, routers, firewalls, and network address translators (NATs). Another subset of nodes uses computer networks. Such nodes (also called "hosts") can run client processes and / or server processes. A client process makes a request for a computing service (such as running a particular application and / or storing a particular amount of data). A server process responds by performing the requested service and / or returning the corresponding data.

[0120] A computer network may be a physical network that includes physical nodes connected by physical links. A physical node is any digital device. A physical node may be a hardware device of a specific function, such as a hardware switch, a hardware router, a hardware firewall, a hardware NAT, etc. Additionally or alternatively, a physical node may be a general-purpose machine configured to run various virtual machines and / or applications that perform respective functions. A physical link is a physical medium that connects two or more physical nodes. Examples of links include coaxial cable, unshielded twisted cable, copper cable, and optical fiber.

[0121] A computer network may be an overlay network. An overlay network is a logical network implemented on top of another network (such as a physical network). Each node in an overlay network corresponds to a respective node in the underlying network. Each node in an overlay network is therefore associated with both an overlay address (which addresses the overlay node) and an underlay address (which addresses the underlay node that implements the overlay node). Overlay nodes may be digital devices and / or software processes (such as virtual machines, application instances, or threads). Links connecting overlay nodes are implemented as tunnels through the underlying network. Overlay nodes on either side of the tunnel treat the underlying multi-hop path between the overlay nodes as a single logical link. Tunneling is achieved by encapsulation and decapsulation.

[0122] In one embodiment, a client may be local to a computer network and / or remote from the computer network. A client may access a computer network through another computer network, such as a private network or the Internet. A client may communicate a request to a computer network using a communication protocol, such as the Hypertext Transfer Protocol (HTTP). The request is communicated through an interface, such as a client interface (e.g., a web browser), a program interface, an application programming interface (API), or the like.

[0123] In one embodiment, a computer network provides connectivity between clients and network resources. The network resources include hardware and / or software configured to execute server processes. Examples of network resources include processors, data storage devices, virtual machines, containers, and / or software applications. The network resources are shared among multiple clients. The clients request computing services from the computer network independently of one another. The network resources are dynamically allocated to requests and / or clients on an on-demand basis. The network resources allocated to each request and / or client may be scaled up or down based on, for example, (a) the computing services requested by a particular client, (b) the aggregated computing services requested by a particular tenant, and (c) the requested aggregated computing services of the computer network. Such a computer network may be referred to as a "cloud network."

[0124] In one embodiment, a service provider offers a cloud network to one or more end users. Various service models may be implemented by the cloud network, including, but not limited to, Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). In SaaS, the service provider offers end users the ability to use the service provider's applications running on the network resources. In PaaS, the service provider offers end users the ability to deploy custom applications on the network resources. The custom applications may be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, the service provider offers end users the ability to set up the processing, storage, network, and other basic computing resources provided by the network resources. Any application may be deployed on the network resources, including operating systems.

[0125] In one embodiment, various deployment models may be implemented by the computer network, including, but not limited to, private cloud, public cloud, and hybrid cloud. In a private cloud, network resources are set up for exclusive use by a particular group of one or more entities (the term "entity" as used herein refers to a corporation, organization, person, or other entity). The network resources may be local to and / or remote from the premises of the particular group of entities. In a public cloud, cloud resources are set up for multiple entities (also called "tenants" or "customers") that are independent of each other. The computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network may be referred to as a "multi-tenant computer network." Several tenants may use the same particular network resource at different times and / or at the same time. The network resources may be local to and / or remote from the tenant's premises. In a hybrid cloud, the computer network comprises a private cloud and a public cloud. The interface between the private cloud and the public cloud allows for data and application portability. Data stored in a private cloud and data stored in a public cloud may be exchanged through an interface. Applications implemented in a private cloud and applications implemented in a public cloud may have dependencies on each other. Calls from applications in a private cloud to applications in a public cloud (and vice versa) may be made through an interface.

[0126] In one embodiment, tenants of a multi-tenant computer network are independent of one another. For example, the business or operations of one tenant may be isolated from the business or operations of another tenant. Different tenants may require different network requirements of the computer network. Examples of network requirements include processing speed, data storage, security requirements, performance requirements, throughput requirements, latency requirements, resilience requirements, quality of service (QoS) requirements, tenant isolation, and / or consistency. The same computer network may need to implement different network requirements required by different tenants.

[0127] In one or more embodiments, in a multi-tenant computer network, tenant isolation is implemented to ensure that applications and / or data of different tenants are not shared with each other. Various tenant isolation techniques may be used.

[0128] In one embodiment, each tenant is associated with a tenant ID. Each network resource in a multi-tenant computer network is tagged with a tenant ID. A tenant is permitted access to a particular network resource only if the tenant and the particular network resource are associated with the same tenant ID.

[0129] In one embodiment, each tenant is associated with a tenant ID. Each application implemented by the computer network is tagged with a tenant ID. Additionally or alternatively, each data structure and / or data set stored by the computer network is tagged with a tenant ID. A tenant is granted access to a particular application, data structure, and / or data set only if the tenant and the particular application, data structure, and / or data set are associated with the same tenant ID.

[0130] As one example, each database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID may access the data of a particular database. As another example, each entry in a database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID may access the data of a particular entry. However, a database may be shared by multiple tenants.

[0131] In one embodiment, the subscription list indicates which tenants are authorized to access which applications. For each application, a list of tenant IDs of tenants authorized to access the application is stored. A tenant is granted access to a particular application only if the tenant's tenant ID is included in the subscription list corresponding to the particular application.

[0132] In one embodiment, network resources (such as digital devices, virtual machines, application instances, threads, etc.) corresponding to different tenants are isolated to tenant-specific overlay networks maintained by a multi-tenant computer network. As an example, packets from any source device in a tenant overlay network can only be sent to other devices in the same tenant overlay network. An encapsulation tunnel is used to prohibit any transmission from a source device on a tenant overlay network to devices in other tenant overlay networks. Specifically, a packet received from a source device is encapsulated in an outer packet. The outer packet is sent from a first encapsulation tunnel endpoint (communicating with a source device in a tenant overlay network) to a second encapsulation tunnel endpoint (communicating with a destination device in a tenant overlay network). The second encapsulation tunnel endpoint removes the encapsulation of the outer packet to obtain the original packet sent by the source device. The original packet is sent from the second encapsulation tunnel endpoint to a destination device in the same particular overlay network.

[0133] 7. Various extensions Embodiments are directed to systems having one or more devices including a hardware processor and configured to perform any of the operations described herein and / or recited in any of the claims below.

[0134] In one embodiment, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more hardware processors, cause performance of any of the operations described herein and / or recited in any of the claims.

[0135] Any combination of the features and functionality described herein may be used in accordance with one or more embodiments. In the foregoing specification, the embodiments are described with reference to many specific details that may vary depending on the implementation. The specification and drawings should accordingly be evaluated in an illustrative and not a limiting sense. The sole and exclusive indication of the scope of the invention, and what the applicant intends as the scope of the invention, is the literal and equivalent scope of the set of claims originating from this application, in the specific form in which such claims arise, including any subsequent amendments.

[0136] 8. Hardware Overview According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hardwired to execute the techniques or may include one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), network processing units (NPUs), and other digital electronic devices that are persistently programmed to execute the techniques, or may include one or more general-purpose hardware processors that are programmed to execute the techniques according to program instructions in firmware, memory, other storage, or combinations thereof. Such special-purpose computing devices may also combine custom hardwired logic, ASICs, FPGAs, or NPUs with custom programming to realize the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, or other devices that incorporate hardwired and / or program logic to implement the techniques.

[0137] For example, Figure 5 is a block diagram illustrating a computer system 500 in which embodiments of the invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor.

[0138] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored on a non-transitory storage medium accessible to processor 504, render computer system 500 into a special-purpose machine that is customized to perform the operations specified in the instructions.

[0139] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk or optical disk, is provided and coupled to bus 502 for storing information and instructions.

[0140] The computer system 500 may be coupled via a bus 502 to a display 512, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to the bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor control device 516, such as a mouse, trackball, cursor direction keys, etc., for controlling cursor movement on the display 512 as well as communicating directional information and command selections to the processor 504. This input device typically has two degrees of freedom along two axes, a first axis (e.g., x) and a second axis (e.g., y), that allow the device to specify a position in a plane.

[0141] Computer system 500 may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, causes computer system 500 to be a special purpose machine or programs computer system 500. According to one embodiment, the techniques herein are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in conjunction with or in combination with software instructions.

[0142] The term "storage medium" as used herein refers to any non-transitory storage medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media can comprise non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tapes or other magnetic data storage media, CD-ROMs, other optical data storage media, any physical media with patterns of holes, RAM, PROMs, and EPROMs, flash EPROMs, NVRAMs, other memory chips or cartridges, content addressable memories (CAMs), and ternary content addressable memories (TCAMs).

[0143] Storage media are distinct from, but may be used in conjunction with, transmission media. Transmission media involves transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0144] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into a dynamic memory of the remote computer and send the instructions over a telephone line using a modem. A modem local to the computer system 500 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on the bus 502. The bus 502 carries the data to main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by main memory 506 may optionally be stored on storage device 510 either before or after execution by the processor 504.

[0145] Computer system 500 also includes a communications interface 518 coupled to bus 502. The communications interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, the communications interface 518 may be an Integrated Services Digital Network (ISDN) card, a cable modem, satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, the communications interface 518 may be a local area network (LAN) card that provides a data communication connection to a compatible LAN. A wireless link may also be implemented. In any such implementation, the communications interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0146] Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connection through a local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. ISP 526 in turn provides data communication services through the world-wide packet data communication network now commonly referred to as the "Internet" 528. Both local network 522 and Internet 528 use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and through communication interface 518 on network link 520, which carry the digital data to and from computer system 500, are exemplary forms of transmission media.

[0147] Computer system 500 can send messages, including program code, and receive data through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0148] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage for later execution.

[0149] In the foregoing specification, embodiments of the invention have been described with reference to many specific details that may vary depending on the implementation. The specification and drawings should accordingly be evaluated in an illustrative and not a limiting sense. The sole and exclusive indication of the scope of the invention, and what the applicant intends as the scope of the invention, is the literal and equivalent scope of the set of claims originating from this application, in the specific form in which such claims arise, including any subsequent amendments.

Claims

1. A program including instructions that, when executed by one or more hardware processors, cause the performance of operations, said operations including: training a cluster-based machine learning model and a classification-based machine learning model to associate data items with corresponding categories within a set of hierarchical categories; receiving a first subject data item to be categorized into a corresponding first category at a first level of the set of hierarchical categories; applying the cluster-based machine learning model to the first target data items to generate a first hierarchical classification, wherein applying the cluster-based machine learning model comprises: identifying a first candidate cluster for the first target data item from among a plurality of clusters; the first candidate cluster corresponds to the first hierarchical classification, and the plurality of clusters are determined by the cluster-based machine learning model based on a first set of training data; (1) comparing a first variance value of the first candidate cluster having the target data item with (2) a second variance value of the first candidate cluster not having the target data item to calculate a difference value; in response to determining that the difference value is less than a threshold difference value, identifying the first hierarchical classification corresponding to the first candidate cluster as a first candidate classification for the first subject data item; applying the classification-based machine learning model to the first target data items to generate a second hierarchical classification, wherein applying the classification-based machine learning model comprises: analyzing attributes corresponding to the first target data item to identify the second hierarchical classification as a second candidate classification for the first target data item; and, in response to determining that the first hierarchical classification determined by the cluster-based machine learning model and the second hierarchical classification determined by the classification-based machine learning model are the same, assigning one of the first hierarchical classification or the second hierarchical classification to the first subject data item as the first category at the first level of the set of hierarchical categories. program.

2. The method of claim 1 , wherein identifying the first candidate cluster is based on an attribute of a first target data item.

3. 2. The method of claim 1, wherein the classification-based machine learning model comprises a neural network, and analyzing the attributes corresponding to the first target data item comprises applying the neural network to the attributes corresponding to the first target data item.

4. The operation comprises: applying the cluster-based machine learning model to second target data items to generate a third hierarchical classification; applying the classification-based machine learning model to the second target data items to generate a fourth hierarchical classification; and and, in response to determining that the third hierarchical classification determined by the cluster-based machine learning model and the fourth hierarchical classification determined by the classification-based machine learning model are different, associating one of the third hierarchical classification or the fourth hierarchical classification with a second category at the first level of the set of hierarchical categories. and not assigning it to the second target data item. The program of claim 1 , further comprising:

5. The operation comprises: at least, applying an additional trained cluster-based machine learning model to determine a first similarity value between a first level category and a second level category associated with the assigned first hierarchical classification or the assigned second hierarchical classification; applying a trained multi-class classification-based machine learning model to determine a second similarity value between the first level category and the second level category associated with the assigned first hierarchical classification or the assigned second hierarchical classification; and in response to determining that the first similarity value and the second similarity value both exceed a threshold, validating the first hierarchical classification or the second hierarchical classification assigned to the first target data item.

2. The program of claim 1, further comprising verifying the first hierarchical classification or the second hierarchical classification assigned to the first target data item by:

6. 2. The program of claim 1, wherein the first data item of interest includes one or more of a resume, a job description, or a job posting, and the first category includes skills of a job seeker.

7. The program of claim 1 , wherein the first hierarchical classification and the second hierarchical classification are generated independently.

8. A method comprising the acts of any one of claims 1 to 7.

9. At least one device including a hardware processor A system comprising:

10. A system comprising means for performing the operations of any of claims 1 to 7.

Images