System and method for generating training data

Abstract

Methods and systems for generating a retraining dataset based on unknown entities are disclosed herein. In one embodiment, a method includes logging a plurality of complete search queries; generating a discarding barrel; determining whether each of the plurality of complete search queries includes an unknown entity and / or a known entity having an unknown relationship; and populating the discard bucket with each complete search query of the plurality of complete search queries determined to include the unknown entity and / or the known entity having the unknown relationship. The method further includes, after a pre-selected time interval, transmitting the discard bucket to a computing device configured to generate an annotated discard bucket, and in response to receiving the annotated discard bucket, generating a formatted file readable by a machine learning training algorithm, and retraining the machine learning model based on the formatted file.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for generating training data for models, and in particular to systems and methods for generating training data for entity and relationship models based on unknown entities and / or known entities with unknown or missing relationships. Background Art

[0002] Training and validating models, such as entity and relationship machine learning models, typically utilizes manually annotated documents in a given domain. Annotation and / or reading / reviewing such documents is typically performed by subject matter experts and therefore consumes considerable time and resources. New models may require even longer training times because larger sets of annotated documents are used for such training. The generation of such annotated documents, especially in resource-constrained enterprises, may consume considerable time and resources. Additionally, when new documents are encountered outside of the domain, the new documents may be utilized to train the model. New documents outside of the domain used for training may reduce overall model accuracy and / or consistency. Similar to the other annotated documents mentioned above, the new documents are manually annotated and used to retrain or further train the model. Summary of the invention

[0003] In view of the foregoing, Applicants have recognized these and other problems in the art, and have recognized a need for enhanced systems and methods for generating training data for models, and in particular, a need for systems and methods for continuously, substantially continuously, and / or in real time generating training data for entity and relationship models based on unknown entities and / or known entities with unknown or missing relationships.

[0004] The present disclosure as a whole relates to a system for solving the above-mentioned related problems. Specifically, the system can realize the continuous, substantially continuous and / or real-time generation of training data and the retraining, further training and / or fine-tuning of models (e.g., entity and relationship models) under limited, minimal, substantially no or no user interaction. Such systems can be configured to receive search queries from one or more computing devices, for example, through a user interface. Search queries can be applied to or transferred to models (e.g., trained entity and relationship models or machine learning models) for analysis and / or processing. In one embodiment, based on search queries input at, for example, a user interface, the search query may include unknown entities, known entities with unknown relationships, and / or known entities with known relationships. The model can indicate whether such entities and / or relationships in the search query are known or unknown based on natural language processing (NLP) (e.g., the discovery or identification of noun chunks, such as nouns and words used to describe nouns). If the entity and / or relationship is determined to be unknown or missing, the system may transfer a portion of the search query (e.g., an unknown entity, a known entity with an unknown or missing relationship, and / or other portion of the search query) or the entire search query to a dropout bucket or file. The system may continue to perform such operations (e.g., adding the search query or portion of the search query to the dropout bucket or file) for a predefined or preselected time period or time interval (or based on another factor), thereby generating a dropout bucket or file with multiple entries. After the predefined or preselected time period or time interval has passed (or other factors are satisfied or passed), the dropout bucket or file may be transferred to one or more computing devices, which are configured to generate an annotated or marked version of the dropout bucket or file (e.g., a version of the dropout bucket or file that includes instructions about what the unknown entity and / or unknown or missing relationship is and / or what the unknown entity and / or unknown or missing relationship will be marked as). In another embodiment, the system may be configured to automatically mark or edit the dropout bucket or file after the predefined or preselected time period or time interval has passed. In yet another embodiment, a user via the system or via one or more configured computing devices may mark or edit a discard bucket or file. In such embodiments, the discard bucket or file may be sorted based on the frequency of inclusion of unknown entities within the discard bucket or file (e.g., using an entity search query that includes the most listed unidentified entities).

[0005] When receiving or generating a marked discard bucket or file, the system may automatically format or format the discard bucket or file. Annotation, automatic annotation or formatting may include formatting the discard bucket or file into a format readable by the training model or machine learning model. The formatted discard bucket or file can be used to further train, retrain or fine-tune the model or machine learning model. After training, retraining or fine-tuning, the model or machine learning model can be deployed for subsequent searches. The generation and filling of the discard bucket or file; the marking or editing of the discard bucket or file; the formatting of the marked or edited discard bucket or file; and the training, retraining or fine-tuning of the model or machine learning model can be an iterative and substantially continuous or continuous process. The amount or number of search queries used for retraining, fine-tuning, further training or for generating a new training set may include many or a large number of search queries. For unknown or missing relationships, in addition to labeling, determined or labeled relationships can also be used to define new ontologies and / or relationships between two or more known entities or unknown entities.

[0006] Therefore, embodiments of the present disclosure relate to a method for generating a training data set based on an unknown entity or one or more of known entities with unknown or missing relationships. The method may include logging multiple complete search queries. The method may include generating a discard bucket. The method may include determining whether each of the multiple complete search queries includes an unknown entity or one or more of known entities with unknown or missing relationships. The method may include filling a discard bucket with each of the multiple complete search queries that are determined to include an unknown entity and / or one or more of known entities with unknown or missing relationships. The method may include: after a pre-selected time interval or period, transmitting the discard bucket to a computing device configured to generate a labeled discard bucket. The method may include, in response to receiving a labeled discard bucket from a computing device, generating a formatted file readable by a machine learning training algorithm based on the labeled discard bucket. The method may include training, retraining, or fine-tuning a machine learning model based on the formatted file.

[0007] In one embodiment, the method may include, prior to transmitting the discard bucket to the computing device, determining the frequency of each unknown entity within the discard bucket and / or the frequency of each known entity with unknown or missing relationships within the discard bucket. The method may include sorting the discard bucket based on the frequency of each unknown entity and / or the frequency of each known entity with unknown or missing relationships. In another embodiment, the method may include, if one of the unknown entities remains unmarked for a preselected time interval, transmitting each unmarked unknown entity to the computing device along with a flag indicating the generation of a new ontology definition.

[0008] In another embodiment, the method may include transmitting each unlabeled known entity with unknown or missing relationship to a computing device along with a flag if one known entity for each known entity with unknown or missing relationship remains unlabeled for a preselected time interval. The flag may indicate that a new ontology definition is generated for the unknown or missing relationship. In another embodiment, the labeled discard bucket may be based on one or more of an internal ontology, an internal enterprise ontology, and / or an organizational ontology. In one embodiment, the labeled discard bucket may include one or more triples. The method may include inserting the triples into a knowledge graph.

[0009] In one embodiment, the method may include retraining, training and / or fine-tuning the query intent identification algorithm using or based on the formatted file. Retraining, training and / or fine-tuning the machine learning model may increase the F score of the machine learning model. The machine learning model may be an entity and relationship machine learning model.

[0010] Another embodiment of the present disclosure relates to a system for generating a training or retraining data set based on one or more of an unknown entity or a known entity with an unknown or missing relationship, which system may include a logging circuit. The logging circuit may be configured to log multiple complete search queries. The system may include a training circuit. The training circuit may be configured to generate a file. The training circuit may be configured to determine whether each of the multiple complete search queries includes an unknown entity or one or more of the known entities with an unknown or missing relationship. The training circuit may be configured to fill the file with each of the multiple complete search queries determined to include an unknown entity or one or more of the known entities with an unknown or missing relationship. The training circuit may be configured to transfer the file to a computing device configured to generate a labeled file after a predetermined or pre-selected time interval or time period. The training circuit may be configured to: automatically format the labeled file in response to receiving the labeled file, thereby generating a machine learning readable file; and retrain, train and / or fine-tune the entity and relationship machine learning model using the machine learning readable file.

[0011] The training circuitry may also be configured to determine a frequency of each instance of each of the one or more unknown entities and a frequency of each instance of each of the one or more known entities with unknown or missing relationships in the file. The file may be ranked based on a frequency of each instance of each of the one or more unknown entities (e.g., for annotation purposes) and / or a frequency of each instance of each of the one or more known entities with unknown or missing relationships.

[0012] In another embodiment, the training circuitry may be configured to determine a first time that the unknown entity remains unlabeled. The training circuitry may be configured to determine a second time that the known entity with the unknown or missing relationship remains unlabeled. The training circuitry may be configured to define a new ontology for the unknown entity that remains unlabeled in response to determining that the first time is greater than a preselected time. The new ontology may be defined based on input from a computing device. The training circuitry may be configured to define new relationships between known entities in response to determining that the second time is greater than a preselected time. The new relationships between known entities may be defined based on input from a computing device and / or input from a user.

[0013] Another embodiment of the present disclosure relates to a non-transitory machine-readable storage medium storing processor-executable instructions, which, when executed by at least one processor, cause the at least one processor to perform operations, processes, and / or steps. The execution of the instructions may cause the processor to log multiple complete search queries. The execution of the instructions may cause the processor to generate a discard bucket or file. The execution of the instructions may cause the processor to determine whether each of the multiple complete search queries includes one or more of an unknown entity or a known entity with an unknown or missing relationship. The execution of the instructions may cause the processor to fill a file with each of the multiple complete search queries that are determined to include an unknown entity or one or more of a known entity with an unknown or missing relationship. The execution of the instructions may cause the processor to transfer the file to a computing device configured to generate a labeled file after a predetermined time interval. The execution of the instructions may cause the processor to, in response to receiving a labeled file: automatically format the labeled file to generate a machine learning readable file; and retrain the entity and relationship machine learning model using the machine learning readable file.

[0014] Another embodiment of the present disclosure relates to a method for generating a training data set based on one or more unknown entities in an unknown entity. The method may include logging multiple complete search queries. The method may include generating a discard bucket. The method may include determining whether each of the multiple complete search queries includes one or more unknown entities in the unknown entity. The method may include filling the discard bucket with each of the multiple complete search queries with one or more unknown entities in the unknown entity. The method may include: after a preselected time interval, transmitting the discard bucket to a computing device configured to generate a labeled discard bucket. The method may include, in response to receiving a labeled discard bucket from a computing device, generating a formatted file readable by a machine learning training algorithm based on the labeled discard bucket. The method may include retraining the machine learning model based on the formatted file.

[0015] Based on the drawings, detailed descriptions and examples herein, additional and / or alternative purposes, features and advantages of the present disclosure will become apparent to those skilled in the art. However, the applicant notes that the drawings, detailed descriptions and examples, while indicating certain embodiments of the present disclosure, are provided for illustrative purposes only and are not intended to limit or imply specific limitations. In addition, based on this detailed description, certain changes and modifications within the spirit and scope of the disclosed technology will become apparent to those of ordinary skill in the relevant art. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The disclosed aspects, features and advantages of the present disclosure will be better understood with respect to the following description, examples, claims and drawings. Applicants note, however, that the drawings illustrate certain embodiments of the present disclosure and should not be considered as limiting the breadth and scope of the present disclosure:

[0017] Figure 1 is a schematic diagram of a system for generating a training or retraining dataset according to certain embodiments of the present disclosure;

[0018] Figure 2 is another schematic diagram of a system for generating a training or retraining dataset according to certain embodiments of the present disclosure;

[0019] Figure 3 is a flow chart for generating a training or retraining dataset according to certain embodiments of the present disclosure;

[0020] Figure 4 is another flow chart for training a model according to certain embodiments of the present disclosure;

[0021] Figure 5 is a user interface (UI) for marking or editing a discard bucket or file according to certain embodiments of the present disclosure;

[0022] Figure 6 is a flow chart for generating a training or retraining dataset and training, retraining, or fine-tuning a model according to certain embodiments of the present disclosure;

[0023] Fig. 7A and Figure 7B is a flowchart for generating a training or retraining dataset and training, retraining, or fine-tuning a model according to certain embodiments of the present disclosure; and

[0024] Fig. 8A and Figure 8B is a flowchart for generating a training or retraining dataset and training, retraining, or fine-tuning a model according to certain embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] Based on the drawings, detailed descriptions and examples herein, additional and / or alternative purposes, features and advantages of the present disclosure will become apparent to those skilled in the art. However, the applicant notes that the drawings, detailed descriptions and examples, while indicating certain embodiments of the present disclosure, are provided for illustrative purposes only and are not intended to limit or imply specific limitations. In addition, based on this detailed description, certain changes and modifications within the spirit and scope of the disclosed technology will become apparent to those of ordinary skill in the relevant art.

[0026] The following definitions are provided for the purpose of clarifying certain terms and phrases of the present disclosure and are in no way intended to unnecessarily or unduly limit any embodiments or aspects related thereto.

[0027] The use of the word "a" or "an" when used in conjunction with the terms "comprising," "including," "containing," or "having" in the claims or the specification may mean "one," but it is also consistent with the meaning of "one or more," "at least one," and "one or more than one."

[0028] The words “comprising” (and any forms of comprising, such as “comprise” and “comprises”), “having” (and any forms of having, such as “have” and “has”), “including” (and any forms of including, such as “includes” and “include”), or “containing” (and any forms of containing, such as “contains” and “contain”) are inclusive or open-ended, and do not exclude additional, unrecited elements or method steps.

[0029] In order that the features and advantages of the embodiments of the systems and methods disclosed herein, as well as other features and advantages that will become apparent, may be understood in greater detail, a more particular description of the embodiments of the systems and methods briefly summarized above may be obtained by reference to the following detailed description of the embodiments thereof, one or more of which are further illustrated in the accompanying drawings, which form a part of this specification. However, it should be noted that the drawings illustrate only various embodiments of the systems and methods disclosed herein, and therefore should not be considered to limit the scope of the systems and methods disclosed herein, as it may also include other effective embodiments.

[0030] Annotating or marking and / or reading / reviewing documents used to train, retrain and / or fine-tune models is typically performed by subject matter experts and consumes considerable time and resources. The generation of such documents, particularly in resource-constrained enterprises, may consume considerable time and resources. In addition, when new documents are encountered outside the domain of the model, the new documents may be used to train the model. New documents outside the domain used for training may reduce overall model accuracy and / or consistency. Similar to the other documents mentioned above, the new documents are manually annotated or marked and used to retrain or further train the model. Therefore, systems and methods for generating training data for models have been developed, and specifically relate to systems and methods for continuously, substantially continuously and / or in real time generating training data for entity and relationship models based on unknown entities and / or known entities with unknown or missing relationships.

[0031] Embodiments of such systems and methods related thereto are beneficially capable of improving the efficiency of training, retraining or fine-tuning models while reducing the time and resources used, as described herein. In addition, the systems and methods described herein can improve the overall accuracy and / or F score or F measure (e.g., a measure of model accuracy) of the model. In some embodiments, the system may be configured to receive multiple search queries (e.g., search queries entered into a user interface (UI) displayed, for example, via a web browser of a computing device) from one or more computing devices. The system may analyze the search query via a model to determine whether the search query includes one or more of unknown entities and / or known entities with unknown or missing relationships. Unknown entities and / or known entities with unknown or missing relationships may be added to a discard bucket or file, or in another example, added to a separate or one or more discard buckets or files. The discard bucket or file may be transmitted to a computing device configured to annotate, mark or edit the discard bucket or file, or the system may perform annotations, markings or edits to the discard bucket or file. The system can utilize annotated, labeled, or edited files to train, retrain, or fine-tune models (e.g., entity and relationship models and / or query intent recognition algorithms). The trained, retrained, or fine-tuned models can then be deployed and used for subsequent search queries.

[0032] Figure 11 is a schematic diagram of a system according to certain embodiments of the present disclosure. System 100 may include a training data system 102 (e.g., a system for generating a training data set). Training data system 102 may include one or more processors 104 and memory 106. Memory 106 may include and / or store instructions, models, and / or classifiers that may be executed by one or more processors 104. For example, memory 106 may include logging instructions 108. Logging instructions 108 may generate discard buckets, files, and / or text files when executed by one or more processors 104, collectively referred to as discard buckets hereinafter. Logging instructions 108 may generate discard buckets for search queries or parts of search queries with unknown entities and / or for known entities with unknown or missing relationships. In other words, logging instructions 108 may generate discard buckets for search queries or parts of search queries with unknown entities and / or generate discard buckets for search queries or parts of search queries with known entities with unknown or missing relationships. In another embodiment, logging instructions 108 may generate a discard bucket for both search queries or portions of search queries with unknown entities and search queries or portions of search queries with known entities with unknown or missing relationships.

[0033] The logging instructions 108, when executed by the one or more processors 104, may determine whether the search query includes one or more of unknown entities and / or known entities with unknown or missing relationships when the search query is entered into a search box (e.g., search) included in a user interface (UI) (e.g., UI 116A, UI 116B, through UI 116N). The search may be performed via an internal search tool (e.g., specific to an organization) or an external search tool (e.g., available to one or more users via the Internet). The internal or external search tool may be displayed to the user via the UI (e.g., UI 116A, 116B, 116N), such as via a graphical user interface (GUI) or a web-based user interface (WUI). When the user navigates to the internal or external search tool via a computing device (e.g., computing device 120A, computing device 120B, and / or through computing device 120N), the user may enter the search query and / or various terms. A search query and / or various terms may be entered to locate relevant documents (eg, documents related to the search query or various terms) or other information.

[0034] In one embodiment, in order to determine whether the search query includes one or more unknown entities and / or one or more known entities with unknown or missing relationships, the logging instructions 108 may transfer or apply the search query and / or various terms to the entity and relationship model 110 and / or other models or algorithms. The entity and relationship model 110 may be a trained model or classifier. The entity and relationship model 110 may be used to determine entities and / or map entities to various classifications (e.g., such as but not limited to organizations, business units, chemistry and / or testing, and other categories or classifications) and generate possible relationships between those entities. In one example, the entity and relationship model 110 may utilize natural language processing to identify or discover noun chunks to make such determinations. Using this determination and / or mapping, together with other models and tools (e.g., query intent recognition algorithms or models, and / or knowledge graphs, and other models or tools), the search tool may generate a list of the most relevant documents or other types of results for the user. As noted, some entities in the search query and / or the relationships between known entities may be unknown or missing. The model may indicate whether the relationship between an entity or a known entity is unknown. In such examples, logging instructions 108 may cause search queries, a portion of search queries, unknown entities, and / or known entities with unknown relationships to be stored in a discard bucket. Logging instructions 108 may continue to add or log such search queries or portions of search queries for a predetermined or pre-selected length of time or time period or time interval. For example, a new search query or portion of a search query may be logged for an hour, a day, a week, a month, or shorter or longer. In another embodiment, rather than logging search queries for a predetermined or pre-selected length of time or time period, search queries may be logged based on the amount of search queries received, for example, search queries may be logged until fifty search queries, one hundred search queries, one thousand search queries, or more are received.

[0035] Once a predetermined or pre-selected length of time or time period or other factors have passed or have been satisfied, the logging instructions 108 may transmit the discard bucket (or multiple discard buckets) or process the discard bucket (or multiple discard buckets) via the training data instructions 114. The training data instructions 114, when executed by one or more processors 104, may transmit the discard bucket to one or more computing devices (e.g., computing device 118A, computing device 1118B, and / or until computing device 118N). The one or more computing devices 118A, 118B, 118N may be configured to annotate, mark, and / or edit the discard bucket, or in another embodiment, generate a user interface configured to display the contents of the discard bucket in a readable format and allow annotation, marking, or editing of the contents of the discard bucket. As used herein, "annotated", "marked", or "edited" may refer to the process of marking unknown entities and / or marking unknown relationships based on one or more ontologies (e.g., internal ontologies, internal enterprise ontologies, organizational ontologies, and / or specified ontologies associated with a specified organization). The computing devices 118A, 118B, 118N may automatically annotate, mark, or edit the discard bucket (e.g., via an algorithm, script, or machine learning algorithm, instruction, or program), or may enable a user to annotate, mark, or edit the discard bucket or the contents of the discard bucket. In another embodiment, the training data system 102 may be configured to automatically annotate, mark, or edit the discard bucket or generate a user interface that allows a user to mark or edit the discard bucket, rather than sending the discard bucket to the computing devices 118A, 118B, 118N.

[0036] In another embodiment, each search query entered or submitted via UI 116A, 116B, 116N may be logged in a discard bucket. Each search query may be analyzed, and if one of the multiple search queries does not include an unknown entity and / or a known entity with an unknown or missing relationship, the search query may be removed or deleted from the discard bucket.

[0037] In another embodiment, known entity tags and / or known relationship tags may not match or be related to any tag of one or more ontologies utilized. In such embodiments, unknown entities and / or unknown relationships may remain unmarked or edited after a specified time period. In other words, when a discard bucket is marked or edited, such unknown entities and / or unknown or missing relationships may remain unmarked. Unmarked unknown entities and / or unknown or missing relationships may be added to subsequent discard buckets. In another embodiment, the training data system 102 may wait for a specified or pre-selected time, and if the unknown entity and / or unknown or missing relationship remains unmarked after a specified or pre-selected time, the training data system 102 may be configured to or may transmit the remaining unknown entities and / or unknown or missing relationships to a computing device, which is configured to generate new ontology definitions for unmarked unknown entities and / or unknown or missing relationships. If the training data system 102 transmits unmarked unknown entities and / or unknown or missing relationships to a computing device, the training data system 102 may mark or include a sign corresponding to the unmarked unknown entity and / or unknown relationship. The sign may indicate that a new ontology may be added or generated for an unmarked unknown entity and / or an unknown or missing relationship. In other words, the sign may indicate that an unmarked unknown entity and / or an unknown or missing relationship has remained unknown for such a long time, and there is no current ontology for the unknown entity and / or the unknown or missing relationship. In a further embodiment, the time length may include a week, a month or even longer. In yet another embodiment, the time length may be added to the discard bucket based on the search query, and in one example, the time length may be greater than or equal to the time when the search query is added to the discard bucket. In one embodiment, instead of the time length or in addition to the time length, other factors may be utilized to determine that there is no current ontology for an unknown entity and / or an unknown or missing relationship, such as but not limited to the number of times that the unmarked unknown entity and / or the unmarked unknown or missing relationship has been reviewed and / or instructions from a computing device and / or a user (e.g., a computing device marks an unmarked unknown entity and / or an unmarked unknown or missing relationship to indicate that there is no current ontology for an unmarked unknown entity and / or an unmarked unknown or missing relationship).

[0038] The discard bucket may be sorted prior to such transmission and / or annotation or marking. For example, the logging instructions 108 may determine the frequency of each instance of each of the one or more unknown entities and the frequency of each instance of each of the one or more known entities with unknown or missing relationships in the discard bucket when executed by the one or more processors 104. The logging instructions 108 may sort the discard bucket based on the frequency of each instance of each of the one or more unknown entities and / or the frequency of each instance of each of the one or more known entities with unknown or missing relationships when executed by the one or more processors 104.

[0039] Once the discard bucket has been annotated, marked, or edited, or once the discard bucket that has been annotated, marked, or edited has been received, the training data instructions 114 or the retraining instructions 112 may automatically format the discard bucket when executed by one or more processors 104, thereby defining a formatted discard bucket or file. The training data instructions 114 or the retraining instructions 112 may convert the discard bucket into a format that can be used or read by a machine learning model or classifier, such as the entity and relationship model 110 and / or the query intent recognition algorithm or model when executed by one or more processors 104. The retraining instructions 112 may then continue to train, retrain, and / or fine-tune a machine learning model or classifier, such as the entity and relationship model 110 and / or the query intent recognition algorithm or model, using the formatted discard bucket when executed by one or more processors 104. Once the machine learning model or classifier (e.g., the entity and relationship model 110 and / or the query intent recognition algorithm) is trained, retrained, and / or fine-tuned, the machine learning model or classifier (e.g., the entity and relationship model 110 and / or the query intent recognition algorithm or model) can be deployed for subsequent searches.

[0040] In some examples, the training data system 102 may be a computing device. The term "computing device" is used herein to refer to any or all of a programmable logic controller (PLC), a programmable automation controller (PAC), an industrial computer, a server, a virtual computing device or environment, a desktop computer, a personal data assistant (PDA), a laptop computer, a tablet computer, a smartbook, a palmtop computer, a personal computer, a smart phone, a virtual computing device, a cloud-based computing device, and similar electronic devices equipped with at least one processor and any other physical components necessary to perform the various operations described herein. Devices such as smart phones, laptop computers, and tablet computers are generally collectively referred to as mobile devices.

[0041] The term "server" or "server device" is used to refer to any computing device that can be used as a server, such as a primary exchange server, a web server, a mail server, a document server, or any other type of server. A server can be a dedicated computing device or a server module (e.g., an application) hosted by a computing device that enables the computing device to operate as a server. A server module (e.g., a server application) can be a full-featured server module, or a lightweight or auxiliary server module (e.g., a lightweight or auxiliary server application) configured to provide synchronization services between dynamic databases on a computing device. A lightweight server or auxiliary server can be a streamlined version of server-type functionality that can be implemented on a computing device such as a smart phone, enabling it to be used as an Internet server (e.g., a corporate email server) only to the extent necessary to provide the functionality described herein.

[0042] As used herein, a "non-transitory machine-readable storage medium" or "memory" may be any electronic, magnetic, optical, or other physical storage device that contains or stores information such as executable instructions, data, etc. For example, any machine-readable storage medium described herein may be any one or a combination of random access memory (RAM), volatile memory, non-volatile memory, flash memory, storage drive (e.g., hard drive), solid-state drive, any type of storage disk, etc. The memory may store or include instructions that may be executed by a processor.

[0043] As used herein, a "processor" or "processing circuitry" may include, for example, one processor or multiple processors included in a single device or distributed across multiple computing devices. A processor (e.g., Figure 1 The processor 104 shown may be at least one of a central processing unit (CPU), a semiconductor-based microprocessor, a graphics processing unit (GPU), a field programmable gate array (FPGA) for retrieving and executing instructions, a real-time processor (RTP), other electronic circuitry suitable for retrieving and executing instructions stored on a machine-readable storage medium, or a combination thereof.

[0044] In one embodiment, the machine learning model or classifier (e.g., entity and relationship model 110, query intent recognition algorithm or model, and / or other model or classifier) ​​can be a supervised or unsupervised learning model. In one embodiment, the machine learning model or classifier can be based on one or more of a decision tree, a random forest model, a random forest using bagging or boosting (such as gradient boosting), a neural network method, a support vector machine (SVM), other supervised learning models, other semi-supervised learning models, other unsupervised learning models, or some combination thereof, as will be readily understood by those of ordinary skill in the art. In one embodiment, the entity and relationship model 110 can determine entities and / or relationships between entities via natural language processing (NLP) instructions, algorithms, or models. For example, the entity and relationship model 110 can find or identify noun chunks, such as, for example, nouns and words used to describe nouns. The entity and relationship model 110 can then use at least the noun chunks to identify different entities and / or relationships between those entities.

[0045] Figure 2 2 is another schematic diagram of a system or apparatus for generating a training or retraining data set according to certain embodiments of the present disclosure. Apparatus 200 may include processing circuitry 202, memory 204, communication circuitry 206, logging circuitry 208, and training circuitry 210, each of which will be described in more detail below. Although the various components are only described in detail below, Figure 2 200 is shown as being connected to the processing circuit 202, but it should be understood that the device 200 may also include a bus ( Figure 2 The apparatus 200 may be configured to perform various operations described herein, such as the above-mentioned Figure 1 And the following combination Figure 3 to Figure 4 and Figures 6 to 8B Describe the operations.

[0046] The processing circuit 202 (and / or a coprocessor or auxiliary processor or any other processor otherwise associated with the processor) may communicate with the memory 204 via a bus for passing information between components of the device. The processing circuit 202 may be embodied in a variety of different ways and may, for example, include one or more processing devices configured to execute independently. In addition, the processor may include one or more processors configured in series via a bus to enable independent execution of software instructions, pipelining and / or multithreading.

[0047] Processing circuit 202 may be configured to execute software instructions stored in memory 204 or otherwise accessible to processing circuit 202 (e.g., software instructions stored on a separate storage device). In some cases, processing circuit 202 may be configured to perform hard-coded functionality. Thus, whether configured by hardware or software methods, or by a combination of hardware and software, processing circuit 202 represents an entity (e.g., physically embodied in a circuit) capable of performing operations according to various embodiments of the present disclosure when configured accordingly. Alternatively, as another example, when processing circuit 202 is embodied as an executor of software instructions, the software instructions may specifically configure processing circuit 202 to perform the algorithms and / or operations described herein when executing the software instructions.

[0048] The memory 204 is non-transitory and may include, for example, one or more volatile and / or non-volatile memories. In other words, for example, the memory 204 may be an electronic storage device (e.g., a computer-readable storage medium). The memory 204 may be configured to store information, data, content, applications, software instructions, etc. for enabling the apparatus 200 to perform various functions according to the example embodiments contemplated herein.

[0049] The communication circuitry 206 may be any device configured to receive and / or transmit data from / to a network and / or any other device, circuit, or module in communication with the apparatus 200, such as a device or circuit embodied in hardware or a combination of hardware and software. In this regard, the communication circuitry 206 may include, for example, a network interface for enabling communication with a wired or wireless communication network. For example, the communication circuitry 206 may include one or more network interface cards, antennas, buses, switches, routers, modems, and supporting hardware and / or software, or any other device suitable for enabling communication via a network. In addition, the communication circuitry 206 may include processing circuitry for enabling the transmission of such signals to the network or for processing the reception of signals received from the network.

[0050] The apparatus 200 may include logging circuitry 208 configured to generate a discard bucket, determine whether a search query includes unknown entities and / or known entities with unknown or missing relationships, fill a discard bucket with all or part of a search query with unknown entities and / or known entities with unknown or missing relationships, and / or format a marked or edited discard bucket. The logging circuitry 208 may be configured to mark or edit a filled discard bucket. In another embodiment, the logging circuitry 208 may be configured to transmit the filled discard bucket to one or more computing devices configured to mark or edit the filled discard bucket. The logging circuitry 208 may perform these operations using the processing circuitry 202, the memory 204, or any other hardware component included in the apparatus 200, as described below in conjunction with Figures 3 to 7B The logging circuit 208 may also utilize the communication circuit 206 to collect data (e.g., all or part of the search query) from various sources (e.g., search queries input into the UI 116A, 116B, 116N via the computing device 120A, 120B, 120N, annotated or marked discard buckets from the computing device 118A, 118B, 118N). The output of the logging circuit 208 may be transmitted to other circuits of the device 200 (e.g., the training circuit 210).

[0051] In addition, the device 200 also includes a training circuit 210 that can format the marked or edited discard buckets; train, retrain and / or fine-tune the machine learning model (e.g., the entity and relationship model 110 and / or the query intent recognition algorithm or model); and / or deploy an updated machine learning model. The training circuit 210 can perform these operations using the processing circuit 202, the memory 204, or any other hardware components included in the device 200, as described below in conjunction with Figures 3 to 7B The training circuit 210 may also utilize the communication circuit 206 to collect data (e.g., annotated or labeled discard buckets or formatted and annotated or labeled discard buckets) from various sources (e.g., logging circuit 208 or computing devices 118A, 118B, 118N), and in some embodiments, the processing circuit 202 and / or memory 204 may be utilized to format the annotated or labeled discard buckets and / or train, retrain, or fine-tune the machine learning model. The output of the training circuit 210 may be transmitted to other circuits of the device 200.

[0052] Although the components 202-210 are described in part using functional language, it should be understood that a specific implementation necessarily includes the use of specific hardware. It should also be understood that some of these components 202-210 may include similar or common hardware. For example, the logging circuit 208 and the training circuit 210 may each utilize the use of the processing circuit 202, the memory 204, or the communication circuit 206 from time to time, so that no repeated hardware is required to facilitate the operation of these physical elements of the device 200 (although in some embodiments, dedicated hardware elements may be used for any of these components, such as those where enhanced parallelism may be desired). Therefore, the use of the terms "circuit" and "engine" with respect to the elements of the device should be interpreted as necessarily including specific hardware configured to perform functions associated with the specific elements described. Of course, although the terms "circuit" and "engine" should be broadly understood to include hardware, in some embodiments, the terms "circuit" and "engine" may additionally refer to the hardware components of the configuration device 200 to perform software instructions for various functions described herein.

[0053] Although the logging circuit 208 and the training circuit 210 may utilize the processing circuit 202, the memory 204, or the communication circuit 206 as described above, it should be understood that any of these elements of the device 200 may include one or more dedicated processors, specially configured field programmable gate arrays (FPGAs), or dedicated interface circuits (ASICs) to perform their corresponding functions, and may accordingly utilize the processing circuit 202 that executes software stored in the memory or storage 204, and the communication circuit 206 for enabling any functions not performed by dedicated hardware elements. However, in all embodiments, it should be understood that the logging circuit 208 and the training circuit 210 are implemented via specific machines designed to perform the functions described herein in conjunction with such elements of the device 200.

[0054] In some embodiments, various components of the device 200 may be remotely hosted (e.g., hosted by one or more cloud servers), and thus need not physically reside on the corresponding device 200. Thus, some or all of the functionality described herein may be provided by third-party circuitry. For example, a given device 200 may access the third-party circuitry via any type of network connection that facilitates the transfer of data and electronic information between the device 200 and one or more third-party circuitry. In turn, the device 200 may communicate remotely with one or more of the other components described above as comprising the device 200.

[0055] As will be appreciated based on this disclosure, the example embodiments contemplated herein may be implemented by the apparatus 200. In addition, some example embodiments may take the form of a computer program product comprising software instructions stored on at least one non-transitory computer-readable storage medium (e.g., memory 204). Any suitable non-transitory computer-readable storage medium may be utilized in such embodiments, some examples of which are non-transitory hard disks, CD-ROMs, flash memories, optical storage devices, and magnetic storage devices. It should be understood that for example, the computer program product may be implemented by the apparatus 200. Figure 2 Certain devices embodied in the apparatus 200 described in the drawings, loading the software instructions onto a computing device or apparatus creates a special purpose machine including means for implementing the various functions described herein.

[0056] Figure 3 300 is a flowchart for generating a training or retraining data set according to certain embodiments of the present disclosure. Unless otherwise specified, the actions of method 300 may be performed within system 100 and / or device 200. Specifically, method 300 may be included in one or more programs, protocols, or instructions loaded into memory 106 of training data system 102 and executed on processor or one or more processors 104. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks may be combined in any order and / or in parallel to implement these methods.

[0057] At block 302, a question, phrase, or various word sequences may be entered into a search bar in a portal, such as a WUI displayed on a computing device. In one embodiment, a user may enter a question, phrase, or various word sequences.

[0058] At block 304, a logging utility may be executed. In one embodiment, execution of the logging utility may include applying questions, phrases, or various word sequences to a model (e.g., an entity and relationship model and / or a query intent identification algorithm or model). The output of the model may be added to a text file or a log file.

[0059] At block 306 , a script may be executed to generate a discard bucket that includes questions, phrases, or various word sequences with unknown entities and / or known entities with unknown or missing relationships.

[0060] At block 308, the discard buckets may be analyzed, for example, via the training data system 102 or the device 200. Such analysis may produce annotated or labeled discard buckets that identify unknown entities and / or known entities with unknown or missing relationships. At block 310, a formatting utility may be utilized to automatically format the labeled discard buckets. At block 312, a model (e.g., an entity and relationship model and / or a query intent recognition algorithm or model) may be retrained or fine-tuned. At block 314, the retrained or fine-tuned model may be deployed or utilized in subsequent searches.

[0061] Figure 4 4 is a flowchart for training a model according to certain embodiments of the present disclosure. Unless otherwise specified, the actions of method 400 may be performed within system 100 and / or device 200. Specifically, method 400 may be included in one or more programs, protocols, or instructions loaded into memory 106 of training data system 102 and executed on processor or one or more processors 104. The order in which the operations are described is not intended to be interpreted as a limitation, and any number of the described blocks may be combined in any order and / or in parallel to implement these methods.

[0062] Application program interface (API) 402 may obtain labeled logs 404 (e.g., from or via labeled discard buckets of training data system 102 or device 200). Labeled logs 404 may be transmitted to formatting UI 406 (e.g., of training data system 102 or device 200). Formatting UI 406 may automatically format labeled logs 404, thereby generating formatted files 408 (e.g., formatted and annotated or labeled logs). Formatted files 408 and any other formatted files generated and / or available at the time may form training set 410. The latest version of existing models 412 (e.g., entity and relationship models and / or query intent recognition algorithms or models) may be obtained via a practical model script for training 414. The training set 410 may then be used to train, retrain, or fine-tune the existing models 412, thereby generating updated models 418.

[0063] The updated model 418 can be used in an inference job or script 422 to obtain or generate a metric 424 based on a test set. In other words, the updated model 418 can be tested using the training set 410. At this point, at 426, any new triples (e.g., two entities and a connecting relationship) can be inserted into a knowledge graph corresponding to, for example, a search tool. Finally, at block 428, the updated first model can be deployed for subsequent searches.

[0064] Figure 5 It is a user interface (UI) for marking or editing a discard bucket or file according to certain embodiments of the present disclosure. A graphical user interface (GUI) 500 is provided, which shows an example of a partially annotated or marked discard bucket. In one example, the contents of the discard bucket can be displayed to a user via GUI 500 on one or more computing devices or, for example, a training data system. Although a portion of a discard bucket is shown, it should be understood that the discard bucket may include multiple search queries, for example, one hundred, one thousand or even more search queries. Each search query may be numbered (e.g., search query one 502, search query two 504, search query three 506, and search query four 508) and sorted based on various factors (e.g., the frequency of terms, such as but not limited to entities, unknown entities, and / or unknown relationships). As shown, some parts of the search query may be identified, for example, for search query one 502, at 510, "analysis" may be identified as a test. In addition, at 512, "polyamide" may be identified as "organic polymer". At 514, the number "622" may be an unassigned entity or an unknown entity. At 516 , “mw” may be identified as the chemical property, and at 518 , “GPC” may be identified as the test method.

[0065] In such embodiments, the user (or, in other examples, the training data system 102 or the device 200) can update, annotate and / or mark any label that is marked as an unassigned entity and / or an unassigned relationship. In another embodiment, the user can update or change any of the labels included in the GUI 500. In yet another example, the user can add a label to any unlabeled term, or can add a label or identify an unknown or missing relationship. For example, the user (or the training data system 102 or the device 200) can update, annotate and / or mark the unassigned entity at 514 (e.g., 622). In another example, the user (or the training data system 102 or the device 200) can identify that the unassigned entity at 514 is polyamide, or in other examples, it is a different defined entity. In addition, the user (or the training data system 102 or the device 200) can update the unassigned entity at 514 to "polyamide", thereby generating a new entity (e.g., based on the label "polyamide" and the word marked as an unknown entity, "622"). In other examples, multiple words may be used to update, annotate, and / or tag one or more unknown entities, and a combination of these words and one or more words tagged as one or more unknown entities may be used to generate new entities.

[0066] Figure 6 600 is a flowchart for generating a training or retraining data set and training, retraining or fine-tuning a model according to certain embodiments of the present disclosure. The method 600 is described in detail with reference to the system 100. Unless otherwise specified, the actions of the method 600 may be performed within the system 100 or the training data system 102. Specifically, the method 600 may be included in one or more programs, protocols or instructions loaded into the memory 106 of the training data system 102 and executed on the processor or one or more processors 104. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks may be combined in any order and / or in parallel to implement these methods.

[0067] At block 602, a system (e.g., training data system 102) may determine whether a search query has been received. The search query may be entered via a UI displayed on a computing device in communication with the system. The search query may be utilized by various parts of the system to generate a list of relevant results. In one embodiment, many search queries may be received at many different times. Each search query may be processed in substantially parallel time frames or timelines or as the search query is received.

[0068] At block 604, the system may generate a discard bucket. The discard bucket may be generated once for each analysis of multiple search queries, or once in total. For example, a discard bucket may be generated, filled, analyzed, and then deleted, with the new discard bucket replacing the previous one. In another embodiment, data (e.g., a marked search query) may be deleted or removed from the discard bucket after analysis. In another embodiment, after analysis, portions of the discard bucket that have been marked may be removed to make room for additional search queries.

[0069] At block 606, the system may determine whether the search query includes one or more unknown entities. The system may utilize models (e.g., entity and relationship models, as well as other models and / or classifiers and / or knowledge graphs) to determine whether any terms in the search query are unknown. An indicator or label may be included in such a determination to indicate that the search query includes unknown entities. At block 608, the system may determine whether the search query includes unknown entities based on the indicator or label corresponding to the search query.

[0070] If the system determines that there is an unknown entity, the system can fill the discard bucket with the complete search query at block 610. In another embodiment, a portion of the search query can be used to fill the discard bucket. In yet another embodiment, the discard bucket can be filled with unknown entities.

[0071] At box 612, if the search query does not include an unknown entity, the system may determine whether the discard bucket includes at least one search query. If there is no search query or part of a search query, the system may wait for submission of additional search queries at box 602. If at least one search query is included in the discard bucket, at box 614, the system may determine whether a predefined or preselected time period or interval has passed. If the predefined or preselected time period or interval has passed, at box 616, the system may transfer the discard bucket to one or more computing devices configured to mark or edit the discard bucket. In another embodiment, the system may be configured to display the discard bucket to one or more users for annotation, marking or editing. In yet another embodiment, the system may automatically annotate, mark or update the discard bucket.

[0072] At box 618, the system may check or determine whether an annotated or marked discard bucket has been received. In another embodiment, the system may check whether the discard bucket has been updated, annotated, marked or edited, rather than being received. At box 620, the system may generate a formatted discard bucket or file readable by a machine learning algorithm or model. In one embodiment, the system may format or automatically format the discard bucket. In one embodiment, annotated or marked search queries may be formatted or edited. The remaining unannotated or unmarked search queries may be retained in the discard bucket, stored in a separate discard bucket, or deleted or removed from the discard bucket. In other words, unannotated or marked search queries may not be formatted and sent for retraining. At box 622, the system may retrain the model with a formatted file. The model may then be deployed for further use, such as in subsequent searches.

[0073] Fig. 7A and Figure 7B 700 is a flowchart for generating a training or retraining data set and training, retraining or fine-tuning a model according to certain embodiments of the present disclosure. Method 700 is described in detail with reference to system 100. Unless otherwise specified, the actions of method 700 may be performed within system 100 or training data system 102. Specifically, method 700 may be included in one or more programs, protocols, or instructions loaded into memory 106 of training data system 102 and executed on processor or one or more processors 104. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks may be combined in any order and / or in parallel to implement these methods.

[0074] At block 702, a system (e.g., training data system 102) may determine whether a search query has been received. The search query may be entered via a UI displayed on a computing device in communication with the system. The search query may be utilized by various parts of the system to generate a list of relevant results. In one embodiment, many search queries may be received at many different times. Each search query may be processed in substantially parallel time frames or timelines or as the search query is received.

[0075] At block 704, the system may generate a discard bucket. The discard bucket may be generated once for each analysis of multiple search queries, or once in total. For example, a discard bucket may be generated, filled, analyzed, and then deleted, with the new discard bucket replacing the previous one. In another embodiment, data (e.g., annotated or tagged search queries) may be deleted or removed from the discard bucket after analysis. In another embodiment, portions of the discard bucket that have been annotated or tagged may be removed after analysis to make room for additional search queries.

[0076] At block 706, the system may determine whether the search query includes one or more unknown entities. The system may utilize models (e.g., entity and relationship models, as well as other models and / or classifiers and / or knowledge graphs) to determine whether any terms in the search query are unknown. An indicator may be included in such a determination to indicate that the search query includes unknown entities. At block 708, the system may determine whether the search query includes unknown entities based on the indicator corresponding to the search query.

[0077] If the system determines that there is an unknown entity, the system can fill the discard bucket with the complete search query at block 710. In another embodiment, a portion of the search query can be used to fill the discard bucket. In yet another embodiment, the discard bucket can be filled with unknown entities.

[0078] At block 712, if the search query does not include unknown entities, the system may determine whether known entities within the search query include unknown relationships. An indicator may be included in such search queries to indicate that the search query includes unknown relationships. In another embodiment, if the search query does not include one or more unknown entities, the system may add the search query to a discard bucket if the relationships are missing, likely missing, or unknown. In other words, in such embodiments, search queries that have no unknown entities and are missing relationships or include unknown relationships may be automatically added to a discard bucket.

[0079] At block 714, the system can determine whether the search query includes unknown or missing relations based on the included indicator. If the search query includes unknown or missing relations, at block 710, the system can populate a discard bucket with the search query, a portion of the search query, or a known entity with an unknown or missing relation.

[0080] At box 716, the system may determine whether the discard bucket includes at least one search query. If not, the system may wait for submission of additional search queries at box 702. If at least one search query is included in the discard bucket, at box 718, the system may determine whether a predefined or preselected time period or interval has passed. If a predefined or preselected time period or interval has passed, at box 720, the system may transmit the discard bucket to one or more computing devices configured to mark or edit the discard bucket. In another embodiment, the system may be configured to display the discard bucket to one or more users for marking or editing. In yet another embodiment, the system may automatically mark or update the discard bucket.

[0081] At box 722, the system may check or determine whether an annotated or marked discard bucket has been received. In another embodiment, the system may check whether the discard bucket has been updated, annotated, marked or edited, rather than being received. At box 724, the system may generate a formatted discard bucket or file readable by a machine learning algorithm or model. In one embodiment, the system may format or automatically format the discard bucket. In one embodiment, annotated or marked search queries may be formatted or edited. The remaining unannotated or unmarked search queries may be retained in the discard bucket, stored in a separate discard bucket, or deleted or removed from the discard bucket. In other words, unannotated or marked search queries may not be formatted and sent for retraining. At box 726, the system may retrain the model with a formatted file. The model may then be deployed for further use, such as in subsequent searches.

[0082] Fig. 8A and Figure 8B 800 is a flowchart for generating a training or retraining data set and training, retraining or fine-tuning a model according to certain embodiments of the present disclosure. Method 800 is described in detail with reference to system 100. Unless otherwise specified, the actions of method 800 may be performed within system 100 or training data system 102. Specifically, method 800 may be included in one or more programs, protocols, or instructions loaded into memory 106 of training data system 102 and executed on processor or one or more processors 104. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks may be combined in any order and / or in parallel to implement these methods.

[0083] At block 802, a system (e.g., training data system 102) may determine whether a search query has been received. The search query may be entered via a UI displayed on a computing device in communication with the system. The search query may be utilized by various parts of the system to generate a list of relevant results. In one embodiment, many search queries may be received at many different times. Each search query may be processed in substantially parallel time frames or timelines or as the search query is received.

[0084] At block 804, the system may generate a discard bucket for unknown entities. The unknown entity discard bucket may be generated once for each analysis of multiple search queries, or once in total. For example, an unknown entity discard bucket may be generated, populated, analyzed, and then deleted, with the new unknown entity discard bucket replacing the previous unknown entity discard bucket. In another embodiment, data (e.g., annotated or tagged search queries) may be emptied from the unknown entity discard bucket after analysis. In another embodiment, after analysis, portions of the unknown entity discard bucket that have been annotated or tagged may be removed to make room or storage for additional search queries.

[0085] At box 806, the system may generate a discard bucket for known entities with unknown or missing relationships. The unknown or missing relationship discard bucket may be generated once for each analysis of multiple search queries, or once in total. For example, an unknown or missing relationship discard bucket may be generated, filled, analyzed, and then deleted, with the new unknown or missing relationship discard bucket replacing the previous unknown or missing relationship discard bucket. In another embodiment, data (e.g., annotated or tagged search queries) may be deleted or removed from the unknown or missing relationship discard bucket after analysis. In another embodiment, after analysis, portions of the unknown or missing relationship discard bucket may be removed to make room for additional search queries.

[0086] At box 808, the system may determine whether the search query includes one or more unknown entities. The system may use models (e.g., entity and relationship models, as well as other models and / or classifiers and / or knowledge graphs) to determine whether any item in the search query is unknown. An indicator may be included in such a determination to indicate that the search query includes unknown entities. At box 810, the system may determine whether the search query includes unknown entities based on the indicator corresponding to the search query. If the system determines that there are unknown entities, the system may fill the unknown entity discard bucket with the complete search query at box 812. In another embodiment, a portion of the search query may be used to fill the unknown entity discard bucket. In yet another embodiment, the unknown entity discard bucket may be filled with unknown entities.

[0087] At box 814, the system may determine whether the known entities and / or unknown entities within the search query include unknown relationships or missing relationships. An indicator may be included in such a search query to indicate that the search query includes an unknown relationship or a missing relationship (e.g., a relationship is not defined for two entities). At box 816, the system may determine whether the search query includes an unknown relationship or a missing relationship based on the included indicator. If an unknown relationship is included in the search query or if a relationship is missing from the search query, at box 818, the system may populate an unknown or missing relationship discard bucket with the search query, a portion of the search query, a known entity with an unknown or missing relationship, or an unknown entity with an unknown or missing relationship.

[0088] At box 820, the system may determine whether the unknown entity discard bucket includes at least one search query. If not, the system may wait for submission of additional search queries at box 802. If at least one search query is included in the unknown entity discard bucket, then at box 822, the system may determine whether a predefined or preselected time period or interval has passed. If the predefined or preselected time period or interval has passed, then at box 824, the system may transmit the unknown entity discard bucket to one or more computing devices configured to mark or edit the unknown entity discard bucket. In another embodiment, the system may be configured to display the discard bucket to one or more users for marking or editing. In yet another embodiment, the system may automatically mark or update the unknown entity discard bucket.

[0089] At box 828, the system may determine whether the unknown or missing relationship discard bucket includes at least one search query. If not, the system may wait for submission of additional search queries at box 802. If at least one search query is included in the unknown or missing relationship discard bucket, then at box 830, the system may determine whether a predefined or preselected time period or interval has passed. If the predefined or preselected time period or interval has passed, then at box 832, the system may transmit the unknown or missing relationship discard bucket to one or more computing devices configured to mark or edit the unknown or missing relationship discard bucket. In another embodiment, the system may be configured to display the unknown or missing relationship discard bucket to one or more users for marking or editing. In yet another embodiment, the system may automatically mark or update the unknown or missing relationship discard bucket.

[0090] At box 826, the system may check or determine whether an annotated or marked unknown entity discard bucket has been received. At box 834, the system may check or determine whether an annotated or marked unknown or missing relationship discard bucket has been received. In another embodiment, the system may check whether the unknown entity discard bucket and / or the unknown or missing relationship discard bucket has been updated, annotated, marked or edited, rather than being received. At box 836, the system may generate a formatted file readable by a machine learning algorithm or model based on one or more of the unknown entity discard bucket or the unknown or missing relationship discard bucket. In one embodiment, the system may format or automatically format one or more of the unknown entity discard bucket or the unknown or missing relationship discard bucket. In one embodiment, annotated or marked search queries may be included in the formatted file. The remaining unannotated or unmarked search queries may be retained in the corresponding discard bucket, stored in a separate discard bucket, or deleted or removed from the discard bucket. In other words, unannotated or marked search queries may not be formatted and sent for retraining. At box 838, the system may retrain the model with a formatted file. The model can then be deployed for further use, such as in subsequent searches.

[0091] Although specific terms and concepts are incorporated in the present disclosure, applicants note that the disclosed terms and concepts are used specifically in a descriptive capacity and therefore should not be understood or interpreted as limiting in any way. Certain embodiments and aspects of the disclosed systems, processes, and methods have been described in detail with particular reference to the illustrated embodiments. However, it is apparent that many and various modifications and changes may be made within the spirit and scope of the embodiments of the systems, processes, and methods described herein, and such modifications and changes are considered equivalent and within the breadth and scope of the present disclosure.

Claims

1. A method for generating a training data set based on one or more of an unknown entity or a known entity with an unknown relationship, the method comprising: Log multiple complete search queries; Generate a discard bucket; determining whether each of the plurality of complete search queries includes one or more of an unknown entity or a known entity with an unknown relationship; populating the discard bucket with each of the plurality of complete search queries having one or more of the unknown entity or the known entities having the unknown relationship; after a predetermined time interval, transmitting the discard bucket to a computing device configured to generate an annotated discard bucket; In response to receiving the annotated discard bucket from the computing device, generating a formatted file readable by a machine learning training algorithm based on the annotated discard bucket; as well as Retrain the machine learning model based on the formatted file.

2. The method according to claim 1, comprising: Prior to transmitting the discard bucket to the computing device: determining a frequency of each unknown entity within the discard bucket; determining a frequency of each known entity having an unknown relationship within the discard bucket; sorting the discard buckets based on the frequency of each unknown entity and the frequency of each known entity having an unknown relationship; In response to one of the unknown entities remaining unannotated for a preselected time interval, transmitting each unannotated unknown entity to the computing device along with a flag indicating generation of a new ontology; as well as In response to one of each of the known entities having the unknown relationship remaining unannotated for a preselected time interval, each unannotated known entity having the unknown relationship is transmitted to the computing device along with a second flag indicating generation of a new relationship definition. 3 . The method of claim 1 , wherein the annotated discard bucket is based on one or more of an internal ontology, an internal enterprise ontology, or an organizational ontology.

4. The method of claim 1, wherein the annotated drop bucket comprises one or more triples, and wherein the method comprises inserting the triples into a knowledge graph.

5. The method according to claim 1, comprising: The query intent recognition algorithm is retrained using the formatted file.

6. The method of claim 1, wherein retraining the machine learning model increases the F-score of the machine learning model, and wherein the machine learning model is an entity and relationship machine learning model.

7. A system for generating a retraining data set based on one or more of an unknown entity or a known entity with an unknown relationship, the system comprising: logging circuitry configured to log a plurality of complete search queries; as well as A training circuit, wherein the training circuit is configured to: Generate files, determining whether each of the plurality of complete search queries includes one or more of an unknown entity or a known entity with an unknown relationship, populating the file with each complete search query, wherein each complete search query in the plurality of complete search queries is determined to include one or more of the unknown entity or the known entity having the unknown relationship, After a predetermined time interval, transmitting the file to a computing device configured to generate a marked file, and In response to receiving the marked file: automatically formatting the tagged file to generate a machine learning readable file, and Retrain entity and relationship machine learning models using the machine learning readable files.

8. The system of claim 7, wherein the training circuit is configured to: The frequency of each instance of each of the unknown entities and the frequency of each instance of each of the known entities having unknown relationships in the document are determined.

9. The system of claim 8, wherein the files are ranked based on the frequency of each instance of each of the unknown entities and the frequency of each instance of each of the known entities having unknown relationships.

10. The system of claim 7, wherein the training circuit is further configured to: Determining the first time an unknown entity remains unlabeled; and It is determined that known entities with unknown relations remain unlabeled a second time.

11. The system of claim 10, wherein the training circuit is configured to: In response to determining that the first time is greater than a preselected time, a new ontology is defined for the unknown entity that remains unlabeled.

12. The system of claim 11, wherein the new ontology is defined based on input from the computing device.

13. The system of claim 10, wherein the training circuit is configured to: In response to determining that the second time is greater than a preselected time, a new relationship between the known entities is defined.

14. The system of claim 13, wherein the new relationship between known entities is defined based on input from the computing device.

15. A method for generating a training data set based on one or more unknown entities in unknown entities, the method comprising: Log multiple complete search queries; Generate a discard bucket; determining whether each of the plurality of complete search queries includes one or more of the unknown entities; populating the discard bucket with each of the plurality of complete search queries having the one or more of the unknown entities; after a preselected time interval, transmitting the discard bucket to a computing device configured to generate a labeled discard bucket; In response to receiving the labeled discard bucket from the computing device, generating a formatted file readable by a machine learning training algorithm based on the labeled discard bucket; as well as Retrain the machine learning model based on the formatted file.

16. The method according to claim 15, comprising: Prior to transmitting the discard bucket to the computing device: determining a frequency of each unknown entity within the discard bucket; as well as The discard buckets are sorted based on the frequency of each unknown entity.

17. The method according to claim 15, comprising: In response to one of the unknown entities remaining unlabeled for a preselected time interval, each unlabeled unknown entity is transmitted to the computing device along with a flag indicating generation of a new ontology.

18. The method of claim 15, wherein the labeled discard bucket is based on one or more of an internal ontology, an internal enterprise ontology, or an organizational ontology.

19. The method of claim 15, wherein the labeled discard bucket comprises one or more triples, and wherein the method further comprises inserting the triples into a knowledge graph.

20. The method according to claim 15, further comprising: The query intent recognition algorithm is retrained using the formatted file.

Images