A system and method for accelerating natural language processing by integrating case-specific knowledge and general knowledge.

Abstract

A system and method for building a predictive machine learning model. The method comprises parsing the text of a document review protocol to extract at least one description of at least one concept to be tagged, and tagging at least a portion of the multiple documents to create multiple tagged documents by applying a language model to the multiple documents, wherein tagging at least a portion of the multiple documents further comprises querying the language model with at least one query generated based on the extracted at least one description, building at least one classifier machine learning model based on the extracted at least one description, and training at least one classifier machine learning model using a training set, wherein the training set comprises multiple tagged documents.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 503341, filed on May 19, 2023, the content of which is incorporated herein by reference.

[0002] This disclosure relates to predictive machine learning, and more specifically, to extracting knowledge to build predictive machine learning models for efficient document classification and review.

Background Art

[0003] The review of legal documents is a labor - intensive and time - consuming process involving manual tagging by human reviewers. Modern machine - learning - based solutions such as Technology - Assisted Review (TAR) and Continuous Active Learning (CAL) require human reviewers to tag documents as examples for the machine to learn. This process requires a huge amount of time and resources and can take days, weeks, or months to complete. Furthermore, this process can cost millions of dollars.

[0004] Therefore, it would be advantageous to provide a solution to overcome the above - mentioned problems.

Summary of the Invention

[0005] A summary of some exemplary embodiments of this disclosure is provided below. This summary is provided for the convenience of the reader to provide a basic understanding of such embodiments and does not fully define the scope of this disclosure. This summary is not intended to be a comprehensive overview of all conceivable embodiments, nor to identify any key or definitive elements of all embodiments, nor to definitively describe the scope of any or all embodiments. Its sole purpose is to present some concepts of one or more embodiments in a simplified form as a prelude to the more detailed descriptions that will be presented later. For convenience, “some embodiments” or “specific embodiments” may be used herein to refer to a single or more embodiments of this disclosure.

[0006] Certain embodiments disclosed herein include a method for building a predictive machine learning model. The method comprises parsing the text of a document review protocol to extract at least one description of at least one concept to be tagged, and tagging at least a portion of the multiple documents to create multiple tagged documents by applying a language model to the multiple documents, wherein tagging at least a portion of the multiple documents further includes querying the language model with at least one query generated based on the extracted at least one description, building at least one classifier machine learning model based on the extracted at least one description, and training at least one classifier machine learning model using a training set, wherein the training set comprises multiple tagged documents.

[0007] Certain embodiments disclosed herein also include a stored non-temporary computer-readable medium which causes a processing circuit unit to execute a process, the process comprising: parsing the text of a document review protocol to extract at least one description of at least one concept to be tagged; tagging at least a portion of the multiple documents to create a plurality of tagged documents by applying a language model to the plurality of documents, the tagging at least a portion of the multiple documents further comprising querying the language model with at least one query generated based on the extracted at least one description; constructing at least one classifier machine learning model based on the extracted at least one description; and training at least one classifier machine learning model with a training set which the training set comprises the plurality of tagged documents.

[0008] Certain embodiments disclosed herein also include a system for building predictive machine learning models. The system comprises a processing circuit and a memory, the memory which, when executed by the processing circuit, parses text of a document review protocol to extract at least one description of at least one concept to be tagged, and tags at least a portion of the multiple documents to create a plurality of tagged documents by applying a language model to the plurality of documents, the tagging of at least a portion of the plurality of documents further includes querying the language model using at least one query generated based on the extracted at least one description, building at least one classifier machine learning model based on the extracted at least one description, and training at least one classifier machine learning model using a training set, the instruction which configures the system such that the training set includes the plurality of tagged documents.

[0009] Certain embodiments disclosed herein include the methods, non-temporary computer-readable media, or systems described above, further comprising or configured to perform one or more of the following steps: determining the structure of a document review protocol; identifying at least one concept to be tagged based on the structure of the document review protocol; identifying text that represents at least one rule for identifying at least one concept to be tagged in a plurality of documents, based on the structure of the document review protocol and the identified at least one concept to be tagged; and extracting the identified text.

[0010] Certain embodiments disclosed herein include the methods, non-temporary computer-readable media, or systems described above, wherein at least one concept to be tagged includes at least one case-specific concept shown in the document review protocol.

[0011] Certain embodiments disclosed herein include methods, non-temporary computer-readable media, or systems described above, further comprising or configured to perform one or more of the following steps: determining a score for each of a plurality of documents based on at least one extracted description of at least one concept to be tagged, wherein the determined score for each document represents the likelihood that the document contains text indicating a portion of at least one concept to be tagged.

[0012] Certain embodiments disclosed herein include methods, non-temporal computer-readable media, or systems described above, further comprising or configured to perform one or more of the following steps: updating at least one classifier machine learning model based on feedback data until each of at least one performance metric of at least one classifier model satisfies its respective performance threshold.

[0013] Certain embodiments disclosed herein include the methods, non-temporary computer-readable media, or systems described above, wherein the feedback data includes at least one feedback tag for multiple documents.

[0014] Certain embodiments disclosed herein include the methods, non-temporary computer-readable media, or systems described above, wherein the feedback data includes at least one feedback modification to a document review protocol.

[0015] Certain embodiments disclosed herein include the methods, non-temporary computer-readable media, or systems described above, wherein each document in at least a portion of a plurality of documents is tagged with its respective first tag, and further comprises or is configured to perform one or more of the following steps: determining a second tag for each document in at least a portion of the plurality of documents based on at least one feedback modification to a document review protocol; identifying at least one first document from the plurality of documents to be reviewed, wherein the second tag for each first document is different from the first tag for the first document; presenting at least one first document to the user for review; and re-tagging the plurality of documents based on the review.

[0016] Certain embodiments disclosed herein include methods, non-temporary computer-readable media, or systems described above, further comprising or configured to perform one or more of the following steps: an iterative determination of a subset of documents to be labeled based on user input, a query to the user based on the subset of documents determined in each iteration, the user providing user input indicating labels based on the subset of documents queried in each iteration, and a tagging of the determined subset of documents based on the labels indicated by the user input. [Brief explanation of the drawing]

[0017] The subject matter disclosed herein is specifically identified and explicitly claimed in the last claim herein. The aforementioned and other purposes, features and advantages of the disclosed embodiments will become apparent from reading the following description in conjunction with the accompanying drawings. [Figure 1] This is a network diagram used to illustrate the various embodiments disclosed. [Figure 2] This flowchart illustrates a method for building a predictive machine learning model for efficient document classification and review, according to one embodiment. [Figure 3] This is a schematic diagram of an artificial intelligence discovery assistant (AIDA) system according to one embodiment. [Figure 4] These are exemplary images used to illustrate the identification of relevant documents in at least some of the disclosed embodiments. [Figure 5] These are exemplary images used to illustrate the identification of relevant documents in at least some of the disclosed embodiments. [Figure 6] This is an illustrative diagram illustrating a performance review of a system configured according to a specific embodiment disclosed. [Modes for carrying out the invention]

[0018] The disclosed embodiments include systems and methods for building predictive machine learning models for document classification. The various disclosed embodiments provide efficient methods for classifying and reviewing documents while significantly reducing the need for human tagging. According to the various disclosed methods, an artificial intelligence (AI) discovery assistant (AIDA) is designed and adapts to handle general-world concepts (e.g., energy, automotive parts) and case-specific concepts (e.g., names or roles of people, organizations, case-specific facts) found in a document review protocol. The disclosed embodiments may be used, for example, to assist in the review of legal documents using a review protocol document, including its natural language content.

[0019] In one embodiment, the factual knowledge base is extracted from a set of case data containing people's names or roles, normalizing different names to link them to the same individuals and integrating various facts across the entire data. The text of the document review protocol is parsed to determine the definitions of concepts to be tagged (e.g., different issues, responsiveness, confidentiality privileges) and to identify specific rules or requirements for each section according to each concept or issue. For this purpose, in another embodiment, positive and negative descriptions within the text are distinguished from each other and these descriptions are structured into an internal machine representation.

[0020] In one embodiment, one or more classifiers are constructed to make probabilistic predictions for the entire collection of documents, scoring each document according to the likelihood that it belongs to a given issue or category described in the review protocol, based on the structured descriptions extracted from the protocol. The machine learning model may be continuously updated and improved for each category or issue based on additional modifications to the review protocol or clear example tags attached to documents (e.g., tags applied by user input). This process may be repeated until the model's performance meets one or more performance objectives (e.g., objectives defined based on one or more performance metrics that satisfy their respective thresholds). Documents with changed tags or scores may be identified and presented to the user for review.

[0021] To improve predictive performance more quickly, one or more active learning techniques are used to identify documents that should be manually tagged. Specifically, an active learning process may be used to determine which subsets of documents should be manually tagged, thereby reducing the amount of manual tagging work compared to manually tagging the entire set of documents, without sacrificing the performance of a model trained on automatically tagged documents. Active learning may be an iterative supervised learning process in which a learning algorithm interactively queries the user about labels. More specifically, the learning algorithm may select a subset of samples for user labeling (e.g., a subset of documents) and query the user about the labels that should be used to tag those samples. By using a learning algorithm that selects a portion of the samples for user labeling, it is possible to reduce the amount of manual labeling required to label the training set containing the samples.

[0022] In one embodiment, case data is a collection of documents or includes a collection of documents. In one embodiment, a document is a set of documents, and each document includes at least text. A document may include text, text and metadata, text and native, combinations thereof, and the like. In another embodiment, a tag or category is a binary classification applied at the document level. In yet another embodiment, the arguments of a document review are tags or categories particularly related to litigation.

[0023] In one embodiment, responsiveness in litigation is defined such that documents tagged with a "responsive" or "responsiveness" tag are considered responsive for such a use case, based on a particular tag or category that is most important for a particular use case or otherwise to be utilized as compared to other tags or categories (e.g., based on one or more predetermined indicators of importance or otherwise, based on an input indicating responsiveness to a tag or category). As a non-limiting example, a document requested by a court or an opposing party during litigation may be considered responsive such that the document may be tagged as "responsive". [[ID=�]]

[0024] In one embodiment, privilege in litigation is defined such that a "privileged" tag indicates that a document has been given privilege with respect to a particular tag or category of documents that should not be seen by an opposing party. As a non-limiting example, a document containing communications with an attorney or a spouse may be made privileged such that the document may be tagged as "privileged".

[0025] [[ID=]] In one embodiment, a classifier is a machine learning model trained to receive as input some data (e.g., a document containing text) and output a classification corresponding to a category or tag for that data. Either a tag or a category may be associated with a predetermined indicator such as responsiveness, privilege, and the like (but not limited thereto).

[0026] Figure 1 shows an exemplary network diagram 100 used to illustrate various embodiments disclosed. In this embodiment, in network diagram 100, a user device 120, an artificial intelligence (AI) discovery assistant (AIDA) system 130, and several databases 140-1 to 140-N (hereinafter, for the sake of brevity, individually referred to as database 140 and collectively as database group 140) communicate via network 110. Network 110 may be, but is not limited to, a wireless network, a cellular network or a wired network, a local area network (LAN), a wide area network (WAN), a metro area network (MAN), the Internet, the World Wide Web (WWW), similar networks, and any combination thereof.

[0027] The user device 120 may be, but is not limited to, a personal computer, laptop, tablet computer, smartphone, wearable computing device, or any other device capable of receiving and displaying notifications. The user device 120 may receive, for example, an electronic document to be tagged from the AIDA system 130, and may also receive input from a user (not shown) indicating the tags to be assigned to that electronic document.

[0028] The AIDA system 130 is configured to perform at least a portion of the disclosed embodiments. To this end, the AIDA system 130 may communicate with the user device 120 to receive data (e.g., text or audio data) indicating a document review protocol that defines concepts to be tagged according to various disclosed embodiments. More specifically, the AIDA system 130 is configured to train and utilize a machine learning model described herein to classify electronic documents based on the review protocol. Furthermore, as described above, the AIDA system 130 may be configured, for example, to allow the user of the user device 120 to select a subset of documents to be manually labeled.

[0029] Database 140 may store data such as, but not limited to, case data, electronic documents, or both, and similar data. Such case data may include, but not limited to, names or roles of people, for example, normalizing different names to link them to the same individual and integrating various facts throughout the data.

[0030] Network diagram 100 shows an exemplary environment in which various embodiments of the disclosed embodiments can be implemented; however, it should be noted that the disclosed embodiments are not limited to the environment illustrated in network diagram 100. The various embodiments of the disclosed embodiments can be deployed in other environments without departing from the scope of the disclosure.

[0031] Figure 2 is an exemplary flowchart illustrating a method for building a predictive machine learning model for efficient document classification and review according to one embodiment. In one embodiment, the method is carried out by the AIDA system 130 of Figure 1.

[0032] In S210, a factual knowledge base is extracted from the case data. In one embodiment, the factual knowledge base is extracted from a set of case data that includes people's names and roles, normalizing different names to link them to the same individuals and integrating various facts from the entire data. In this regard, the factual knowledge base indicates case-specific information (e.g., specific entities related to a given situation, entities with a certain quality, etc.) that may be subject to a document review protocol. That is, the document review protocol may indicate concepts to be tagged about a document, and the factual knowledge base may provide information about text that may indicate those concepts to be tagged.

[0033] In one embodiment, S210 includes applying a set of concept identification rules to the case data to identify portions of the case data that represent a concept represented in the case data. Such concept identification rules may define terms, phrases, grammatical structures, or other indicators of a potential concept. The concept identification rules may differ by embodiment. In a non-limiting embodiment, concept identification rules in a legal embodiment may include rules that identify concepts such as legal entities, individuals, legal documents (e.g., contracts), language that suggests legally relevant facts (e.g., language that suggests an offer, acceptance, or consideration of a contract), combinations thereof, and similar.

[0034] In S220, the text of the document review protocol is analyzed. Analysis may be performed to identify and understand the definitions of concepts defined in the document review protocol. To this end, in one embodiment, the text is analyzed to extract one or more descriptions of concepts to be tagged. In other words, analysis is performed to identify a structure that describes how documents should be tagged.

[0035] In one embodiment, analyzing a document review protocol includes determining the structure of the document review protocol and identifying concepts to be tagged within the text of the document review protocol based on that structure. Text indicating rules for identifying concepts to be tagged is identified from the document review protocol based on the structure and the identified concepts. The identified text may be extracted for later use.

[0036] More specifically, the structure of the document review protocol may be analyzed to identify which parts of the protocol contain rules for identifying concepts, and the text to be identified may be the text contained within those parts that contains rules based on the concepts to be tagged. For this purpose, a part of the document review protocol is identified that contains such rules, as determined based on the structure, and that indicates one of the concepts to be tagged. In other words, the identified text contains text that indicates the rules used to identify the concepts to be tagged.

[0037] In an unrestrictive implementation, the document review protocol may define certain concepts as privileged, certain concepts as responsive, or both, and similarly. The portions of data corresponding to such definitions within the document review protocol are parsed from the rest of the data and extracted for use as definitions of concepts to be tagged during subsequent processing.

[0038] In another embodiment, parsing the text of a document review protocol includes distinguishing between positive and negative statements within the text and structuring these statements into an internal machine representation. Positive and negative statements may be identified accordingly, and such identification of positive or negative statements may be used, for example, to aid in document tagging.

[0039] In S230, to create a set of tagged documents, at least a portion of the set of documents is tagged. In one embodiment, a portion of the documents is tagged by applying a language model to the documents. In another embodiment, tagging a portion of the documents includes querying a language model (e.g., LLM) using one or more queries generated based on the extracted descriptions of concepts to be tagged, and providing the documents to the language model. The language model returns output containing document tags indicating which documents contain text representing concepts to be tagged.

[0040] In some embodiments, at least some documents in a set of documents may be tagged based on user input. In such embodiments, a learning model may be applied to the documents to determine which subset of documents should be labeled based on user input. To present the subset of documents to the user (e.g., via user device 120 in Figure 1), the user is queried with the determined subset of documents and user input indicating the labels of the documents is received from the user.

[0041] In another embodiment, an active learning method is used to identify documents that should be manually tagged via user input over multiple iterations. In such an embodiment, a learning model may be applied to the documents in each iteration to determine a subset of documents to be labeled and to query the user based on the subset of documents determined in that iteration. In each subsequent iteration after the initial iteration, the labels applied by the user via user input during the previous iteration are input to the learning model along with the documents. The learning model is trained to select documents to label based on the concepts to be tagged and such past labels.

[0042] In this regard, the active learning process enables iterative labeling to reduce the amount of manual labeling and speed up the labeling process. That is, labels may be determined in iterations, and the learning model determines which documents should be manually labeled in each iteration. Labels made in previous iterations may be used in subsequent iterations to further identify which documents should be labeled based on user input in those subsequent iterations, thereby reducing the amount of documents that would be selected to be manually labeled in each subsequent iteration. The result is a reduction in the overall number of documents that need to be manually labeled, while the model trained using documents tagged with the labels determined in this step achieves equivalent or better performance.

[0043] In S240, one or more classifier machine learning models (also called classifiers) are constructed based on descriptions extracted and constructed from the protocol to make probabilistic predictions for the entire collection of documents. Each classifier is a machine learning model configured to output one or more classifications for an input document. To this end, each classifier may be configured to output a set of scores corresponding to each classification for a given document, each score indicating the likelihood that its respective classification is accurate for that document. Thus, the scores may be used to determine one or more classifications for an input document.

[0044] In some embodiments, a classifier may be constructed for each latent concept in the knowledge base, and each such classifier may output a score representing the likelihood that an input document exhibits that concept. In another embodiment, when attempting to tag documents according to a document review protocol, only a subset of classifiers is applied to the documents. Such a subset of classifiers may include classifiers corresponding to concepts to be tagged, extracted from the document review protocol.

[0045] In one embodiment, the classifier is constructed to output a score for each classification corresponding to different concepts to be tagged. That is, each classification may correspond to one of the concepts to be tagged, and the score for that classification may represent whether the input document belongs to that concept or otherwise contains text related to that concept.

[0046] In S250, the classifier machine learning model is trained based on tagged documents. In one embodiment, the classifier is trained using a training set which includes a set of training inputs and a set of training outputs. The set of training inputs includes training documents or a set of training text for documents, and the training outputs include each classification of the training documents or the set of training document texts. More specifically, for each tagged document, the document or text from the document may be used as the training input, and the classification indicated by the document's tags may be used as the training label for that input (i.e., the label represents the corresponding output).

[0047] In S260, the classifier machine learning model is updated and improved based on feedback data. In one embodiment, each classifier may be updated continuously, iteratively, or otherwise until one or more performance metrics meet their respective performance thresholds. The feedback data may include, but is not limited to, feedback tags (e.g., tags applied as user input indicating a “correct” tag and compared to the classifier's output), feedback modifications to a document review protocol (e.g., a new version of the document review protocol or a part thereof), or both and similar.

[0048] When feedback data includes feedback tags, these tags may be used to adjust the weights of the corresponding classifier accordingly. For this purpose, in such embodiments, the feedback tags may be used as labels for each document during additional training iterations, instead of the original tags used in the pre-training iterations, thereby updating the model based on this additional training using the feedback tags.

[0049] When feedback data includes feedback modifications to the document review protocol, updating the classifier machine learning model may further include determining a second new tag for the document using concepts to be tagged extracted from the modified version of the document review protocol (i.e., the version containing the modifications or a different new version of the document review protocol). To this end, in such embodiments, updating the classifier may further include parsing the modified document review protocol to extract a new set of concepts to be tagged (e.g., using the process described above with respect to S220), and querying the language model about the document using queries generated based on the newly extracted concepts to be tagged, and retraining the classifier based on the output of the language model.

[0050] In another embodiment, the new tag for each document may be compared to a first previous tag to determine whether the new tag for that document differs from the previous tag for that document (i.e., whether the tag has been changed after the document review protocol has been modified), and each document with a changed tag may be presented to the user for review. The review may include the user manually reviewing each such document and providing a set of user inputs including manually created labels for that document. Documents may be re-tagged based on the review, for example, by re-tagging the document with labels applied by the user. This change detection and review would allow for the correction of potentially misidentified tags using a language model to ensure that a model trained with tagged documents functions properly, while further reducing the number of documents that need to be manually tagged.

[0051] In S270, the trained classifier is applied. More specifically, the trained model is applied to one or more documents to determine their classification.

[0052] The applied models, when trained using at least some of the techniques described herein, will be able to be trained more efficiently, and in at least some embodiments, will achieve similar or better performance with less training data (especially human-generated tags) compared to various existing solutions.

[0053] With the optional S280, documents to which a trained model has been applied may be tagged using the classification output by the model.

[0054] Figure 3 is an exemplary schematic diagram of the hardware layer 300 of an AIDA system, such as the AIDA system 130, according to one embodiment. The hardware layer 300 includes a memory 320, a storage device 330, and a processing circuit section 310 connected to a network interface 340. In one embodiment, the components of the hardware layer 300 may be communicated via a bus 350.

[0055] The processing circuit section 310 may be implemented as one or more hardware logic components and circuits. Examples of hardware logic components that can be used, but are not limited to these, include, but are, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-chip systems (SOCs), graphics processing units (GPUs), tensor processing units (TPUs), general-purpose microprocessors, microcontrollers, digital signal processors (DSPs), and similar devices, or any other hardware logic components capable of performing calculations or other operations on information.

[0056] Memory 320 may be volatile (e.g., random access memory), non-volatile (e.g., read-only memory, flash memory), or a combination thereof.

[0057] In one configuration, software for carrying out one or more embodiments disclosed herein may be stored in the storage device 330. In another configuration, memory 320 may be configured to store such software. Software should be broadly interpreted to mean any type of instruction, whether called software, firmware, middleware, microcode, hardware description language, or otherwise. Instructions may include code (e.g., in source code format, binary code format, executable code format, or any other suitable code format). When executed by the processing circuit unit 310, the instructions cause the processing circuit unit 310 to perform the various processes described herein.

[0058] The storage device 330 may be a magnetic storage device, an optical storage device, or similar, and may be implemented as, for example, flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital general-purpose disc (DVD), or any other medium that can be used to store desired information.

[0059] The network interface 340 allows the hardware layer 300 to communicate with, for example, user devices 120, database groups 140, and similar devices.

[0060] It should be understood that the embodiments described herein are not limited to the specific architecture illustrated in Figure 3, and other architectures may also be used without departing from the scope of the disclosed embodiments.

[0061] The disclosed embodiments provide benefits that outweigh certain solutions such as manual keyword searches and tape archiving tools, including but not limited to efficiently providing responsive document snippets. Examples of such benefits, as can be seen in Figures 4 and 5, are non-limiting embodiments; these figures illustrate exemplary images 400 and 500, respectively, showing sample snippets. Specifically, Figure 4 shows image 400 of a document review protocol snippet defining responsiveness, and Figure 5 shows image 500 of a snippet from a document that has been found to be responsive (e.g., by an AIDA system 130) according to various embodiments disclosed.

[0062] As illustrated in Figures 4 and 5, various keywords or terms are highlighted by dotted rectangles surrounding their respective texts. In Figure 4, the terms “Executive Management”410, “Official”420, and “Natural Gas Regulation”430 are highlighted. In Figure 5, the terms “Ken Lei”510, “Vice President”520, and “National Energy Policy”530 are highlighted. The highlighted terms indicate relevant concepts used to identify the document as responsive in the various embodiments disclosed.

[0063] Despite the lack of a common language between documents and review protocol snippets, AIDA systems (e.g., AIDA System 130) can efficiently identify relevant documents based on protocol specifications. In contrast, manual keyword searches or solutions such as TAR tools may face additional challenges in identifying relevant documents.

[0064] Some existing solutions rely on learning from exemplary documents, meaning that in order to train machine learning models to identify related documents (i.e., documents that are related insofar as they are responsive to the portion of the document illustrated in Figure 4), such as the exemplary portion of the related document illustrated in Figure 5, reviewers must tag a large number of documents. Therefore, these tools lack the ability to leverage external knowledge to address the problem, do not efficiently recognize individual roles or responsibilities, and require reviewers to tag even more examples to gain sufficient applicability. The result is a massive amount of manual tagging of the samples used for training.

[0065] Furthermore, analysis using a manual approach would require tedious keyword searches to identify potential words that link document snippets to the responsiveness requirements of the review protocol. This process is time-consuming, expensive, and prone to errors, as it involves thorough searches across numerous keywords and relies on a potentially incomplete list of people and their roles. Moreover, this process often involves a great deal of subjective judgment on behalf of the analyst. As a result, this manual process can lead to sets of tagged documents that vary significantly among manual reviewers.

[0066] The disclosed embodiments are capable of overcoming at least these challenges. In particular, such challenges may be overcome at least in part by analyzing a complete document collection to extract structured knowledge and utilizing a Large-Scale Language Model (LLM) to understand general concepts such as “energy” without requiring reviewers to manually tag numerous examples that apply to a topic. More specifically, the extracted structured knowledge may include case-specific entities such as people and organizations, as well as their relationships, including roles and positions, or it may include case-specific entities such as people and organizations, as well as their relationships, including roles and positions. Identifying these case-specific entities enables the recognition of individual executives based on a document review protocol via machine learning, without requiring manual example tagging.

[0067] The combination of structured case-specific knowledge and general knowledge from the LLM, as disclosed in various embodiments, enables the system to leverage the description of the document review protocol to identify relevant documents without or with a reduction in the number of human-tagged examples. This reduces the amount of manual review and minimizes potential inconsistencies among manual reviewers that arise from accumulated differences in subjective judgment.

[0068] Therefore, the various embodiments disclosed reduce the amount of manual review and tagging while achieving the same or better performance (e.g., in terms of accuracy or recall) than at least some existing solutions. By building the initial model directly from the review protocol, at least some of the disclosed embodiments accelerate the review process, resulting in cost and time savings. Furthermore, the various embodiments disclosed overcome the challenges of the cold-start problem faced by existing machine learning solutions, providing the active learning system with sufficient information to identify examples worthy of manual tagging, and consequently improving predictive performance.

[0069] Figure 6 is an exemplary figure illustrating the efficiency benefits of at least some of the disclosed embodiments. To assess the efficiency of AIDA, a comprehensive evaluation was conducted using a benchmark set of five cases provided by law firms. AIDA's performance was directly compared to both manual reviews and certain existing solutions of TAR systems.

[0070] As illustrated in Figure 6, the evaluation focused on comparing the number of documents that each system, including AIDA, needed to review manually to reach its target performance level in terms of accuracy and recall. This metric was chosen as a key indicator of the efficiency and effectiveness of the document review process because reducing the number of documents that require manual review can significantly reduce the time and cost of the review process.

[0071] The evaluation results demonstrated improved performance of AIDA systems configured according to at least some of the disclosed embodiments compared to existing automated and manual solutions. In some non-restrictive embodiments, AIDA proved to require 1 / 3 to 1 / 12 less data for documents reviewed by humans compared to certain existing solutions. Such a substantial reduction in the amount of documents reviewed by humans to achieve similar or better performance highlights the effectiveness of AIDA's approach, which combines the extraction of structured, case-specific knowledge with general knowledge from large-scale language models to better identify relevant documents based on document review protocols.

[0072] This evaluation confirms that AIDA can be used to improve efficiency compared to both traditional manual review methods and existing solutions for TAR systems, and that AIDA is a highly effective and efficient solution for document review processes in legal environments.

[0073] It is important to note that the embodiments disclosed herein are merely examples of many advantageous uses of the innovative teachings herein. In general, the descriptions set forth in this specification are not necessarily limited to any of the various embodiments claimed. Furthermore, some descriptions may apply to some inventive features but not to others. In general, unless otherwise indicated, singular elements may be plural and vice versa without loss of generality. In the drawings, similar numbers refer to the same part through multiple drawings.

[0074] The various embodiments disclosed herein can be implemented as hardware, firmware, software, or any combination thereof. Software may also be implemented as an application program tangibly embodied in a program storage device or computer-readable medium consisting of components, or certain devices and / or combinations of devices. The application program may be uploaded to and executed by a machine having any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware (such as one or more central processing units ("CPU"), memory, and input / output interfaces). The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be executed by a CPU, whether such a computer or processor is explicitly shown, and may be part of the microinstruction code, part of the application program, or any combination thereof. In addition, various other peripheral devices, such as additional data storage devices and printing devices, may be connected to the computer platform. Furthermore, non-transient computer-readable medium is any computer-readable medium except for transient propagating signals.

[0075] All embodiments and conditional language described herein are intended for educational purposes to help the reader understand the principles of the disclosed embodiments and the inventors' concepts for the advancement of the art, and should be construed as not being limited to the embodiments and conditions described herein. Furthermore, all descriptions herein describing the principles, aspects and embodiments of the disclosed embodiments, as well as specific examples thereof, are intended to encompass both their structural and functional equivalents. In addition, such equivalents include both currently known equivalents and future-developed equivalents, i.e., any elements developed to perform the same function, regardless of their structure.

[0076] It should be understood that references to elements using markings such as “First,” “Second,” etc., in this specification do not generally limit the number or order of those elements. Rather, these markings are commonly used in this specification as a convenient way to distinguish between two or more elements or between two or more instances of an element. Therefore, references to the First and Second elements do not mean that only two elements can be adopted, or that the First element must somehow precede the Second element. Also, unless otherwise stated, a set of elements includes one or more elements.

[0077] As used herein, the phrase “at least one of” following an enumeration of items means that any of the enumerated items may be used individually, or any combination of two or more of the enumerated items may be used. For example, if it is stated that a system includes “at least one of A, B, and C,” the system may include A alone, B alone, C alone, 2A, 2B, 2C, 3A, a combination of A and B, a combination of B and C, a combination of A and C, a combination of A, B, and C, a combination of 2A and C, a combination of A, 3B, and 2C, and so on.

Claims

1. A method for building predictive machine learning models, The text of the document review protocol is parsed to extract at least one description of at least one concept that should be tagged, Tagging at least a portion of a plurality of documents in order to create a plurality of tagged documents by applying a language model to the plurality of documents, wherein tagging at least a portion of the plurality of documents further includes querying the language model using at least one query generated based on the extracted at least one description, Constructing at least one classifier machine learning model based on the at least one extracted description, A method for training the at least one classifier machine learning model using a training set, wherein the training set includes the plurality of tagged documents.

2. Analyzing the aforementioned document review protocol is To determine the structure of the document review protocol, Based on the structure of the document review protocol, identify at least one concept to be tagged, Based on the structure of the document review protocol and the identified at least one concept to be tagged, identify text that shows at least one rule for identifying the at least one concept to be tagged within the plurality of documents, The method according to claim 1, further comprising extracting the identified text.

3. The method according to claim 1, wherein the at least one concept to be tagged includes at least one case-specific concept shown in the document review protocol.

4. The method according to claim 1, further comprising determining a score for each of the plurality of documents based on the extracted at least one description of the at least one concept to be tagged, wherein the determined score for each document represents the likelihood that the document contains text indicating a portion of the at least one concept to be tagged.

5. The method according to claim 1, further comprising updating the at least one classifier machine learning model based on feedback data until each of the at least one performance metrics of the at least one classifier machine learning model satisfies its respective performance threshold.

6. The method according to claim 5, wherein the feedback data includes at least one feedback tag for the plurality of documents.

7. The method according to claim 5, wherein the feedback data includes at least one feedback modification to the document review protocol.

8. Each of the documents, at least a portion of the aforementioned plurality of documents, is tagged with its respective first tag. Based on the at least one feedback modification to the document review protocol, a second tag is determined for each of the documents in at least a portion of the plurality of documents, Identifying at least one first document to be reviewed from among the plurality of documents, wherein the second tag for each of the first documents is different from the first tag for the first document. To present the user with at least one of the first documents for the purpose of the review, The method of claim 7, further comprising re-tagging the plurality of documents based on the aforementioned review.

9. The process involves iteratively determining a subset of the multiple documents to be labeled based on user input, and querying the user based on the subset of documents determined in each iteration, wherein the user provides user input indicating the label based on the subset of documents queried in each iteration, and the process involves iterative determination and querying the user. The method according to claim 1, further comprising tagging a subset of the determined documents based on the labels indicated by the user input.

10. A non-temporary computer-readable medium storing instructions for executing a process in a processing circuit unit, wherein the process is: The text of the document review protocol is parsed to extract at least one description of at least one concept that should be tagged, Tagging at least a portion of a plurality of documents in order to create a plurality of tagged documents by applying a language model to the plurality of documents, wherein tagging at least a portion of the plurality of documents further includes querying the language model using at least one query generated based on the extracted at least one description, Constructing at least one classifier machine learning model based on the at least one extracted description, Training the at least one classifier machine learning model using a training set, wherein the training set comprises the plurality of tagged documents in non-temporary computer-readable media.

11. A system that uses artificial intelligence to assist in the review of legal documents, Processing circuit section, It includes memory, and the memory, when executed by the processing circuit, The text of the document review protocol is parsed to extract at least one description of at least one concept that should be tagged, Tagging at least a portion of a plurality of documents in order to create a plurality of tagged documents by applying a language model to the plurality of documents, wherein tagging at least a portion of the plurality of documents further includes querying the language model using at least one query generated based on the extracted at least one description, Constructing at least one classifier machine learning model based on the at least one extracted description, A system comprising training the at least one classifier machine learning model using a training set, the instruction to configure the system such that the training set includes the plurality of tagged documents.

12. The aforementioned system, To determine the structure of the document review protocol, Based on the structure of the document review protocol, identify at least one concept to be tagged, Based on the structure of the document review protocol and the identified at least one concept to be tagged, identify text that shows at least one rule for identifying the at least one concept to be tagged within the plurality of documents, The system according to claim 11, further configured to extract the identified text.

13. The system according to claim 11, wherein the at least one concept to be tagged includes at least one case-specific concept shown in the document review protocol.

14. The aforementioned system, The system according to claim 11, further configured to determine a score for each of the plurality of documents based on the extracted at least one description of the at least one concept to be tagged, wherein the determined score for each document represents the likelihood that the document contains text indicating a portion of the at least one concept to be tagged.

15. The aforementioned system, The system according to claim 11, further configured to update the at least one classifier machine learning model based on feedback data until each of the at least one performance metrics of the at least one classifier machine learning model satisfies its respective performance threshold.

16. The system according to claim 15, wherein the feedback data includes at least one feedback tag for the plurality of documents.

17. The system according to claim 15, wherein the feedback data includes at least one feedback modification to the document review protocol.

18. Each of the documents in at least a portion of the plurality of documents is tagged with its respective first tag, and the system Based on the at least one feedback modification to the document review protocol, a second tag is determined for each of the documents in at least a portion of the plurality of documents, Identifying at least one first document to be reviewed from among the plurality of documents, wherein the second tag for each first document is different from the first tag for the first document. To present the user with at least one of the first documents for the purpose of the review, The system according to claim 17, further configured to re-tag the plurality of documents based on the aforementioned review.

19. The aforementioned system, The process involves iteratively determining a subset of the multiple documents to be labeled based on user input, and querying the user based on the subset of documents determined in each iteration, wherein the user provides user input indicating the label based on the subset of documents queried in each iteration, and the process involves iterative determination and querying the user. The system according to claim 11, further configured to tag a subset of documents determined based on the labels indicated by the user input.

Images