Structured document generation in natural language processing systems

Abstract

A system may receive from a client machine a request to determine a structured document based on an exemplar document and a natural language input instruction characterizing content to include in the structured document. The system may determine document structure information based on an analysis of the exemplar document. The document structure information may identify a plurality of document sections and may characterize content to include in the plurality of document sections. The system may query a database system to identify a plurality of example document portions corresponding to the plurality of document sections. The system may determine a structured document including a plurality of novel text portions corresponding with the plurality of document sections. The novel text portions may be generated by a large language model based on the plurality of example document portions and the natural language input instruction.

Description

STRUCTURED DOCUMENT GENERATION IN NATURAL LANGUAGE PROCESSING SYSTEMSCross-Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application 63 / 511,297 (Attorney Docket No. CSTXP008P) by DeFoor et al., titled "STRUCTURED DOCUMENT GENERATION IN NATURAL LANGUAGE PROCESSING SYSTEMS", filed on June 30, 2023, which is incorporated herein by reference in its entirety and for all purposes.Field of Technology

[0002] This patent document relates generally to natural language processing systems and more specifically to systems involving large language models.Background

[0003] Natural language processing systems are increasingly sophisticated. Large language models are pre-trained to generate text. A large language model may be provided with input text, such as a question. The model may then provide output text in response, such as an answer to the question. Recent advances have led large language models to become increasingly powerful, often able to produce text that approaches that which would be generated by humans. However, large language models typically perform best when generating small portions of text and perform worst when generating large, structured documents. Accordingly improved techniques for generating structured documents are needed.Summary

[0004] Described herein are systems, methods, and computer readable media having instructions stored thereon for generating structured documents. According to various embodiments, operations may involve receiving from a client machine a request to determine a structured document based on an exemplar document and a natural language input instruction characterizing content to include in the structured document, determining via a processordocument structure information based on an analysis of the exemplar document, the document structure information identifying a plurality of document sections and characterizing content to include in the plurality of document sections. A database system may be queried to identify a plurality of example document portions corresponding to the plurality of document sections. A structured document including a plurality of novel text portions corresponding with the plurality of document sections may be determined. The plurality of novel text portions may be generated by a large language model based on the plurality of example document portions and the natural language input instruction. The structured document may be transmitted to the client machine.

[0005] According to various embodiments, operations may involve determining an initialized structured document based on the document structure information and including a plurality of document structure portions corresponding to the plurality of document sections and contextual information portions characterizing content to be included in the document structure portions.

[0006] According to various embodiments, the plurality of document structure portions and the plurality of contextual information portions may be determined by the large language model based on the exemplar document and a document initialization natural language instruction to determine the initialized structured document.

[0007] According to various embodiments, operations may involve determining a document initialization prompt based on a document initialization prompt template, the document initialization prompt and the document initialization prompt template both including the document initialization natural language instruction, where determining the document initialization prompt template involves filling a fillable portion included in the document initialization prompt template with a portion of the exemplar document.

[0008] According to various embodiments, operations may involve determining a designated portion of the exemplar document corresponding with adesignated document section and extracting one or more clauses from the designated portion of the exemplar document, where the database system is queried based on the one or more clauses.

[0009] According to various embodiments, the one or more clauses are extracted by the large language model based on a metadata extraction prompt that includes the designated portion of the exemplar document and a metadata extraction natural language instruction to extract the one or more clauses.

[0010] In some aspects, querying the database system may involve executing a designated database query determined by a large language model based on a query prompt that includes a designated portion of the exemplar document corresponding with a designated document section and a query identification natural language instruction to identify the designated database query, where the designated database query returns a plurality of designated example document portions identified by the database system as being similar to the designated portion of the exemplar document.

[0011] In some aspects, determining the document structure information involves determining a document structure prompt based on a document structure prompt template, the document structure prompt and the document structure prompt template both including the document structure natural language instruction, where determining the document structure prompt template involves filling a fillable portion included in the document structure prompt template with a portion of the exemplar document.

[0012] According to various embodiments, determining the document structure information may include providing the document structure prompt to the large language model and extracting the document structure information from a completed document structure prompt determined by the large language model based on the document structure prompt.

[0013] According to various embodiments, determining the structured document may involve determining a first novel text portion based on the natural language input instruction, determining a second novel text portion ofthe plurality of text portions based on the natural language input instruction and the first novel text portion, and determining a third novel text portion of the plurality of text portions based on the natural language input instruction and the first and second novel text portions.

[0014] According to various embodiments, determining the second novel text portion may involve determining a structured document generation prompt based on a structured document generation prompt template, the structured document generation prompt and the structured document generation prompt template both including a structured document generation natural language instruction to generate novel text based on input, where determining the structured document generation prompt involves filling one or more fillable portions of the structured document generation prompt template with the structured document generation natural language instruction and the first novel text portion.

[0015] According to various embodiments, determining the second novel text portion may involve parsing a first completed structured document generation prompt determined by the large language model, extracting the first novel text portion from the first completed structured document generation prompt, providing the structured document generation prompt to the large language model, and extracting the second novel text portion from a second completed structured document generation prompt determined by the large language model in response to the structured document generation prompt.Brief Description of the Drawings

[0016] The included drawings are for illustrative purposes and serve only to provide examples of possible structures and operations for the disclosed inventive systems, apparatus, methods, and computer program products for natural language processing. These drawings in no way limit any changes in form and detail that may be made by one skilled in the art without departing from the spirit and scope of the disclosed implementations.

[0017] Figure 1 illustrates an overview method for generating novel text, performed in accordance with one or more embodiments.

[0018] Figure 2 illustrates a text generation system, configured in accordance with one or more embodiments.

[0019] Figure 3 illustrates a document parsing method, performed in accordance with one or more embodiments.

[0020] Figure 4 illustrates a text generation method, performed in accordance with one or more embodiments.

[0021] Figure 5 illustrates a method of sharding text, performed in accordance with one or more embodiments.

[0022] Figure 6 illustrates a method for determining a text chunk, performed in accordance with one or more embodiments.

[0023] Figure 7 illustrates one example of a computing device, configured in accordance with one or more embodiments.

[0024] Figure 8 illustrates a hallucination detection method, performed in accordance with one or more embodiments.

[0025] Figure 9 illustrates a method for determining a document structure, performed in accordance with one or more embodiments.

[0026] Figure 10 illustrates a method for initializing a document structure, performed in accordance with one or more embodiments.

[0027] Figure 11 illustrates a method for determining a structured document, performed in accordance with one or more embodiments.

[0028] Figure 12 illustrates a database generation and querying overview method, performed in accordance with one or more embodiments.

[0029] Figure 13 illustrates a database system updating method, performed in accordance with one or more embodiments.

[0030] Figure 14 illustrates a database system query and filter determination method, performed in accordance with one or more embodiments.

[0031] Figure 15 illustrates a database system query and filter execution method, performed in accordance with one or more embodiments.Detailed Description

[0033] Techniques and mechanisms described herein provide for the generation of structured documents. One or more exemplar documents may be analyzed to determine a document structure, including a document outline and / or information about the rhetorical content of a document. The document structure may then be combined with input text to initialize the structured document. The initialized structure document may be used as a basis for generating the final structured document.

[0034] According to various embodiments, techniques and mechanisms described herein provide for novel text generation in domain-specific contexts. A text generation interface system may take as input one or more arbitrary documents, process them via optical text recognition, segment them into portions, and process the segmented text via various tasks based on need. Different workflows are provided for different tasks, and this application describes a number of examples of such workflows. In many workflows, an input document is divided into chunks via a chunking technique. Then, chunks are inserted into prompt templates for processing by a large language model such as the GPT-3 or GPT-4 available from OpenAL The large language model's response is then parsed and potentially used to trigger additional analysis, such as one or more database searches, one or more additional prompts sent back to the large language model, and / or a response returned to a client machine.

[0035] According to various embodiments, techniques and mechanisms described herein provide for a sophisticated document processing pipeline. The pipeline receives one or more input documents, identifies text that should be kept together, identifies extraneous text such as headers, footers, and line numbers, and segments the text accordingly. In this way, the quality of the text provided to the rest of the system is improved.

[0036] According to various embodiments, techniques and mechanisms described herein provide for new approaches to text segmentation. Large language models often receive as input a portion of input text and generate in response a portion of output text. In many systems, the large language modelimposes a limit on the input text size. Accordingly, in the event that the large language model is asked to summarize a length document, the document may need to be segmented into portions in order to achieve the desired summarization.

[0037] Conventional text segmentation techniques frequently create divisions in text that negatively affect the performance of the model, particularly in domains-specific contexts such as law. For example, consider a caption page of a legal brief, which includes text in a column on the left that encompasses the parties, text in a column on the right that includes the case number, a title that follows lower on the page, and line numbering on the left. In such a configuration, the text in the different columns should not be mixed and should be treated separately from the line numbers, while both columns should precede the document title, when converting the document to an input query for a large language model. However, conventional techniques would result in these semantically different elements of text being jumbled together, resulting in an uninformative query provided to the large language model and hence a low-quality response. In contrast to these conventional techniques, techniques and mechanisms described herein provide for a pipeline that cleans such raw text so that it can be provided to a large language model.

[0038] According to various embodiments, techniques and mechanisms described herein provide for the division of text into chunks, and the incorporation of those chunks into prompts that can be provided to a large language model. For instance, a large language model may impose a limit of, for instance, 8,193 tokens on a task, including text input, text output, and task instructions. In order to process longer documents, the system may split them. However, splitting a document can easily destroy meaning depending on where and how the document is split. Techniques and mechanisms described herein provide for evenly splitting a document or documents into chunks, and incorporating those chunks into prompts, in ways that retain the semantic content associated with the raw input document or documents.

[0039] In some embodiments, techniques and mechanisms described herein may be applied to generate novel text in domain-specific contexts, such as legal analysis. Large language models, while powerful, have a number of drawbacks when used for technical, domain-specific tasks. When using conventional techniques, large language models often invent "facts" that are actually not true. For instance, if asked to summarize the law related to non-obviousness in the patent context, a large language model might easily invent a court case, complete with caption and ruling, that in fact did not occur. In contrast to conventional techniques, techniques and mechanisms described herein provide for the generation of novel text in domain-specific contexts while avoiding such drawbacks.

[0040] According to various embodiments, techniques and mechanisms described herein may be used to automate complex, domain-specific tasks that were previously the sole domain of well-trained humans. Moreover, such tasks may be executed in ways that are significantly faster, less expensive, and more auditable than the equivalent tasks performed by humans. For example, a large language model may be employed to produce accurate summaries of legal texts, to perform legal research tasks, to generate legal documents, to generate questions for legal depositions, and the like.

[0041] In some embodiments, techniques and mechanisms described herein may be used to divide text into portions while respecting semantic boundaries and simultaneously reducing calls to the large language model. The cost of using many large language models depends on the amount of input and / or output text. Accordingly, techniques and mechanisms described herein provide for reduced overhead associated with prompt instructions while at the same time providing for improved model context to yield an improved response.

[0042] In some embodiments, techniques and mechanisms described herein may be used to process an arbitrary number of unique documents (e.g., legal documents) that cannot be accurately parsed and processed via conventional optical character recognition and text segmentation solutions.

[0043] In some embodiments, techniques and mechanisms described herein may be used to link a large language model with a legal research database, allowing the large language model to automatically determine appropriate searches to perform and then ground its responses to a source of truth (e.g., in actual law) so that it does not "hallucinate" a response that is inaccurate.

[0044] In some embodiments, techniques and mechanisms described herein provide for specific improvements in the legal domain. For example, tasks that were previously too laborious for attorneys with smaller staffs may now be more easily accomplished. As another example, attorneys may automatically analyze large volumes of documents rather than needing to perform such tasks manually. As another example, text chunking may reduce token overhead and hence cost expended on large language model prompts. As yet another example, text chunking may reduce calls to a large language model, increasing response speed. As still another example, text chunking may increase and preserve context provided to a large language model by dividing text into chunks in semantically meaningful ways.

[0045] Figure 1 illustrates an overview method 100 for generating a structured document, performed in accordance with one or more embodiments. In some implementations, the method 100 may be performed at a text generation interface system such as the system 200 shown in Figure 2. For instance, the method 100 may be performed at the text generation interface system 210.

[0046] At 102, a request to determine a structured document is received from a client machine. According to various embodiments, the request may include a natural language input instruction characterizing content to include in the structured document. The request may include an exemplar document. The exemplar document may be a structured document that includes text portions divided into structural portions such as sections, subsections, paragraphs, and the like.

[0047] At 104, document structure information is determined. According to various embodiments, the document structure information may be determined based on an analysis of the exemplar document. The document structureinformation may identify document sections and may include characterizing content to include in the document sections.

[0048] According to various embodiments, the document structure information may include elements such as an outline of a document to be generated, information about the rhetorical content of various portions of a document to be generated, and other such information. Determining the document structure information may involve analyzing one or more exemplar documents. Additional details regarding the determination of a document structure are discussed with respect to the method 900 shown in Figure 9.

[0049] At 106, an initialized structured document is determined. According to various embodiments, determining the initialized structured document may involve operations such as determining content input information for the initialized structured document and then combining this content input information with the document structure information identified as discussed with respect to operation 102. Additional details regarding the determination of an initialized structured document are discussed with respect to the method 1000 shown in Figure 10.

[0050] At 108, a database system is queried to identify example document portions corresponding to the document sections. At 110, a structured document is determined. According to various embodiments, the structured document may include novel text portions corresponding with the document sections and generated by a large language model based on the example document portions and the natural language input instruction. For instance, the example document portions may be used to create novel text portions that are similar and that reflect the natural language input instruction.

[0051] According to various embodiments, determining the structured document may involve using the initialized structured document and potentially other input as a basis for generating the various text passages needed to complete the structured document. Additional details regarding the determination of a structured document are discussed with respect to the method 1100 shown in Figure 11.

[0052] Figure 2 illustrates a text generation system 200, configured in accordance with one or more embodiments. The text generation system 200 includes client machines 202 through 204 in communication with a text generation interface system 210, which in turn is in communication with a text generation modeling system 270. The text generation modeling system 270 includes a communication interface 272, a text generation API 274, and a text generation model 276. The text generation interface system 210 includes a communication interface 212, a database system 214, a testing module 220, and an orchestrator 230. The testing module 220 includes a query cache 222, a test repository 224, and a prompt testing utility 226. The orchestrator 230 includes skills 232 through 234, and prompt templates 236 through 238. The orchestrator also includes a chunker 240 and a scheduler 242. The orchestrator also includes API interfaces 250, which include a model interface 252, an external search interface 254, an internal search interface 256, and a chat interface 258.

[0053] According to various embodiments, a client machine may be any suitable computing device or system. For instance, a client machine may be a laptop computer, desktop computer, mobile computing device, or the like. Alternatively, or additionally, a client machine may be an interface through which multiple remote devices communicate with the text generation interface system 210.

[0054] According to various embodiments, a client machine may interact with the text generation interface system in any of various ways. For example, a client machine may access the text generation interface system via a text editor plugin, a dedicated application, a web browser, other types of interactions techniques, or combinations thereof.

[0055] According to various embodiments, the text generation modeling system 270 may be configured to receive, process, and respond to requests via the communication interface 272, which may be configured to facilitate communications via a network such as the internet.

[0056] In some embodiments, some or all of the communication with the text generation modeling system 270 may be conducted in accordance with the text generation API 274, which may provide remote access to the text generation model 276. The text generation API 274 may provide functionality such as defining standardized message formatting, enforcing maximum input and / or output size for the text generation model, and / or tracking usage of the text generation model.

[0057] According to various embodiments, the text generation model 276 may be a large language model. The text generation model 276 may be trained to predict successive words in a sentence. It may be capable of performing functions such as generating correspondence, summarizing text, and / or evaluating search results. The text generation model 276 may be pre-trained using many gigabytes of input text and may include billions or trillions of parameters.

[0058] In some embodiments, large language models impose a tradeoff. A large language model increases in power with the number of parameters and the amount of training data used to train the model. However, as the model parameters and input data increase in magnitude, the model's training cost, storage requirements, and required computing resources increase as well. Accordingly, the large language model may be implemented as a general- purpose model configured to generate arbitrary text. The text generation interface system 210 may serve as an interface between the client machines and the text generation modeling system 270 to support the use of the text generation modeling system 270 for performing complex, domain-specific tasks in fields such as law. That is, the text generation interface system 210 may be configured to perform one or more methods described herein.

[0059] According to various embodiments, the orchestrator 230 facilitates the implementation of one or more skills, such as the skills 232 through 234. A skill may act as a collection of interfaces, prompts, actions, data, and / or metadata that collectively provide a type of functionality to the client machine. For instance, a skill may involve receiving information from a client machine,transmitting one or more requests to the text generation modeling system 270, processing one or more response received form the text generation modeling system 270, performing one or more searches, and the like. Skills are also referred to herein as text generation flows.

[0060] In some embodiments, a skill may be associated with one or more prompts. For instance, the skill 234 is associated with the prompt templates 236 and 238. A prompt template may include information such as instructions that may be provided to the text generation modeling system 270. A prompt template may also include one or more fillable portions that may be filled based on information determined by the orchestrator 230. For instance, a prompt template may be filled based on information received from a client machine, information returned by a search query, or another information source.

[0061] In some implementations, the chunker 240 is configured to divide text into smaller portions. Dividing text into smaller portions may be needed at least in part to comply with one or more size limitations associated with the text. For instance, the text generation API 274 may impose a maximum size limit on prompts provided to the text generation model 276. The chunker may be used to subdivide text included in a request from a client, retrieved from a document, returned in a search result, or received from any other source.

[0062] According to various embodiments, the API interfaces 250 include one or more APIs for interacting with internal and / or external services. The model interface 252 may expose one or more functions for communicating with the text generation modeling system 270. For example, the model interface 252 may provide access to functions such as transmitting requests to the text generation modeling system 270, receiving responses from the text generation modeling system 270, and the like.

[0063] In some embodiments, the external search interface 254 may be used to search one or more external data sources such as information repositories that are generalizable to multiple parties. For instance, the external search interface254 may expose an interface for searching legal case law and secondary sources.

[0064] In some implementations, the internal search interface 256 may facilitate the searching of private documents. For instance, a client may upload or provide access to a set of private documents, which may then be indexed by the text generation interface system 210.

[0065] According to various embodiments, the chat interface 258 may facilitate text-based communication with the client machines. For instance, the chat interface 258 may support operations such as parsing chat messages, formulating responses to chat messages, identifying skills based on chat messages, and the like. In some configurations, the chat interface 258 may orchestrate text-based chat communication between a user at a client machine and the text generation model 276, for instance via web sockets.

[0066] In some embodiments, the query cache 222 may store queries such as testing queries sent to the text generation modeling system 270. Then, the query cache 222 may be instructed to return a predetermined result to a query that has already been sent to the text generation modeling system 270 rather than sending the same query again.

[0067] In some embodiments, the prompt testing utility 226 is configured to perform operations such as testing prompts created based on prompt templates against tests stored in the test repository 224.

[0068] In some embodiments, the communication interface 212 is configured to facilitate communications with the client machines and / or the text generation modeling system 270 via a network such as the internet. The scheduler 242 may be responsible for scheduling one or more tasks performed by the text generation interface system 210. For instance, the scheduler may schedule requests for transmission to the text generation modeling system 270.

[0069] In some embodiments, the database system 214 is configured to store information determined based on natural language. For example, the database system 214 may be configured to store one or more database tables thatinclude fields corresponding with information extracted from natural language documents. As another example, the database system 214 may be configured to store metadata information about documents based on information extracted from those documents. As yet another example, the database system 214 may be configured to store linkages between documents and document portions.

[0070] According to various embodiments, the database system 214 may be configured using any of a variety of suitable database technologies. For instance, the database system 214 may be configured as a relational database system, a non-relational database system, or any other type of database system capable of supporting the storage and querying of information described herein.

[0071] Figure 3 illustrates a document parsing method 300, performed in accordance with one or more embodiments. According to various embodiments, the method 300 may be performed on any suitable computing system. For instance, the method 300 may be performed on the text generation interface system 230 shown in Figure 2. The method 300 may be performed in order to convert a document into usable text while at the same time retaining metadata information about the text, such as the page, section, and / or document at which the text was located.

[0072] A request to parse a document is received at 302. In some embodiments, the request to parse a document may be generated when a document is identified for analysis. For example, as discussed herein, a document may be uploaded or identified by a client machine as part of communication with the text generation interface system 230. As another example, a document may be returned as part of a search result.

[0073] The document is converted to portable document format (PDF) or another suitable document format at 304. In some embodiments, the document need only be converted to PDF if the document is not already in the PDF format. Alternatively, PDF conversion may be performed even on PDFs to ensure that PDFs are properly formatted. PDF conversion may be performed,for instance, by a suitable Python library or the like. For instance, PDF conversion may be performed with the Hyland library.

[0074] Multipage pages are split into individual pages at 306. In some implementations, multipage pages may be split into individual pages via a machine learning model. The machine learning model may be trained to group together portions of text on a multipage page. For instance, a caption page in a legal decision may include text in a column on the left that encompasses the parties, text in a column on the right that includes the case number, a title that follows lower on the page, and line numbering on the left. In such a configuration, the machine learning model may be trained to treat separately the text in the different columns, and to separate the text from the line numbers. The document title may be identified as a first page, with the left column identified as the second page and the right column identified as the third page.

[0075] Optical character recognition is performed on individual pages or on the document as a whole at 308. In some implementations, optical character recognition may be performed locally via a library. Alternatively, optical character recognition may be performed by an external service. For instance, documents or pages may be sent to a service such as Google Vision. Performing optical character recognition on individual pages may provide for increased throughout via parallelization.

[0076] Individual pages are combined in order at 310. In some implementations, combining pages in order may be needed if optical character recognition were applied to individual pages rather than to the document as a whole.

[0077] Inappropriate text splits are identified and corrected at 312. In some embodiments, inappropriate text splits include instances where a paragraph, sentence, word, or other textual unit was split across different pages. Such instances may be identified by, for example, determining whether the first textual unit in a page represents a new paragraph, sentence, word, or other unit, or if instead it represents the continuation of a textual unit from theprevious page. When such a split is identified, the continuation of the textual unit may be excised from the page on which it is located and moved to the end of the previous page. Such an operation may be performed by, for instance, the Poppier library available in Python.

[0078] Segmented JSON text is determined at 314. In some embodiments, the segmented JSON text may include the text returned by the optical character recognition performed at operation 308. In addition, the segmented JSON text may include additional information, such as one or more identifiers for the page, section, and / or document on which the text resides. The output of the segmented JSON may be further processed, for instance via the text sharding method 500 shown in Figure 5 and / or the text chunking method 600 shown in Figure 6.

[0079] Figure 4 illustrates a text generation method 400, performed in accordance with one or more embodiments. According to various embodiments, the method 400 may be performed on any suitable computing system. For instance, the method 400 may be performed on the text generation interface system 230 shown in Figure 2. The method 400 may be performed in order to identify and implement a text generation flow based on input text.

[0080] A request from a client machine to generate a novel text portion is received at 402. In some embodiments, the request may include a query portion. The query portion may include natural language text, one or more instructions in a query language, user input in some other format, or some combination thereof. For instance, the query portion may include an instruction to "write an email", "summarize documents", or "research case law".

[0081] In some embodiments, the request may include an input text portion. For example, the request may link to, upload, or otherwise identify documents. As another example, the request may characterize the task to be completed. For instance, the request may discuss the content of the desired email or other correspondence. The particular types of input text included in the request may depend in significant part on the type of request. Accordingly, many variations are possible.

[0082] A text generation flow is determined at 404. In some embodiments, the text generation flow may be explicitly indicated as part of the request received from the client machine. For instance, the client machine may select a particular text generation flow from a list. Alternatively, the text generation flow may be determined at least in part by analyzing the request received from the client machine. For example, the request may be analyzed to search for keywords or other indications that a particular text generation flow is desired. As another example, all or a portion of the request may be provided to a machine learning model to predict the requested text generation flow. In some configurations, a predicted text generation flow may be provided to the client machine for confirmation before proceeding.

[0083] Input text is determined at 406. In some embodiments, the input text may be determined by applying one or more text processing, search, or other operations based on the request received from the client machine. For example, the input text may be determined at least in part by retrieving one or more documents identified in or included with the request received from the client machine. As another example, the input text may be determined at least in part by applying one or more natural language processing techniques such as cleaning or tokenizing raw text.

[0084] In some embodiments, determining input text may involve executing a search query. For example, a search of a database, set of documents, or other data source may be executed based at least in part on one or more search parameters determined based on a request received from a client machine. For instance, the request may identify one or more search terms and a set of documents to be searched using the one or more search terms.

[0085] In some embodiments, determining input text may involve processing responses received from a text generation modeling system. For instance, all or a portion of the results from an initial request to summarizing a set of text portions may then be used to create a new set of more compressed input text, which may then be provided to the text generation modeling system for further summarization or other processing.

[0086] One or more prompt templates are determined at 408 based on the input text and the text generation flow. As discussed with respect to Figure 2, different text generation flows may be associated with different prompt templates. Prompt templates may be selected from the prompt library based on the particular text generation flow.

[0087] At 410, one or more prompts based on the prompt templates are determined. In some embodiments, a prompt may be determined by supplementing and / or modifying a prompt template based on the input text. For instance, a portion of input text may be added to a prompt template at an appropriate location. As one example, a prompt template may include a set of instructions for causing a large language model to generate a correspondence document. The prompt template may be modified to determine a prompt by adding a portion of input text that characterizes the nature of the correspondence document to be generated. The added input text may identify information such as the correspondence recipient, source, topic, and discussion points.

[0088] The one or more prompts are transmitted to a text generation modeling system at 412. In some embodiments, the text generation modeling system may be implemented at a remote computing system. The text generation modeling system may be configured to implement a text generation model. The text generation modeling system may expose an application procedure interface via a communication interface accessible via a network such as the internet.

[0089] One or more text response messages are received from the remote computing system at 414. According to various embodiments, the one or more text response messages include one or more novel text portions generated by a text generation model implemented at the remote computing system. The novel text portions may be generated based at least in part on the prompt received at the text generation modeling system, including the instructions and the input text.

[0090] The one or more responses are parsed at 416 to produce a parsed response. In some embodiments, parsing the one or more responses may involve performing various types of processing operations. For example, in some systems a large language model may be configured to complete a prompt. Hence, a response message received from the large language model may include the instructions and / or the input text. Accordingly, the response message may be parsed to remove the instructions and / or the input text.

[0091] In some implementations, parsing the one or more responses may involve combining text from different responses. For instance, a document may be divided into a number of portions, each of which is summarized by the large language model. The resulting summaries may then be combined to produce an overall summary of the document.

[0092] A determination is made at 418 as to whether to provide a response to the client machine. In some embodiments, the determination made at 418 may depend on the process flow. For example, in some process flows, additional user input may be solicited by providing a response message determined based at least in part on one or more responses received from the text generation modeling system. As another example, in some process flows, a parsed response message may be used to produce an output message provided to the client machine.

[0093] If a response is to be provided to the client machine, then a client response message including a novel text passage is transmitted to the client machine at 420. In some embodiments, the client response message may be determined based in part on the text generation flow determined at 404 and in part based on the one or more text response messages received at 414 and parsed at 416.

[0094] A determination is made at 422 as to whether to generate an additional prompt. According to various embodiments, the determination as to whether to generate an additional prompt may be made based in part on the text generation flow determined at 404 and in part based on the one or more text response messages received at 414 and parsed at 416. As a simple example, atext generation flow may involve an initial set of prompts to summarize a set of portions, and then another round of interaction with the text generation modeling system to produce a more compressed summary.

[0095] According to various embodiments, the operations shown in Figure 4 may be performed in an order different from that shown. Alternatively, or additionally, one or more operations may be omitted, and / or other operations may be performed. For example, a text generation flow may involve one or more search queries executed outside the context of the text generation modeling system. As another example, a text generation flow may involve one or more processes for editing, cleaning, or otherwise altering text in a manner not discussed with respect to Figure 4. Various operations are possible.

[0096] Figure 5 illustrates a method 500 of sharding text, performed in accordance with one or more embodiments. According to various embodiments, the method 500 may be performed on any suitable computing system. For instance, the method 500 may be performed on the text generation interface system 230 shown in Figure 2. The method 500 may be performed in order to divide a body of text into potentially smaller units that fall beneath a designated size threshold, such as a size threshold imposed by an interface providing access to a large language model. For instance, a text generation modeling system implementing a large language model may specify a size threshold in terms of a number of tokens (e.g., words). As one example of such a threshold, a text generation modeling system may impose a limit of 8,193 tokens per query.

[0097] In particular embodiments, a size threshold may be adjusted based on considerations apart from a threshold imposed by an external text generation modeling system. For instance, a text generation interface system may formulate a prompt that includes input text as well as metadata such as one or more instructions for a large language model. In addition, the output of the large language model may be included in the threshold. If the external text generation modeling system imposes a threshold (e.g., 8,193 tokens), the text generation interface system 230 may need to impose a somewhat lowerthreshold when dividing input text in order to account for the metadata included in the prompt and / or the response provided by the large language model.

[0098] A request to divide text into one or more portions is received at 502. According to various embodiments, the request may be received as part of the implementation of one or more of the workflows shown herein. The request may identify a body of text. The body of text may include one or more documents, search queries, instruction sets, search results, and / or any other suitable text. In some configurations, a collection of text elements may be received. For instance, a search query and a set of documents returned by the search query may be included in the text.

[0099] In some implementations, text may be pre-divided into a number of different portions. Examples of divisions of text into portions may include, but are not limited to: lists of documents, documents, document sections, document pages, document paragraphs, and document sentences. Alternatively, or additionally, text may be divided into portions upon receipt at the text generation interface system 230. For instance, text may be divided into a set of portions via a text chunker, document parser, or other natural language processing tool.

[0100] A maximum text chunk size is identified at 504. In some embodiments, the maximum text chunk size may be identified based on one or more configuration parameters. In some configurations, the maximum text size may be imposed by the text generation interface system 230. Alternatively, or additionally, a size threshold may be imposed by an interface providing access to a large language model. As one example of a maximum text chunk size may be 100 kilobytes of text, 1 megabyte of text, 10 megabytes of text, or any other suitable chunk size.

[0101] A portion of the text is selected at 506. In some embodiments, as discussed herein, text may be pre-divided into text portion. Alternatively, or additionally, text may be divided into text portions as part of, or prior to, the operation of the method 500. As still another possibility, text may not bedivided into portions. In such a configuration, the initial portion of text that is selected may be the entirety of the text. Then, the identification of one or more updated text portions at 512 may result in the division of the text into one or more portions as part of the operation of the method 500.

[0102] A determination is made at 508 as to whether the length of the selected text portion exceeds the maximum text chunk size. In some embodiments, the determination may be made by computing a length associated with the selected text portion and then comparing it with the maximum text chunk size. The calculation of the length associated with the selected text portion may be performed in different ways, depending on how the maximum text chunk size is specified. For instance, the maximum text chunk size may be specified as a memory size (e.g., in kilobytes or megabytes), as a number of words, or in some other fashion.

[0103] If it is determined that the length of the selected text portion exceeds the maximum text chunk size, then at 510 one or more domain-specific text chunking constraints are identified. In some embodiments, domain-specific text chunking constraints may be identified based on one or more pre-determined configuration parameters. For example, one domain-specific text chunking constraint may discourage division of a question and answer in a deposition transcript or other question / answer context. As another example, a domainspecific text chunking constraint may discourage splitting of a contract clause. As yet another example, a domain-specific text chunking constraint may discourage splitting of a minority and majority opinion in a legal opinion.

[0104] An updated text portion that does not exceed the maximum text chunk size is identified at 512. In some embodiments, the updated text portion may be determined by applying a more granular division of the text portion into small portions. For example, a document may be divided into sections, pages, or paragraphs. As another example, a document page or section may be divided into paragraphs. As another example, a paragraph may be divided into sentences. As still another example, a sentence may be divided into words. In particular embodiments, the updated text portion may be the sequentially firstportion of the selected text portion that falls below the maximum text chunk size threshold identified at operation 504.

[0105] The text portion is assigned to a text chunk at 514. In some embodiments, the text may be associated with a sequence of text chunks. The text portions selected at 506 and identified at 512 may be assigned to these text chunks, for instance in a sequential order. That is, text portions near to one another in the text itself may be assigned to the same text chunk where possible to reduce the number of divisions between semantically similar elements of the text.

[0106] In particular embodiments, some attention may be paid to text divisions such as document, document section, paragraph, and / or sentence borders when assigning text portions to chunks. For instance, text portions belonging to the same document, document section, paragraph, and / or sentence may be grouped together when possible to ensure semantic continuity.

[0107] In particular embodiments, the method 500 may be performed in conjunction with the method 600 shown in Figure 6. In such a configuration, operation 514 may be omitted. Alternatively, the assignment of text portions into text chunks in operation 514 may be treated as provisional, subject to subsequent adjustment via the method 600 shown in Figure 6.

[0108] In some implementations, the identification of an updated text portion may result in the creation of two or more new text portions as a consequence of the division. In this case, the updated text portion may be assigned to a text chunk at 514, while the remainder portion or portions may be reserved for later selection at 506. Alternatively, or additionally, if two or more of the text portions resulting from the division at 512 each fall below the maximum text chunk size, then each of these may be assigned to a text chunk or chunks at operation 514.

[0109] A determination is made at 516 as to whether to select an additional portion of the text. According to various embodiments, additional portions of the text may continue to be selected as long as additional portions areavailable, or until some other triggering condition is met. For example, the system may impose a maximum amount of text for a particular interaction. As another example, the amount of text may exceed a designated threshold, such as a cost threshold.

[0110] Figure 6 illustrates a text chunk determination method 600, performed in accordance with one or more embodiments. According to various embodiments, the method 600 may be performed on any suitable computing system. For instance, the method 600 may be performed on the text generation interface system 230 shown in Figure 2. The method 600 may be performed in order to assign a set of text portions into text chunks.

[0111] In some embodiments, the method 600 may be used to compress text portions into text chunks of smaller size. For instance, the method 600 may receive as an input a set of text portions divided into text chunks of highly variable sizes, and then produce as an output a division of the same text portions into the same number of text chunks, but with the maximum text chunk size being lower due to more even distribution of text portions across text chunks.

[0112] A request is received at 602 to divide a set of text portions into one or more chunks. In some embodiments, the request may be automatically generated, for instance upon completion of the method 500 shown in Figure 5. The request may identify, for instance, a set of text portions to divide into text chunks.

[0113] An initial maximum text chunk size is identified at 604. In some embodiments, the initial maximum text chunk size may be identified in a manner similar to that for operation 504 shown in Figure 5.

[0114] A text portion is selected for processing at 606. In some embodiments, text portions may be selected sequentially. Sequential or nearly sequential ordering may ensure that semantically contiguous or similar text portions are often included within the same text chunk.

[0115] A determination is made at 608 as to whether the text portion fits into the latest text chunk. In some embodiments, text portions may be processed via the method 500 shown in Figure 5 to ensure that each text portion is smaller than the maximum chunk size. However, a text chunk may already include one or more text portions added to the text chunk in a previous iteration.

[0116] In the event that the text portion fits into the last text chunk size, the text portion is inserted into the last text chunk at 610. If instead the text portion is the first to be processed, or the text portion does not fit into the last text chunk size, then the text portion is inserted into a new text chunk at 612. The new chunk may be created with a maximum size in accordance with the maximum text chunk size, which may be the initial maximum text chunk upon the first iteration or the reduced maximum text chunk size upon subsequent iterations.

[0117] A determination is made at 614 as to whether to select an additional text portion for processing. In some embodiments, additional text portions may be selected until all text portions have been added to a respective text chunk.

[0118] A determination is made at 616 as to whether the number of text chunks has increased relative to the previous maximum text chunk size. If the number of text chunks increases, then a reduced maximum text chunk size is determined at 618, and the text portions are again assigned into chunks in operations 606 through 614.

[0119] According to various embodiments, for the first iteration, the number of chunks will not have increased because there was no previous assignment of text portions into text chunks. However, for the second and subsequent iterations, reducing the maximum text chunk size at 618 may cause the number of text chunks needed to hold the text portions to crease because the reduced maximum text chunk size may cause a text portion to no longer fit in a chunk and instead to spill over to the next chunk.

[0120] In some embodiments, the first increase of the number of text chunks may cause the termination of the method at operation 620. Alternatively, a different terminating criterion may be met. For instance, an increase in the number of text chunks may be compared with the reduction in text chunk size to produce a ratio, and additional reductions in text chunk size may continue to be imposed so long as the ratio falls below a designated threshold.

[0121] In some embodiments, the reduced text chunk size may be determined at 618 in any of various ways. For example, the text chunk size may be reduced by a designated amount (e.g., 10 words, 5 kilobytes, etc.) As another example, the text chunk size may be reduced by a designated percentage (e.g., 1%, 5%, etc.).

[0122] When it is determined that the number of text chunks has unacceptably increased, then at 620 the previous maximum text chunk size and assignment of text portions into chunks is returned. In this way, the number of text chunks may be limited while at the same time dividing text portions more equally into text chunks. The number of text chunks may be strictly capped at the input value, or may be allowed to increase to some degree if a sufficiently improved division of text portions into text chunks is achieved.

[0123] Figure 7 illustrates one example of a computing device 700, configured in accordance with one or more embodiments. According to various embodiments, a system 700 suitable for implementing embodiments described herein includes a processor 701, a memory module 703, a storage device 705, an interface 711, and a bus 715 (e.g., a PCI bus or other interconnection fabric.) System 700 may operate as variety of devices such as an application server, a database server, or any other device or service described herein. Although a particular configuration is described, a variety of alternative configurations are possible. The processor 701 may perform operations such as those described herein. Instructions for performing such operations may be embodied in the memory 703, on one or more non-transitory computer readable media, or on some other storage device. Various specially configured devices can also be used in place of or in addition to the processor 701. The interface 711 may beconfigured to send and receive data packets over a network. Examples of supported interfaces include, but are not limited to: Ethernet, fast Ethernet, Gigabit Ethernet, frame relay, cable, digital subscriber line (DSL), token ring, Asynchronous Transfer Mode (ATM), High-Speed Serial Interface (HSSI), and Fiber Distributed Data Interface (FDDI). These interfaces may include ports appropriate for communication with the appropriate media. They may also include an independent processor and / or volatile RAM. A computer system or computing device may include or communicate with a monitor, printer, or other suitable display for providing any of the results mentioned herein to a user.

[0124] Figure 8 illustrates a hallucination detection method 800, performed in accordance with one or more embodiments. The method 800 may be performed by the text generation interface system 210 shown in Figure 2.

[0125] In some embodiments, the method 800 may be performed in order to determine whether novel text generated by a text generation modeling system includes one or more hallucinations. Generative text systems sometimes generate text that includes inaccurate claims. For example, in the legal sphere, a request to summarize a set of judicial opinions about a point of law may result in a summary text that includes a citation to a non-existent opinion.

[0126] A request is received at 802 to identify one or more hallucinations in novel text generated by a text generation model. In some embodiments, the request may be received as part of one or more methods shown herein. For example, the method 800 may be performed to evaluate a response returned by the text generation modeling system. When employed in this way, the method 800 may be used to prompt the system to revise the response. Alternatively, or additionally, the method 800 may be used to prompt the system to generate a new response, to flag the error to a systems administrator, and / or to inform a response recipient of a potentially inaccurate response.

[0127] In some implementations, the request may be received as part of a training and / or testing procedure. For instance, one or more prompts may betested by the prompt testing utility 226 against one or more tests stored in the test repository 224. A test result may be evaluated using the method 800 to determine whether a prompt constructed from a prompt template being tested resulted in the generation of a hallucination, which may be treated as a test failure.

[0128] One or more factual assertions in the novel text are identified at 804. In some embodiments, the one or more factual assertions may be identified by transmitting a prompt to the text generation modeling system. For instance, the novel text may be included in a prompt requesting that the text generation modeling system identify factual claims in the novel text. The resulting completed prompt may be parsed to identify the one or more factual assertions.

[0129] A factual assertion is selected for analysis. Factual assertions identified at 804 may be analyzed in sequence, in parallel, or in any suitable order.

[0130] One or more search terms associated with the factual assertion are determined at 808. In some embodiments, one or more search terms may be returned by the text generation modeling system at 804. Alternatively, or additionally, one or more search terms may be determined based on a separate request sent to the text generation modeling system for the factual assertion being analyzed.

[0131] A search query to identify one or more search results based on the one or more search terms is executed at 810. According to various embodiments, one or more searches may be executed against any suitable database. Such databases may include, but are not limited to: public sources such as the internet, internal document databases, and external document databases.

[0132] The one or more search results are summarized at 812. In some embodiments, summarizing the one or more search results may involve, for instance, dividing documents into chunks and transmitting the one or more chunks to the text generation modeling system within summarization prompts.

[0133] At 814, the factual assertion is evaluated against the one or more search results. In some embodiments, evaluating the factual assertion may involve transmitting to the text generation modeling system a prompt that includes a request to evaluate the factual assertion, information characterizing the factual assertion, and a summary of the one or more search results determined as discussed at 812.

[0134] A determination is made at 816 as to whether the factual assertion is accurate. In some embodiments, the determination may be made by parsing the response returned by the text generation modeling system at 814. For instance, the text generation modeling system may complete the prompt by indicating whether the factual assertion is true, false, or uncertain based on the provided summary of search results.

[0135] If it is determined that the factual assertion is inaccurate, then at 818 the factual assertion is identified as a hallucination. In some embodiments, identifying the factual assertion as a hallucination may cause one or more consequences in an encompassing process flow. For example, in a testing phase, the detection of a hallucination may cause the test to fail. As another example, in a production phase, the detection of a hallucination may cause the system to initiate a flow to revise the novel text to remove the hallucination.

[0136] Figure 9 illustrates a method 900 for determining a document structure, performed in accordance with one or more embodiments. According to various embodiments, the method 900 may be performed on any suitable computing system. For instance, the method 900 may be performed on the text generation interface system 230 shown in Figure 2. The method 900 may be performed in order to determine a document structure for the purpose of creating a structured document.

[0137] A request to determine a document structure is received at 902. According to various embodiments, the request may be generated as part of a chat flow. Alternatively, or additionally, the request may be received via an application procedure interface. The request may identify an exemplar document to use as a basis for generating a structured document.

[0138] A document structure prompt template is identified at 904. According to various embodiments, the document structure prompt template may include one or more natural language instructions for determining structural information from an exemplar document. The document structure prompt template may also include one or more fillable portions in which to add some or all of the text from the exemplar document.

[0139] An example of a document structure prompt template is as follows. In the following example, the passage "{{ request }}" indicates a fillable portion where a chunk of the exemplar document may be added to create the document structure prompt.# InstructionsYou are a sophisticated legal writing Al. A lawyer has submitted a request for a document they would like you to draft. Your task is to create a detailed, paragraph-by-paragraph outline of what the document could look like based on the user's request. If the user provides a desired number of pages, try to tailor the outline to fit that number; otherwise, use your best judgment.# Output formatRespond with an XML document containing tags for each heading and paragraph. The top-level items in the outline should be headers (<hl> tags) corresponding to document sections. You can include additional levels as headings for subsections (<h2> tags and so on). The bottomlevel items should be the paragraph descriptions in tags. Here is an example of what the format might look like:<document><hl>Heading< / hl><h2>Subheading l< / h2>Paragraph lParagraph 2<h2>Subheading 2< / h2>Paragraph 3<hl>Heading 2< / hl><h2>Paragraph 4< / h2>Paragraph 4Paragraph 5< / document># TaskLet's do this!Request: {{ request }}< | endofprompt | >Outline:

[0140] An exemplar document for initializing the structured document is identified at 906. In some embodiments, the exemplar document may be identified in the request received at 902. Alternatively, or additionally, the system may store exemplar documents that may be selected for use depending on the nature of the request received at 902. The exemplar document may include text divided into sentences, paragraphs, sections, and the like. The text may also include headings, subheadings, and / or other such divisions.

[0141] One or more text chunks in the exemplar document are determined at 908. According to various embodiments, determining the one or more text chunks may involve applying preprocessing operations to the text of the exemplar document. Examples of such preprocessing operations are discussed throughout the application, for instance with respect to Figure 3, Figure 5, and Figure 6. For example, the method 600 shown in Figure 6 may be used to divide the text of the document into chunks.

[0142] A text chunk is selected for analysis at 910. According to various embodiments, text chunks may be selected in sequence, in parallel, or in any suitable order. In particular embodiments, text chunks may be processedsequentially so that the output from processing one or more earlier text chunks may be used to facilitate the processing of one or more subsequent text chunks, as discussed below.

[0143] At 912, a document structure prompt is determined based on the document structure prompt template and the selected text chunk. In some embodiments, determining the document structure prompt may involve filling one or more fillable portions of the document structure prompt template with the text in the selected text chunk.

[0144] In particular embodiments, prompt chaining may be used to maintain structural information across chunk boundaries. For instance, some or all of the first structural information determined based on one or more previously- analyzed text chunks may be added in a fillable portion of the document structure prompt template when determining a document structure prompt for a subsequent text chunk so that the second structural information determined for the subsequent text chunk logically supplements the first structural information.

[0145] Document structure information is determined at 914 based on the document structure prompt. In some embodiments, the document structure portion may be determined by providing the document structure prompt to a large language model, for instance as discussed with respect to Figure 2 and Figure 4. The large language model may then provide a response that includes a completed prompt. The completed prompt may be parsed to determine the document structure information.

[0146] According to various embodiments, the document structure information may include any or all of a variety of types of information. For example, the document structure information may include a document outline based on the exemplar document. As another example, the document structure information may include one or more headings, section titles, subsection titles, and / or other structural indicators. As yet another example, the document structure information may include sample text and / or one or more descriptions of a rhetorical purpose of a text passage. For instance, the document structureinformation may characterize the type of information to be included in a particular portion of a document.

[0147] A determination is made at 916 as to whether to select an additional text chunk for analysis. According to various embodiments, additional text chunks may continue to be selected until a terminating condition is met. For instance, additional text chunks may continue to be selected until all available text chunks have been analyzed.

[0148] Upon determining not to select an additional text chunk for analysis, at 918 a document structure summary is determined based on the document structure information. According to various embodiments, the document structure summary may include some or all of the document structure information determined for individual text chunks in the exemplar document. For instance, the document structure summary may include an outline of a document that includes headings, section titles, subsection titles, and / or other structural indicators, along with contextual information identifying the type of content included in various text passages.

[0149] Figure 10 illustrates a method 1000 for determining a document structure, performed in accordance with one or more embodiments. According to various embodiments, the method 1000 may be performed on any suitable computing system. For instance, the method 1000 may be performed on the text generation interface system 230 shown in Figure 2. The method 900 may be performed in order to initialize a document based on information about the structure and information about the content to be included within the structure.

[0150] A request to initialize a structured document is received at 1002. In some embodiments, the request may be generated in conjunction with and / or at the completion of the method 900 shown in Figure 9. As discussed with respect to operation 902, the request may be generated as part of a chat flow. Alternatively, or additionally, the request may be received via an application procedure interface. The request may identify document content input information used to generate a structured document.

[0151] A structured document initialization prompt template is determined at 1004. According to various embodiments, the structured document initialization prompt template may include natural language instructions to generate a structured document based on the other information included in the prompt.

[0152] In some embodiments, the structured document initialization prompt template may include one or more fillable portions. For example, a document structure summary fillable portion may indicate a location at which to insert document structure summary information determined as discussed with respect to Figure 9. As another example, a document content fillable portion may indicate a location at which to insert document content input information. Fillable portions may be indicated by, for instance, markup provided in compliance with a markup language such as XML or JSON.

[0153] An example of a structured document initialization prompt template is as follows. In the following example, the fillable portion "{{ prior_paras }}" indicates where to insert text portions previously generated for the initialized structured document. The fillable portion "{{ user_request }}" indicates where to insert the document content input information. The fillable portion "{{ description }}" indicates where to insert any section-specific instructions for generating a document section.# InstructionsYou are a sophisticated legal writing AL A lawyer has submitted a request for a document they would like you to draft. You created a detailed paragraph-by-paragraph outline of the document, and your task now is to draft a single paragraph. The paragraph should contain 3- 5 sentences and be drafted with the skill of an experienced lawyer. Follow any style or formatting guidelines provided in the lawyer's request, and include citations or direct quotes when appropriate.Pay attention to the outline provided and anticipate how the paragraph you are drafting should be structured to flow naturally from thepreceding paragraph and to the next paragraph. Never repeat paragraphs or try to close a document prematurely.# Real taskHere is the user's request: {{ user_request }}Here is what you've drafted so far:{{ prior_paras }}And here are the instructions for the specific paragraph you are drafting now:{{ description }}Omit any header included in the outline.{% if nextjtems is not none %}For reference only, here is an outline of some items you will draft next (don't actually draft these right now):{{ nextjtems }}{% endif %}Let's do this!< | endofprompt | >Paragraph:

[0154] A document structure summary is identified at 1006. According to various embodiments, the document structure summary may be determined as discussed with respect to the method 900 shown in Figure 9. Any or all of the information discussed in Figure 9 may be included in the document structure summary.

[0155] Document content input information is identified at 1008. According to various embodiments, the document content input information may be retrieved from one or more input documents, provided via a chat interface, received via an API call, or identified in any other suitable way.

[0156] In some embodiments, the document content input information may include natural language instructions, such as instructions provided by an end user. The particular type of content included in the document content input information thus may depend largely on the type of structured document being generated. For example, in the case of a lease agreement, the document content input information may include text such as: "Draft a lease agreement for a basement bedroom apartment. No dogs, rent on the 15th, utilities included."

[0157] A portion of the document structure included in the document structure summary is selected at 1010. According to various embodiments, document structure portions may be selected in any suitable order, in sequence or in parallel. In particular embodiments, portions may be select in sequence so that document text corresponding to previous structure portions may be included in a prompt used to generate subsequent structure portions.

[0158] A structured document initialization prompt is determined at 1012 based on the structured document initialization prompt template, the document structure summary, and the document content input information. In some embodiments, determining the structured document initialization prompt may involve filling the fillable portions of the structured document initialization prompt template with the information identified at 1006 and 1008.

[0159] An initialized structured document portion is determined at 1014 based on the structured document initialization prompt. In some embodiments, the initialized structured document may be determined by providing the structured document initialization prompt to a large language model, for instance as discussed with respect to Figure 2 and Figure 4. The large language model may then provide a response that includes a completed prompt. The completed prompt may be parsed to determine the initialized structured document portion. The initialized structured document portion may include, for instance, a portion of text generated by the large language model in accordance with the instructions and content included in the structured document initialization prompt template.

[0160] An initialized structured document is determined or updated at 1016 based on the initialized structured document portions. In some embodiments, the initialized structured document may be determined by combining the structured document portions determined at 1014. In some instances, the structured document portions may be combined with structural information determined as discussed with respect to the method 900 in Figure 9. For instance, paragraphs or sections of novel text created by the large language model may be placed in the appropriate structural regions of the document.

[0161] In some embodiments, an initialized structured document may be created by starting with one or more structural portions and then iteratively building the initialized structured document by adding text generated by a large language model. Prompt chaining may be used by providing all or a portion of the portion of the initialized structured document that has already been generated in subsequent prompts used to generate subsequent portions of the initialized structured document. In this way, subsequently generated text may reflect the style and / or content of previously generated text to provide for a seamless initialized structured document.

[0162] At 1018, a determination is made as to whether to select an additional portion of the document structure for analysis. According to various embodiments, additional portions may continue to be selected until all parts of the document structure have been selected.

[0163] Figure 11 illustrates a method 1100 for determining a structured document, performed in accordance with one or more embodiments. In some implementations, the method 1100 may be performed at a text generation interface system such as the system 200 shown in Figure 2. For instance, the method 1100 may be performed at the text generation interface system 210.

[0164] A request to update an initialized structured document is received at 1102. According to various embodiments, the request may be generated as discussed with respect to operation 108 shown in Figure 1.

[0165] Revision information for the initialized structured document is received at 1104. According to various embodiments, the initialized structured document is updated at 1106 based on the revision information. The revision information may be determined based on user input. It may be provided via a chat interface, graphical user interface, application procedure interface, or any other suitable interface.

[0166] In some embodiments, the revision information may include one or more natural language instructions to update a portion of the initialized structured document. For instance, the revision information could include an instruction such as: "Update the limitation of liability clause to disclaim liability cause by flood damage."

[0167] A portion of the initialized structured document is selected for analysis at 1108. According to various embodiments, initialized structured document portions may be selected in any suitable order, in sequence or in parallel. In particular embodiments, portions may be select in sequence so that document text corresponding to previous portions may be included in a prompt used to generate subsequent portions.

[0168] A database query to identify one or more clauses similar to the selected portion is determined at 1110. The database query is executed at 1112 to identify the one or more similar clauses. At 114, a replacement document text portion is determined based on the selected portion and the one or more similar clauses. Additional details regarding the determination and execution of database queries to identify reference text portions and use those portions to generate reference text are discussed with respect to Figure 12, Figure 13, Figure 14, and Figure 15.

[0169] At 1116, a determination is made as to whether to select an additional portion of the initialized structured document for analysis. According to various embodiments, additional portions may continue to be selected until all parts of the initialized structured document have been selected.

[0170] The structured document is determined at 1118 based on the initialized structured document and the replacement document text portions. In some embodiments, the structured document may be determined by replacing any portion of the initialized structured document with the corresponding replacement document text portion when such a portion has been determined.

[0171] A determination is made at 1120 as to whether to revise the structured document. In some embodiments, the determination may be based on user input. For instance, a user may review the structured document and decide whether to perform additional updating. The user input may be provided via a chat interface, graphical user interface, application procedure interface, or any other suitable interface.

[0172] Figure 12 illustrates a database generation and querying overview method 1200, performed in accordance with one or more embodiments. In some implementations, the method 1200 may be performed at a text generation interface system such as the system 200 shown in Figure 2. For instance, the method 1200 may be performed at the text generation interface system 210.

[0173] A database table characterizing a set of documents along one or more dimensions is determined at 1202. In some embodiments, the database table may be determined by generating a set of prompts provided to a text generation modeling system. A prompt may include a portion of text from one or more of the documents. The prompt may also include an instruction to identify data corresponding to one or more fields for the included portion of text. The text generation modeling system may complete the prompts and provide a response using structured text, such as JSON. The text generation interface system may then create or update a database system based on the structured text. Additional details regarding the creation or updating of a database system are discussed with respect to the method 1300 shown in Figure 13.

[0174] A subset of the documents is identified at 1204 based on a query of the database table using one or more query terms based on a query. In someembodiments, the query may include a natural language element. Alternatively, or additionally, the query may include one or more search terms specified in a different format, such as Boolean logic. The search terms may be generated based on an interaction with the text generation modeling system. For instance, all or a portion of the query may be used to create a query evaluation prompt which is completed by the text generation modeling system. The query evaluation prompt may instruct the model to identify one or more search terms based on the query and a set of fields associated with the database system. The resulting search terms may be used to search the database and identify the subset of the documents. Additional details regarding the identification of the subset of the documents are discussed with respect to the method 1400 shown in Figure 14.

[0175] An answer to the query is determined at 1206 by evaluating the text of the subset of the documents based on the query. In some embodiments, the answer to the query may be determined based at least in part on an interaction with a text generation modeling system. For instance, the identified documents may be used to create one or more prompts to the text generation modeling system. A prompt may include a portion of text from the identified documents and instructions based at least in part on the query. The text generation modeling system may complete the prompt, and the text generation interface system may determine an overall answer to the query based on the response or responses provided by the text generation modeling system. Additional details regarding the answering of a query based on an evaluation of the text of a subset of documents are discussed with respect to the method 1500 shown in Figure 15.

[0176] Figure 13 illustrates a database system updating method 1300, performed in accordance with one or more embodiments. The method 1300 may be performed at a text generation system such as the system 200 shown in Figure 2.

[0177] A request is received at 1302 to update a database system based on one or more natural language documents. In some embodiments, the request maybe received via a chat interface. Alternatively, the request may be received in some other way, such as via an API request. The request may be generated automatically or based on user input, and may be received from a client machine.

[0178] According to various embodiments, the natural language documents may be identified in various ways. For example, documents may be uploaded from a client machine, identified based on a search query, retrieved from a repository based on one or more document identifiers, or identified in any other suitable way.

[0179] Clauses included in the natural language documents are identified at 1304. In some embodiments, each clause may include some portion of a natural language document. For instance, a clause may include a single phase, a collection of phrases, a single sentence, a collection of sentences, a section, a page, one or more pages, or any other unit of analysis.

[0180] According to various embodiments, clauses may be identified based on one or more natural language processing techniques. For instance, a document may be tokenized into words. Words may then be grouped into phrases and / or sentences based on indicators such as punctuation and semantic content. Sentences may be grouped into sections such as paragraphs or other units. Clauses may then be identified based on the structure.

[0181] In particular embodiments, the identification of clauses may involve domain-specific logic. For instance, the identification of clauses in a general- purpose non-fiction text may be different from the identification of clauses in a legal contract. Accordingly, the text generation interface system may store domain-specific instructions for identifying clauses in one or more contexts.

[0182] One or more data fields associated with the one or more natural language documents are identified at 1306. In some embodiments, one or more data fields may be identified based on a query. Additional details regarding query parsing are discussed with respect to query parsing are discussed with respect to the method 1400 shown in Figure 14.

[0183] In some implementations, one or more data fields may be identified based on the structure of a table in a database system or other such configuration parameters. For instance, if metadata for a set of documents is intended to be combined with metadata for other documents already reflected in one or more database tables, then fields associated with those database tables may be identified so as to identify values corresponding to the existing table structure.

[0184] One or more clauses are selected for analysis at 1308. A text chunk is determined at 1304 based on the natural language documents. In some embodiments, the one or more may be determined by dividing the clauses identified at 1304 into chunks based on a chunk size. Examples of techniques for determining text chunks are discussed with respect to the method 600 shown in Figure 6. In some contexts, a text chunk may be limited to text from a single document. Alternatively, a single text chunk may include text from more than one document.

[0185] An input metadata extraction prompt is determined at 1310 based on the text chunk and a clause splitting prompt template. In some embodiments, the input metadata extraction prompt may be determined by supplementing and / or modifying the input metadata extraction prompt based on the one or more clauses and the one or more data fields. For instance, the one or more clauses and a description of the one or more data fields may be added to a prompt template at an appropriate location. As one example, a prompt template may include a set of instructions for causing a large language model to identify values for the one or more data fields based on the one or more clauses. The prompt template may also include one or more additional instructions, such as an instruction to format the text generated by the text generation model as structured text. For instance, the structured text may be implemented as a JSON list. An example of a prompt template for identifying information and clauses relevant for answering a query is as follows:Purpose: Find information in a contract that is highly relevant to a question.The following Clauses are from a {{ context }}For each of the Contract Clauses below, decide whether the Contract Clause contains language that is necessary or highly relevant to answer the question. If it does, provide the IDs of the clauses that contain the information necessary or highly relevant to answer the question.A few guidelines regarding what constitutes relevance:* It will often be the case that nothing in the Contract Clauses answers the question. This is not a problem. When this happens, simply respond by saying "none" (all lower case)* Sometimes, multiple clauses will contain information highly relevant or necessary to answer the question. If that happens, please list all such relevant clauses in your answer.* If there is / are Clause(s) that only partially answer the question, include them in your answer.* If the answer to a question can be inferred from a Clause, include that Clause in your answer list, even if the Clause does not directly answer the question.* If a Clause contains information that could potentially help answer the question if it were combined with other information not seen here, include this Clause in your answer list.* If a question is asking whether something is present or missing, a Clause closely related to the subject of the question that is missing the element is still helpful in answering the question.* If a header Clause is relevant, then list all the Clauses under that header as relevant as well.Question: {{ query.text }}Contract Clauses XML:< co n t ra ct_c I a u se s>{% for contract_section in paragraphs %}<section><id>CC{{ loop.indexO }}< / id><text>{{ contract_section.text }}< / text>< / section>{% endfor %}< / contract_clauses>Give your answer in the following format:<question_comprehension>[restate what the Question is trying to ask in clear terms to show that you understood the question]< / question_comprehension><what_to_look_for>[briefly summarize what sorts of clauses you should be looking for to answer the question, but never refer to a specific clause ID here. It is very important that you not include the clause IDs in this section]< / what_to_look_for><clauses>[if there are Clauses containing information highly relevant or necessary to answer the question, provide your answer as a pipe- character-separated list of the clause ID's here, for example: CC1 | CC2 | CC5 | CC9< / clauses>Then give a very brief explanation of your answer.< | endofprompt | >{% if question_comprehension %}<question_comprehension>{{question_comprehension}}< / question_comprehension><what_to_look_for>{{ what_to_look_for }}< / what_to_look_for><clauses>{% else %}<question_comprehension>{%- endif %}

[0186] A completed metadata extraction prompt is determined at 1312 based on a request sent to a remote text generation modeling system. In some embodiments, the completed metadata extraction prompt may be determined by sending the input metadata extraction prompt to the remote text generationmodeling system via an API request. A text generation model implemented at the remote text generation modeling system may then complete the prompt, after which it may be sent back to the text generation interface system.

[0187] Clause-level field values corresponding to the identified data fields are determined at 1314. In some embodiments, the clause-level field values may be determined by parsing the completed metadata extraction prompt. For instance, structured text such as JSON included in the completed metadata extraction prompt may be parsed to identify data values corresponding with data fields for clauses included in the metadata extraction prompt.

[0188] A determination is made at 1316 as to whether to determine an additional one or more clauses for analysis. In some implementations, additional clauses may continue to be selected for analysis until all of the natural language documents have been processed.

[0189] Document-level field values are determined at 1318 based on the clause-level field values. In some embodiments, the document-level field values may be determined by first identifying and then aggregating clause-level field values for a given document. For example, in the legal context, a data field may indicate whether a contract includes an indemnification clause. One or more metadata extraction prompts may be used to identify, for each clause in the document, whether that clause is an indemnification clause. Although most clauses in the document will not be an indemnification clause, the data field value for the document as a whole will be true if even one of the clauses for the document is identified as an indemnification clause.

[0190] As another example, in the legal context, a data field may indicate whether a contract involves an exchange valued at more than a threshold value. In this context, one or more metadata extraction prompts may be used to identify the exchange value, if any, associated with each clause in the document. The data field value for the document may then be determined by identifying the maximum exchange value determined for any of the clauses.

[0191] In particular embodiments, determining the document-level field values may involve domain-specific logic. This domain-specific logic may be reflected in one or more configuration parameters and / or subroutines included in the text generation system.

[0192] A database system is updated at 1320 to include one or more entries identifying the field values. In some embodiments, the database system may maintain one or more tables at the document level, as well as one or more tables at the clause level. The database system may link documents with clauses. The text of the clauses may be included within the database system itself and / or may be identified by location within the text of the associated document. The one or more tables may include the field values to facilitate searching the documents and / or clauses on the basis of the field values. Additional details regarding the searching of natural language documents based on data field values are discussed with respect to the method 1500 shown in Figure 15.

[0193] According to various embodiments, the operations discussed in Figure 13 may be performed in various orders, and in sequence or in parallel. For instance, a set of prompts may be created in one phase and then sent to the text generation modeling system in a subsequent phase.

[0194] Figure 14 illustrates a database system query and filter determination method 1400, performed in accordance with one or more embodiments. The method 1400 may be performed at a text generation system such as the system 200 shown in Figure 2. For instance, the method 1400 may be performed at the text generation interface system 210.

[0195] A request to query a database system is received at 1402. In some embodiments, the request may be received as part of a chat flow. Alternatively, the request may be received via an API call. In either case, the request may be received from a client machine in communication with the text generation interface system 210 via the internet. The request may, for instance, include a natural language query to identify, count, summarize, or other interact with documents that meet one or more criteria. For instance, the request mayinclude a natural language query to determine the proportion of contracts for the purchase of goods or services valued over $100,000 signed by parties within California in the last 10 years where the contract includes a mandatory arbitration clause.

[0196] A query and filter comprehension prompt is determined at 1404 based on the request. In some embodiments, the query and filter comprehension prompt may be determined by combining some or all of the query received with the request at 1402 with a query and filter comprehension prompt template. The query and filter comprehension prompt template may include one or more fillable elements that may be filled with text, such as "{{ query. text }}". The query and filter comprehension prompt template may also include an instruction to the text generation modeling system to restate the query and filter request included in the query and filter comprehension prompt template. The prompt template may also include one or more additional instructions, such as an instruction to format the text generated by the text generation model as structured text. For instance, the structured text may be implemented as a JSON list. An example of a template for generating a summary of a query is as follows:Purpose: Find information in a contract that is highly relevant to a question.Question: {{ query.text }}Give your answer in the following format:<question_comprehension>[restate what the Question is trying to ask in clear terms to show that you understood the question]< / question_comprehension>Then give a very brief explanation of your answer.< | endofprompt | ><question_comprehension>

[0197] A query and filter description is determined at 1406 based on the prompt. In some embodiments, the query and filter description may be determined by transmitting the query and filter comprehension prompt to a remote text generation modeling system, for instance via an API call. The remote text generation modeling system may then complete the prompt and return it to the text generation interface system. The text generation interface system may extract from the completed prompt a description of the query and filter request included in the prompt.

[0198] The query and filter description is transmitted for feedback at 1408. In some embodiments, the query and filter description may be transmitted to a client machine, such as the client responsible for generating the request received at 1402. For instance, the query and filter description may be transmitted for feedback via a chat session or response to an API call.

[0199] A determination is made at 1410 as to whether to receive an updated request to query the database system. In some embodiments, the determination may be made based at least in part on user input. For instance, a user may review the description and provide feedback as to whether the description produced by the text generation modeling system accurately characterizes the user's initial intent when formulating the query. The user may then provide feedback either accepting or updating the query requested.

[0200] If it is determined to receive an updated request to query the database system, then an updated request to query the database system is received at 1402. The updated request may then be re-evaluated. In this way, the text generation system may ensure that the text generation modeling system more accurately interprets the user's intent when formulating the query.

[0201] If instead it is determined not to receive an updated request to query the database system, then a query generation prompt is determined at 1412. In some embodiments, the query generation prompt may be determined by combining some or all of the query received with the request at 1402 and / or the query and filter description determined at 1406 with a query generation prompt template. The query generation prompt template may include one ormore fillable elements that may be filled with text, such as "{{ query text }}". The query generation prompt template may also include an instruction to the text generation modeling system to determine one or more query and / or filter parameters based on the query generation prompt. The prompt template may also include one or more additional instructions, such as an instruction to format the text generated by the text generation model as structured text. For instance, the structured text may be implemented as a JSON list. In particular embodiments, a query generation prompt may be used to generate multiple queries, each of which may be executed against a suitable database. An example of a prompt template for generating a query is as follows:We are generating queries for a search engine given a user's original query. The search engine output must follow a specific output format which we will explain to you soon. The search engine, called AllSearch, can search with two different modes, "parallel" (aka Parallel Search) and "kw" (aka Keyword Searches). Parallel Searches are vector-based searches. This means that input queries must resemble full sentences. The full sentences are encoded as dense vectors and used to retrieve the K nearest neighbors in the index's vector space.For example, if a user wanted to know if refusal to wear a mask at work constituted employment discrimination, a good query for parallel search would be:"McVader's termination of Skywalker for refusal to wear a mask cannot be construed as discriminatory."If the user provided a name, then it's good to use the name, but if no name is given, it's ok to make one up (in this case "McVader").Keyword searches are bag-of-words based retrieval searches that use ranking methods such as BM-25 or TF-IDF. In these searches, it's important for queries to make exact word or phrase matches in order to get relevant results.A good query would use single words and / or short phrases with words that we would guess are likely to appear in the search corpus. For example, if the user who wanted to know if refusal to wear a mask at work constituted employment discrimination was making a keyword search, good queries would include:- apparel workplace discrimination- employee discrimination- mask mandates workplace- religious exemption employment law and so forth.Finally, Keyword Searches can use terms and connectors. The purpose of using terms and connectors is less so to answer a question, but to help someone search over a corpus of documents that may be responsive to the query. Turn the user's question into three terms-and- connectors searches, including using proximity searching, "OR" and "AND" parameters, root expansion (using I), and parentheses using the following guidelines:The terms and connectors search terms should cover all the substantive aspects of the question.Examples of good terms-and-connectors searches: '(reject! or refus!) / s settl! / s fail! / s mitigat!', '((sexual / 2 (assault! OR harass! OR misconduct)) / p "first amendment") AND (school OR university OR college)'Given the user's original query: "{{ query_text }}",{% if query_comprehension_text %} And given this supplemental information about the query that the user approved: {{ query_comprehension_text }},{% endif %}Generate several XML documents (bounded by the '<q>' tag), with each document representing a search query.The documents must conform to the following schema:<q><t>[string - the query text that you generate]< / t><m>[the mode, must be exactly one of "kw" or "parallel"]< / m>< / q>You must provide at least two of each: parallel search, keyword search without terms and connectors, and keyword search with terms and connectors.Provide three more queries of any mode.< | endofprompt | >Here are the XML documents and nothing else:

[0202] The query generation prompt is transmitted to a text generation modeling system at 1414. Then, a query generation prompt response message is received at 1416. According to various embodiments, the query generation prompt may be transmitted to the text generation modeling system via an API request. The text generation modeling system may then complete the prompt via a text generation model implemented at the text generation modeling system, and send a response that includes the completed prompt.

[0203] A database query is determined at 1418 based on the query generation prompt response message. In some embodiments, determining the database query may involve extracting one or more database query parameters from the query generation response message. For instance, the query generation response message may include a JSON portion that encodes a list of database query parameters. The database query parameters may then be combined with a query template to generate the database query. Alternatively, the querygeneration prompt response message may include a fully formed database query.

[0204] According to various embodiments, the particular operations involved in determining the database query may depend in part on the type of database system employed. For example, the query structure may depend on whether the database system is a relational database system or a nonrelational database system. As another example, the query structure may depend on the structure of tables within the database system. Additional details regarding the querying of the database system are discussed with respect to the method 1500 shown in Figure 15.

[0205] At 1420, a text filter is determined based on the query generation prompt response message. In some embodiments, the text filter may include any suitable information for providing to a text generation modeling system for filtering results returned by the database query determined at 1418. For example, the text filter may include one or more qualitative restrictions capable of being evaluated by the text generation modeling system. As another example, the text filter may include one or more restrictions that are not reflected by information stored in the database system. Additional details regarding the filtering of results returned by the database system are discussed with respect to the method 1500 shown in Figure 15.

[0206] Figure 15 illustrates a database system query and filter execution method 1500, performed in accordance with one or more embodiments. The method 1500 may be performed at a text generation system such as the system 200 shown in Figure 2. For instance, the method 1400 may be performed at the text generation interface system 210.

[0207] A request to execute a database system is received at 1402. In some embodiments, the request may be generated automatically, for instance after a database query is generated as discussed with respect to operation 1418 shown in Figure 14. The request may be generated as part of a chat flow or based on an API request. In either case, the request may be generated based oninteraction with a client machine in communication with the text generation interface system 210 via the internet.

[0208] A database system query is identified at 1504. According to various embodiments, the database system query may be determined as discussed with respect to operation 1418 shown in Figure 14.

[0209] One or more query response clauses and associated documents are determined at 1506. In some embodiments, the one or more query response clauses and associated documents may be determined by executing the query identified at 1504 against the database system. As discussed herein, for instance with respect to Figure 13, the database system may store metadata characterizing documents portions of text from documents. Executing the query may result in the database system returning one or more documents, document portions, and / or identifiers that identify documents and / or document portions.

[0210] One or more relevance prompts are determined at 1508 based on the one or more query response clauses. In some embodiments, a relevance prompt may be determined by combining some or all of the query results received at 1506 with a relevance prompt template. The relevance prompt template may include one or more fillable elements that may be filled with text. One or more of the fillable elements may be filled with some or all of the query results received at 1506. Additionally, one or more of the fillable elements may be filled with relevance information. The relevance information may include some or all of the text filter determined at 1420. Alternatively, or additionally, the relevance information may include some or all of the query received at 1402, the query and filter description determined at 1406, and / or the database query determined at 1418.

[0211] In some embodiments, the relevance prompt template may also include an instruction to the text generation modeling system to evaluate and / or rank the included search result or results for relevance against the relevance information. The prompt template may also include one or more additional instructions, such as an instruction to format the text generated by the textgeneration model as structured text. For instance, the structured text may be implemented as a JSON list. An example of a relevance prompt template is as follows:Evaluate whether these documents are relevant to this research request or query: "{{text}}"$$DOCUMENTS$${{documents}}$$ / DOCUMENTS$$* Only respond with relevant documents. In order to be deemed relevant, a document must directly answer the request or query. A document should also be considered relevant if it reaches a conclusion in opposition to the research request.* If there are no relevant documents, do not include any in your response.* Assign a relevance score to each document, judging its relevance to the research request or query: "{{text}}". The score should correlate to these values:5 - the document is directly on-point (i.e., it precisely responds to every aspect of the query or request, even if it is in opposition to the request, and not a similar but different issue; it fully and conclusively settles the question raised in the request either in favor or against the intention of the request, if any)4 - the document may provide a useful analogy to help answer the request, but is not directly responsive3 - the document is roughly in the same topical area as the request, but otherwise not responsive2 - the document might have something to do with the request, but there is no indication that it does in the text provided1 - the document is in no way responsive to the requestReturn a JSON array of objects, each object representing a relevant case, ordered with the most relevant case first. Each object in the array will have the keys:* \'result_id\' - string, the result ID* \'reason_relevant\' - string, a description of how the document addresses the research request or query: "{user_request}". In drafting this response, only draw from the excerpted language of the document; do not include extraneous information.* \'relevance_score\' - number, between 1-5, of how relevant the document is to the research request or query: "{user_request}"* \'quotes\' - array of strings. For each document, quote the language from the document that addresses the request. In finding these quotes, only draw from the excerpted language; do not include extraneous information. Do not put additional quotation marks around each quote beyond the quotation marks required to make valid JSON.Only valid JSON. Quotation marks within strings must be escaped with a backslash (\'\\V)- Examples for reason_relevant: \'"The concept of W'equitable tollingW" applies in this case."\', \'"The case overturns a lower court decision that found a state abortion restriction unconstitutional based on Roe v. Wade and Casey, and argues that the viability rule from those cases is not the \\"central holding.\\" This case calls into question the continued validity of Roe v. Wade."\'If there are no relevant documents, respond with an empty array. < | endofprompt | >Here's the JSON:

[0212] In some implementations, more than one relevance prompt may be determined. For instance, if many query response clauses are determined at 1506, then these query responses may be divided into groups for the purpose of relevancy analysis. The size of the groups may be determined based on a chunk threshold. Additional details regarding the division of text into chunks are discussed with respect to the method 600 shown in Figure 6.

[0213] A subset of the query response clauses that meet a relevancy threshold based on communication with a text generation modeling system are identified at 1510. In some embodiments, the subset of the query response clauses may be identified by transmitting the prompt or prompts determined at 1508 to a remote text generation modeling system. The remote text generation modeling system may then respond with one or more completed prompts. The text generation interface system may then extract relevancy information from the completed prompts.

[0214] According to various embodiments, the relevance threshold may be determined in any of various ways. For example, all results that exceed a designated relevance threshold (e.g., 3 out of a scale of 1-5 as shown in the example prompt template included above) may be identified. As another example, the most relevant results that are able to fit in a designated number (e.g., one or two) chunks may be identified.

[0215] A query and filter synthesis prompt is determined at 1512 based on the subset of the query response clauses. In some embodiments, the query and filter synthesis prompt may be determined by combining a query and filter synthesis prompt template with information about the query and with query response clauses deemed suitable relevant at operation 1510. The query information may include some or all of the query received at 1402, the query and filter description determined at 1406, the database query determined at 1418, and / or the text filter determined at 1420. An example of a query and filter synthesis prompt template in the legal context is as follows:You are helping a lawyer research the prevailing market consensus on a given type of contract clause.Using the following list of contract clauses, analyze the range of different terms for this type of clause in the context of this request from the lawyer: "{{text}}"$$CONTRACT_CLAUSE_LIST$${{documents}}$$ / CONTRACT_CLAUSE_LIST$$Based on these contract clauses, and in the context of the lawyer's request, prepare:1. Range of Terms: An extensive analysis of the range of different provisions included in these clauses, following these instructions:* List the dimensions on which the clauses differ, and explain the range of provisions along each of the dimensions.* Focus on the range of favorability to one side or another* Only draw from the language in this list of clauses; do not include extraneous information.2. Average Terms: State what the average terms over the above list of contracts is over the dimensions you analyzed for question 1 above.3. Suggested Language: Draft a contract clause that is approximately average in terms when compared to the above list of clauses.4. List the clauses that were most relevant to your analysis, following this guidance:* Do not include in this list any clauses that are not relevant to the request.* If none of the clauses are relevant, return an empty array for results.Respond with nothing but a JSON object, with the following keys: \'range_of_terms\': your analysis of the range of provisions in the clause list, in the context of the lawyer's request.\'average_terms\': your analysis of the average provisions over the clauses in the list, in the context of the lawyer's request.\'suggested_language\': your draft clause with approximately average terms.\'ids\': (array of strings), in order of relevance, the document IDs of the documents that are most relevant to the request.Only valid JSON; check to make sure it parses, and that quotes within quotes are escaped or turned to single quotes. For the \'answer\' key, this could look like: "This is an answer with \\"proper quotingW"" < | endofprompt | >Here's the JSON:

[0216] A query and filter response message is determined at 1514 based on communication with the text generation modeling system. In some embodiments, determining the query and filter response message may involve transmitting the prompt determined at 1512 to the remote text generation modeling system. The remote text generation modeling system may then respond with one or more completed prompts. The text generation interface system may then extract information for providing the query and filter response message. The extracted information may be used as-is or may be edited, supplemented, or otherwise altered to create the query and filter response message.

[0217] A query and filter response message is transmitted at 1516. In some embodiments, the query and filter response message may be provided to a client machine. The message may be sent in response to an API request, transmitted via a chat session, or provided in some other way.

[0218] Any of the disclosed implementations may be embodied in various types of hardware, software, firmware, computer readable media, and combinations thereof. For example, some techniques disclosed herein may be implemented, at least in part, by computer-readable media that include program instructions, state information, etc., for configuring a computing system to perform various services and operations described herein. Examples of program instructions include both machine code, such as produced by a compiler, and higher-level code that may be executed via an interpreter. Instructions may be embodied in any suitable language such as, for example, Java, Python, C++, C, HTML, any other markup language, JavaScript, ActiveX, VBScript, or Perl. Examples of computer-readable media include, but are not limited to: magnetic media such as hard disks and magnetic tape; optical media such as flash memory, compact disk (CD) or digital versatile disk (DVD); magneto-optical media; and other hardware devices such as read-only memory ("ROM") devices and randomaccess memory ("RAM") devices. A computer-readable medium may be any combination of such storage devices.

[0219] In the foregoing specification, various techniques and mechanisms may have been described in singular form for clarity. However, it should be noted that some embodiments include multiple iterations of a technique or multiple instantiations of a mechanism unless otherwise noted. For example, a system uses a processor in a variety of contexts but can use multiple processors while remaining within the scope of the present disclosure unless otherwise noted. Similarly, various techniques and mechanisms may have been described as including a connection between two entities. However, a connection does not necessarily mean a direct, unimpeded connection, as a variety of other entities (e.g., bridges, controllers, gateways, etc.) may reside between the two entities.

[0220] In the foregoing specification, reference was made in detail to specific embodiments including one or more of the best modes contemplated by the inventors. While various implementations have been described herein, it should be understood that they have been presented by way of example only, and not limitation. For example, some techniques and mechanisms are described herein in the context of large language models. However, the techniques disclosed herein apply to a wide variety of language models. Particular embodiments may be implemented without some or all of the specific details described herein. In other instances, well known process operations have not been described in detail in order to avoid unnecessarily obscuring the disclosed techniques. Accordingly, the breadth and scope of the present application should not be limited by any of the implementations described herein, but should be defined only in accordance with the claims and their equivalents.

Claims

CLAIMS1. A method comprising: receiving from a client machine a request to determine a structured document based on an exemplar document and a natural language input instruction characterizing content to include in the structured document; determining via a processor document structure information based on an analysis of the exemplar document, the document structure information identifying a plurality of document sections, the document structure information characterizing content to include in the plurality of document sections; querying a database system to identify a plurality of example document portions corresponding to the plurality of document sections; determining a structured document via a processor, the structured document including a plurality of novel text portions corresponding with the plurality of document sections, the plurality of novel text portions being generated by a large language model based on the plurality of example document portions and the natural language input instruction; and transmitting the structured document to the client machine.

2. The method recited in claim 1, the method further comprising: determining an initialized structured document based on the document structure information, the initialized structured document including a plurality of document structure portions corresponding to the plurality of document sections, the initialized structured document further including a plurality of contextual information portions characterizing content to be included in the document structure portions.

3. The method recited in claim 2, wherein the plurality of document structure portions and the plurality of contextual information portions are determined by the large language model based on the exemplar document and a documentinitialization natural language instruction to determine the initialized structured document.

4. The method recited in claim 3, the method further comprising: determining a document initialization prompt based on a document initialization prompt template, the document initialization prompt and the document initialization prompt template both including the document initialization natural language instruction, wherein determining the document initialization prompt template involves filling a fillable portion included in the document initialization prompt template with a portion of the exemplar document.

5. The method recited in any of claim 1 through claim 4, the method further comprising: determining a designated portion of the exemplar document corresponding with a designated document section; and extracting one or more clauses from the designated portion of the exemplar document, wherein the database system is queried based on the one or more clauses.

6. The method recited in claim 5, wherein the one or more clauses are extracted by the large language model based on a metadata extraction prompt that includes the designated portion of the exemplar document and a metadata extraction natural language instruction to extract the one or more clauses.

7. The method recited in any of claim 1 through claim 6, wherein querying the database system involves executing a designated database query determined by a large language model based on a query prompt that includes a designated portion of the exemplar document corresponding with a designated document section and a query identification natural language instruction to identify the designated database query, wherein the designated database query returns aplurality of designated example document portions identified by the database system as being similar to the designated portion of the exemplar document.

8. The method recited in any of claim 1 through claim 7, wherein determining the document structure information comprises: determining a document structure prompt based on a document structure prompt template, the document structure prompt and the document structure prompt template both including the document structure natural language instruction, wherein determining the document structure prompt template involves filling a fillable portion included in the document structure prompt template with a portion of the exemplar document.

9. The method recited in any of claim 1 through claim 8, wherein determining the document structure information further comprises: providing the document structure prompt to the large language model, and extracting the document structure information from a completed document structure prompt determined by the large language model based on the document structure prompt.

10. The method recited in any of claim 1 through claim 9, wherein determining the structured document involves: determining a first novel text portion based on the natural language input instruction; determining a second novel text portion of the plurality of text portions based on the natural language input instruction and the first novel text portion; and determining a third novel text portion of the plurality of text portions based on the natural language input instruction and the first and second novel text portions.

11. The method recited in claim 10, wherein determining the second novel text portion comprises: determining a structured document generation prompt based on a structured document generation prompt template, the structured document generation prompt and the structured document generation prompt template both including a structured document generation natural language instruction to generate novel text based on input, wherein determining the structured document generation prompt involves filling one or more fillable portions of the structured document generation prompt template with the structured document generation natural language instruction and the first novel text portion.

12. The method recited in claim 10, wherein determining the second novel text portion further comprises: parsing a first completed structured document generation prompt determined by the large language model, extracting the first novel text portion from the first completed structured document generation prompt, providing the structured document generation prompt to the large language model, and extracting the second novel text portion from a second completed structured document generation prompt determined by the large language model in response to the structured document generation prompt.

13. A system comprising: a communication interface operable to receive from a client machine a request to determine a structured document based on an exemplar document and a natural language input instruction characterizing content to include in the structured document; a processor operable to determine document structure information based on an analysis of the exemplar document, the document structure information identifying a plurality of document sections, the document structureinformation characterizing content to include in the plurality of document sections; a database system interface operable to query a database system to identify a plurality of example document portions corresponding to the plurality of document sections, wherein the processor is further operable to determine a structured document including a plurality of novel text portions corresponding with the plurality of document sections, the plurality of novel text portions being generated by a large language model based on the plurality of example document portions and the natural language input instruction, and wherein the communication interface is operable to transmit the structured document to the client machine.

14. The system recited in claim 13, wherein the processor is further operable to: determine an initialized structured document based on the document structure information, the initialized structured document including a plurality of document structure portions corresponding to the plurality of document sections, the initialized structured document further including a plurality of contextual information portions characterizing content to be included in the document structure portions, wherein the plurality of document structure portions and the plurality of contextual information portions are determined by the large language model based on the exemplar document and a document initialization natural language instruction to determine the initialized structured document.

15. The system recited in claim 14, wherein the system is further operable to: determine a document initialization prompt based on a document initialization prompt template, the document initialization prompt and the document initialization prompt template both including the document initialization natural language instruction, wherein determining the document initialization prompt template involves filling a fillable portion included in thedocument initialization prompt template with a portion of the exemplar document.

16. The system recited in any of claims 13 through 15, wherein querying the database system involves executing a designated database query determined by a large language model based on a query prompt that includes a designated portion of the exemplar document corresponding with a designated document section and a query identification natural language instruction to identify the designated database query, wherein the designated database query returns a plurality of designated example document portions identified by the database system as being similar to the designated portion of the exemplar document.

17. A system comprising: means for receiving from a client machine a request to determine a structured document based on an exemplar document and a natural language input instruction characterizing content to include in the structured document; means for determining via a processor document structure information based on an analysis of the exemplar document, the document structure information identifying a plurality of document sections, the document structure information characterizing content to include in the plurality of document sections; means for querying a database system to identify a plurality of example document portions corresponding to the plurality of document sections; means for determining a structured document via a processor, the structured document including a plurality of novel text portions corresponding with the plurality of document sections, the plurality of novel text portions being generated by a large language model based on the plurality of example document portions and the natural language input instruction; and means for transmitting the structured document to the client machine.

18. The system recited in claim 17, further comprising:means for determining an initialized structured document based on the document structure information, the initialized structured document including a plurality of document structure portions corresponding to the plurality of document sections, the initialized structured document further including a plurality of contextual information portions characterizing content to be included in the document structure portions, wherein the plurality of document structure portions and the plurality of contextual information portions are determined by the large language model based on the exemplar document and a document initialization natural language instruction to determine the initialized structured document; and means for determining a document initialization prompt based on a document initialization prompt template, the document initialization prompt and the document initialization prompt template both including the document initialization natural language instruction, wherein determining the document initialization prompt template involves filling a fillable portion included in the document initialization prompt template with a portion of the exemplar document.

19. The system recited in claim 17 or claim 18, further comprising: means for determining a designated portion of the exemplar document corresponding with a designated document section; and means for extracting one or more clauses from the designated portion of the exemplar document, wherein the database system is queried based on the one or more clauses, wherein the one or more clauses are extracted by the large language model based on a metadata extraction prompt that includes the designated portion of the exemplar document and a metadata extraction natural language instruction to extract the one or more clauses.

20. The system recited in any of claims 17 through 19, further comprising: means for determining a first novel text portion based on the natural language input instruction;means for determining a second novel text portion of the plurality of text portions based on the natural language input instruction and the first novel text portion, wherein determining the second novel text portion comprises determining a structured document generation prompt based on a structured document generation prompt template, the structured document generation prompt and the structured document generation prompt template both including a structured document generation natural language instruction to generate novel text based on input, wherein determining the structured document generation prompt involves filling one or more fillable portions of the structured document generation prompt template with the structured document generation natural language instruction and the first novel text portion; and means for determining a third novel text portion of the plurality of text portions based on the natural language input instruction and the first and second novel text portions.

Images