Method and apparatus for facilitating construction of large language model pipelines

Abstract

A textual description of a question related to at least a portion of a device is received as input. A data store is accessed and a plurality of knowledge documents corresponding to the input are retrieved. Language generation cues are then generated from the input and the plurality of knowledge documents and output to a task-specific decoder that generates candidate suggestions to solve the above-mentioned problem. The candidate suggestions are output to at least one human reviewer that reviews the candidate suggestions according to a plurality of knowledge documents and then provides corresponding human verification suggestions to solve the problem. Human verification suggestions in combination with corresponding textual description inputs may be used to retrain the task-specific decoder.

Description

Technical Field

[0001] These teachings relate generally to large language models, and more particularly to large language model pipelines. Background Art

[0002] Large language models are known in the art. They are complex artificial intelligence systems designed to understand and generate human language. Large language models are typically trained on large amounts of text data to learn patterns, grammar, and context, enabling them to generate coherent and contextually appropriate responses. Using deep learning techniques, large language models can analyze and process language at a complex level, enabling them to understand nuanced queries and generate highly relevant and accurate output.

[0003] Large language models can be enhanced through knowledge retrieval. This enhancement typically involves integrating external information sources to enhance the model's understanding and response capabilities. By incorporating a knowledge retrieval system, large language models can access information that may potentially include relevant topic content, thereby facilitating accurate and up-to-date responses. This integration can allow the model to go beyond its pre-trained knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various needs are met, at least in part, by providing methods and apparatus for facilitating the construction of large language model pipelines as described in the following detailed description, particularly when studied in conjunction with the accompanying drawings, which set forth a complete and enabling disclosure of various aspects of the present description, including the best mode thereof, to one of ordinary skill in the art, with reference to the accompanying drawings, wherein:

[0005] Figure 1 includes a block diagram of an example apparatus for building a large language model according to various embodiments of these teachings;

[0006] Figure 2 Including showing that it can be, for example, Figure 1 Flowcharts of various methods performed by illustrative devices;

[0007] Figure 3 Including can be for example by Figure 1 A schematic diagram of an illustrative device-performed maintenance retrieval enhancement model;

[0008] Figure 4 Including can be, for example, Figure 1 a schematic diagram of a training workflow performed by an illustrative device; and

[0009] Figure 5 Including can be, for example, Figure 1 A schematic diagram of an illustrative device performing a semantic search and hint creation flow chart.

[0010] The elements in the figures are shown for simplicity and clarity and are not necessarily drawn to scale. For example, the size and / or relative position of some elements in the figures may be exaggerated relative to other elements to help improve understanding of the various embodiments of the present teachings. In addition, common but well-understood elements that are useful or necessary in commercially feasible embodiments are generally not depicted to provide a less obstructed view of the various embodiments of the present teachings. Certain actions and / or steps may be described or depicted in a particular order of occurrence, but those skilled in the art will understand that such specificity in order is not actually required. DETAILED DESCRIPTION

[0011] Many pieces of equipment, such as jet turbine engines, are complex in design and operation. Maintaining, repairing and / or overhauling ("maintenance") such equipment in a timely and appropriate manner can be challenging. Automation is often viewed as a method that can help in both of these areas, but automating maintenance activities for complex equipment can lead to frustrating errors and results. In fact, automation efforts can be counterproductive in this regard, particularly when compared to similar non-automated instructions provided by subject matter experts. Applicants have determined that attempting to automate these activities with large language models (even large language models enhanced with retrieval of relevant subject matter knowledge) may not produce the expected benefits. In the context of aviation applications, where complex equipment is common, these are significant challenges.

[0012] Generally speaking, various aspects of the present disclosure can be used in conjunction with various methods and / or devices to facilitate the construction of a large-scale language model pipeline enhanced with knowledge retrieval to automate maintenance recommendations for parts of equipment. These teachings can be applied to any of a variety of equipment, including but not limited to jet turbine engines. With respect to jet turbine engines, the aforementioned parts of the equipment may include one or more of a compressor, a heat exchanger, a turbine, and an exhaust nozzle.

[0013] By one approach, these teachings may include receiving as input a textual description of a problem associated with at least a portion of a device, accessing at least one data store, and retrieving a plurality of knowledge documents based, at least in part, on information corresponding to the input. The teachings may then include generating a language generation prompt based, at least in part, on the input and the plurality of knowledge documents, and outputting the language generation prompt to a task-specific decoder, which generates, based, at least in part, on the language generation prompt, at least one candidate suggestion for resolving the problem associated with at least a portion of the device. Outputting the at least one candidate suggestion to at least one human reviewer, which reviews the at least one candidate suggestion based, at least in part, on at least a portion of the plurality of knowledge documents, and then provides a corresponding human-verified suggestion for resolving the problem associated with at least a portion of the device. For example, these teachings may also be adapted to allow a reviewer to verify the set of retrieved knowledge documents used to generate the candidate suggestions. The reviewer-verified list of knowledge documents may also, as desired, identify documents not initially retrieved by the retriever, as well as documents retrieved by the retriever that the reviewer identified as incorrect, incomplete, outdated, under-supported, etc.

[0014] These teachings are then adapted to retrain the task-specific decoder at least in part using the corresponding human-verified suggestions combined with the corresponding textual description input. When available, the aforementioned verified (or otherwise characterized and / or annotated) knowledge documents (e.g., verified by human reviewers) can be used to periodically update the retriever model to improve the model's context-specific retrieval.

[0015] The foregoing may be performed at least in part by a control circuit. By one approach, the control circuit is at least part of a retrieval enhancement generative model.

[0016] By one approach, outputting the language generation hints to the task-specific decoder can include outputting the language generation hints incorporating at least some of the plurality of knowledge documents to the task-specific decoder. By one approach, outputting the language generation hints incorporating at least some of the plurality of knowledge documents to the task-specific decoder can include outputting at least some of the plurality of knowledge documents without specifying any length restrictions. If desired, these teachings can accommodate outputting at least some of the plurality of knowledge documents to the task-specific decoder by each outputting at least some of the plurality of knowledge documents as a single large language model knowledge item.

[0017] These teachings are flexible in practice and will accommodate various modifications and / or supplemental actions. As one example of these aspects, these teachings will accommodate extracting, at least in part, semantic context information from the input to provide extracted semantic context information. In this case, accessing at least one data store and retrieving a plurality of knowledge documents based, at least in part, on information corresponding to the input may include accessing at least one data store and retrieving a plurality of knowledge documents based, at least in part, on the extracted semantic context information.

[0018] By one approach, these teachings can be implemented as a human-in-the-loop maintenance assistant that automates maintenance recommendations for engine parts by building a large language model pipeline augmented by knowledge retrieval, thereby improving the efficiency and productivity of the maintenance recommendation workflow.

[0019] By one approach, the human may be a subject matter expert.

[0020] These teachings can be used to extract the semantic context that maintains the question description and retrieve relevant document sets of different knowledge types. The question description and the retrieved knowledge document sets are then used to generate language generation hints, which are used by a task-specific decoder to generate candidate suggestions / answers to the question, which are output to the subject matter expert.

[0021] The task-specific decoder can be trained using relevant background topic-related data. The retrieved knowledge documents provide the foundation for suggestion generation and can also serve as explanations for the system's responses. Subject matter experts verify the candidate answers and explanations, modify the answers as necessary, and provide the human-verified answers to the end user. Furthermore, this pair of question descriptions and human-verified answers constitutes additional expert-verified data that can be used to update the system's models on an ongoing or occasional basis.

[0022] The terms and expressions used herein have the ordinary technical meanings assigned to them by those skilled in the art, unless otherwise specified herein. Unless otherwise specified, the term "or" as used herein should be interpreted as having a disjunctive rather than a conjunctive structure. Unless otherwise specified herein, the terms "coupled," "fixed," "attached to," and the like refer to both direct coupling, fixing, or attachment and indirect coupling, fixing, or attachment through one or more intermediate components or features.

[0023] The singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0024] As used herein throughout the specification and claims, approximating language is used to modify any quantitative representation that can be permissibly varied without resulting in a change in the basic function to which it is related. Thus, a value modified by one or more terms (e.g., "about," "approximately," and "substantially") is not limited to the precise value specified. In at least some cases, approximating language may correspond to the precision of an instrument used to measure the value, or the precision of a method or machine used to construct or manufacture a component and / or system. For example, approximating language may refer to within a 10% margin.

[0025] These and other advantages may become more apparent upon a thorough review and study of the following detailed description. Figure 1 , an illustrative device 100 compatible with many of these teachings will now be presented.

[0026] In this particular example, enabling device 100 includes control circuitry 101. Thus, as a "circuit," control circuitry 101 includes a structure that includes at least one (and typically many) conductive paths (e.g., paths composed of a conductive metal such as copper or silver) that carry electrical power in an orderly manner, and that typically also includes corresponding electrical components (both passive (e.g., resistors and capacitors) and active (e.g., any of a variety of semiconductor-based devices) to allow the circuitry to implement the control aspects of these teachings.

[0027] Such control circuit 101 may include a fixed-purpose hardwired hardware platform (including but not limited to an application-specific integrated circuit (ASIC) (which is an integrated circuit customized by design for a specific purpose rather than for general use), a field programmable gate array (FPGA), etc.), or may include a partially or fully programmable hardware platform (including but not limited to a microcontroller, a microprocessor, etc.). These architectural options for such a structure are well known and understood in the art and do not require further description here. The control circuit 101 is configured (e.g., by using corresponding programming that will be well understood by those skilled in the art) to perform one or more steps, actions and / or functions described herein.

[0028] It should be understood that the control circuit 101 may comprise a single integrated platform or may comprise a plurality of such circuits working in conjunction with each other.

[0029] The control circuit 101 is operably coupled to a memory 102. The memory 102 may be integrated into the control circuit 101 or may be physically separate (in whole or in part) from the control circuit 101 as desired. The memory 102 may also be local to the control circuit 101 (where, for example, the two share a common circuit board, chassis, power supply, and / or housing) or may be partially or completely remote to the control circuit 101 (where, for example, the memory 102 is physically located in another facility, metropolitan area, or even country than the control circuit 101).

[0030] Like the control circuit 101 , the memory 102 may include a single structure or may include multiple memory platforms, which together constitute the “memory” of the device 100 .

[0031] For illustrative purposes and without limitation, control circuitry 101 may be configured, at least in part, to retrieval-enhance at least a portion of a generative model. A retrieval-enhancement generative model is a natural language processing model that combines elements of retrieval and generation methods. In this model, a retrieval mechanism is employed to retrieve relevant information from a pre-existing dataset or knowledge base. The retrieved information is then used to enhance a generation process, where the model generates new, contextually relevant content based on the retrieved knowledge.

[0032] In addition to information such as the knowledge documents described herein, the memory 102 may also be used, for example, to non-transitory store computer instructions that, when executed by the control circuit 101, cause the control circuit 101 to operate as described herein. (As used herein, references to "non-transitory" will be understood to refer to the non-transitory state of the stored contents (thus, except when the stored contents constitute merely a signal or wave), rather than the volatility of the storage medium itself, and thus include non-volatile memory (e.g., read-only memory (ROM)) as well as volatile memory (e.g., dynamic random access memory (DRAM)).)

[0033] By an optional approach, the control circuitry 101 is further operatively coupled to one or more user interfaces 103. The one or more user interfaces 103 may include any of a variety of user input mechanisms (e.g., but not limited to, a keyboard and keypad, a cursor control device, a touch-sensitive display, a voice recognition interface, a gesture recognition interface, etc.) and / or user output mechanisms (e.g., but not limited to, a visual display, an audio transducer, a printer, etc.) to facilitate receiving information and / or instructions from a user and / or providing information to a user.

[0034] If desired, the control circuit 101 may also be operably coupled to a network interface 104. Thus configured, the control circuit 101 may communicate with other components (within the device 100 and / or external to it, such as one or more remote resources 105) via one or more intermediate networks 106 (e.g., but not limited to the Internet). Network interfaces, including both wireless and non-wireless platforms, are well understood in the art and require no further explanation.

[0035] Now refer to Figure 2 , a process 200 will be described that can be performed at least in part by the device 100 described above. For illustrative purposes, the steps are performed by the control circuit 101 described above (e.g., if the control circuit 101 is, for example, an ASIC or FPGA, via a hardwired configuration, or by executing corresponding software instructions stored in the memory 102 described above). Generally speaking, the process 200 is used to facilitate the construction of a large-scale language model pipeline enhanced with knowledge retrieval to automate maintenance, repair, and overhaul recommendations for parts of equipment, such as jet turbine engines (where the parts may include at least some of the compressor, heat exchanger, turbine, and exhaust nozzle).

[0036] At box 201, the process 200 provides for receiving as input (e.g., via the user interface 103 or network interface 104 described above) a text description of a problem associated with at least a portion of the above-mentioned equipment (e.g., but not limited to a compressor, heat exchanger, turbine, and / or exhaust nozzle of a jet turbine engine). For example, this text can be entered by a technician working on the equipment (e.g., such a technician might refer to this text to refer to a possible problem with a turbine blade). This text description can conform to some form or format of choice, or can essentially include a free-form submission. The text description can be received via any desired mechanism, including but not limited to text messages, emails, scans and OCRed documents, input received via a smartphone application or browser-based service, etc. The length of the text description can be limited in length as desired or without any such restrictions. These teachings will also accommodate receiving a text description in a first human language (e.g., Chinese) and automatically translating the text description into a second, different human language (e.g., English).

[0037] These teachings will also accommodate receiving one or more categorical inputs, such as engine model or part identification. At a later stage in process 200, this information can be used to preselect / prefilter the knowledge index relevant to the selected categorical input. Alternatively, if desired, this categorical information can be included in the retriever and language generation prompts described herein. An illustrative example prompt might be: engine_model:xyz description:[text description].

[0038] At optional block 202, the process 200 provides for extracting semantic context information at least in part from the above-mentioned input to provide extracted semantic context information. Generally speaking, automatically extracting semantic context information from text involves using natural language processing techniques to understand the meaning and relationship between words, phrases and sentences in a given text. Semantic content extraction can be performed by a method that starts by breaking the text into individual words or tags. This helps to simplify processing and understand the basic building blocks of the content. Each tag (noun, verb, adjective, etc.) can then be labeled with its corresponding part of speech. This helps to understand the grammatical structure and the role of each word in the sentence. Named entity recognition can be used to identify key information in the text and classify it into predefined categories, such as the name of a part or component, location, time expression, quantity, various measurements, etc. Dependency parsing can then be used to establish the relationship between the so-called central word and the words that modify these central words. Coreference resolution can be used to identify expressions in the text that refer to the same entity. Semantic role labeling can be used to assign labels to words or phrases in a sentence to indicate the semantic role of these words / phrases in the context of a predicate or action - for example, what component presents what condition? Relation extraction can be used to identify and classify the relationships between entities within the text. Based on the extracted entities, relations, and roles, an ontology or taxonomy can be constructed to represent knowledge in a structured form. And if necessary, the extracted information can be enriched and verified by cross-references with external databases or knowledge bases such as DBpedia, YAGO, or any of the various available ontologies. Other methods can be used as needed.

[0039] At block 203, the process 200 provides for accessing at least one data store and retrieving a plurality of knowledge documents based at least in part on information corresponding to the input. Some or all of the data store may reside in the aforementioned memory 102 and / or one or more remote resources 105 (e.g., private and / or public information servers accessible via the Internet).

[0040] These teachings will be applicable to a variety of different types of knowledge documents. Examples include, but are not limited to, proprietary or public product and / or operator manuals, repair manuals, white papers, master's and doctoral theses, seminar papers, and the like.

[0041] By one approach, these teachings can be adapted to allow these knowledge documents to be grouped into predefined knowledge types. For example, one predefined knowledge type might be related to jet turbine engine heat exchangers, another might be related to jet turbine engine compressors, and yet another might be related to jet turbine engine turbine blades. As used below, these grouped knowledge documents can be used as evidence / explanation to subject matter experts and end users who will receive the output recommendations of process 200.

[0042] By one approach, the data store may comprise, at least in part, a store of vectors corresponding to the types of knowledge populated and queried by a domain-specific encoder / retrieval machine (ie, a language model).

[0043] By one approach, in lieu of or in combination with the above approach, knowledge types may be defined based on semantics specific to the organization adopting process 200 and / or specific to a particular industry or regulatory standard developed by a third party, including an industry group, a standards-setting body, and / or a government regulator.

[0044] By one approach, the number of relevant knowledge items retrieved for each knowledge group can be fixed (e.g., via a corresponding configuration file) or can be flexibly adjusted based on a semantic similarity score that can be calculated by the retriever. The threshold for such a similarity score can itself be fixed or dynamically adjusted on an ongoing basis based on feedback from subject matter experts as discussed herein.

[0045] At block 204, the process 200 provides for generating a language generation prompt based, at least in part, on the input and the plurality of knowledge documents. A language generation prompt is an input or command to a language model, typically used to generate human-like text or responses. Such prompts typically serve as a starting point or instruction for the model to understand the desired output. The prompt can be in the form of a question, statement, or any form of textual input to guide the model in generating relevant and coherent language-based content.

[0046] It is possible that no relevant knowledge documents generated by the above steps exist. In this case, if necessary, the teachings will be adapted to use language generation prompts consisting solely of input provided to a separate task-specific decoder that is trained to generate suggestions using only the input description text. By one approach, the suggestions generated in this case will require mandatory manual attachment of evidence (i.e., knowledge documents selected by human intervention, for example) to support the subject matter expert's final recommendation.

[0047] At block 205, process 200 provides for outputting a language generation prompt to a task-specific decoder (which can be performed by the control circuitry 101 described above, or can be implemented in whole or in part by an independent circuit platform, if desired), which generates at least one candidate suggestion based at least in part on the language generation prompt to solve the original problem associated with at least a portion of the device. By one approach, outputting the language generation prompt to the task-specific decoder includes outputting the language generation prompt and at least some of a plurality of knowledge documents to the task-specific decoder. By one approach, all of the plurality of knowledge documents are submitted collectively to the task-specific decoder. An example of providing a knowledge document to the task-specific decoder is provided below. By one approach, the control circuitry 101 described above provides such a knowledge document.

[0048] In accordance with these teachings, the at least one candidate suggestion can be output to at least one human reviewer, who reviews the at least one candidate suggestion based at least in part on at least a portion (or all) of the plurality of knowledge documents and provides a corresponding human-verified suggestion to resolve the above-mentioned issue related to at least a portion of the device.

[0049] By one approach, human reviewers are subject matter experts in the context of the device / part / knowledge document. These teachings will support the use of subjective and / or objective criteria to identify subject matter experts, as needed. Objective criteria may include being a controlling authority holder in a specific engineering field (a controlling authority holder in a specific engineering field is an individual with the authority and responsibility to oversee and manage technical aspects, standards, and practices within that field), having earned a specific degree or one or more degrees in specific disciplines, etc.

[0050] Performing such a review of at least one candidate recommendation based, at least in part, on at least a portion (or all) of the plurality of knowledge documents can help provide the subject matter expert with an evidence base to understand and inform the candidate recommendation. In turn, this evidence base can help ensure the accuracy and relevance of the recommendation and help avoid so-called illusions that can occasionally occur with such models.

[0051] These teachings will support any of the various modifications of the above aspects. For example, by one approach, a particular portion of a candidate suggestion may include a footnote or active link that informs the reader of a particular knowledge document (or a particular portion of a particular knowledge document) that supports (or may contradict) the particular portion. Color coding may also play a role in these same aspects. As another example, relevant portions of a given dependent / utilized knowledge document may be included inline or otherwise as comments within the text of the candidate suggestion. As another example, weighted tags may be employed to convey the weight that the responding system places on a given knowledge document. This may be particularly useful when the location, interpretation, or conclusions of the knowledge documents are inconsistent.

[0052] As described above, some or all of the multiple knowledge documents can be submitted to a task-specific decoder. By one approach, the above method can use a traditional retrieval-augmented generation (RAG) model that utilizes fixed-length sentence chunks (e.g., traditional dense paragraph retrieval), where the length of the so-called sentence chunk is specified in terms of tokens. In a typical large language model protocol, this length is fixed, i.e., only X tokens are considered. As an illustrative example, a document with 10 sentences is broken into tokens and the system will only accept the first X tokens. For example, the first such token might be the first sentence and half of the second sentence, while the second token only contains another part of the second sentence.

[0053] Applicants have determined that, in the context of the present teachings, the above-described approach may present problems. In particular, each such token typically represents only a few words, and therefore each token may have a correspondingly relatively low semantic value. Furthermore, Applicants have determined that a typical application of these teachings would limit the approach to a much smaller number of tokens than a downstream decoder can technically practically accept.

[0054] Thus, if desired, one approach allows some or all submitted knowledge documents to be submitted without specifying any length restrictions. One approach allows submitted knowledge documents to be submitted as a single, large language model knowledge item. That is, by avoiding the length parameters that characterize other approaches, a single token can contain / represent thousands of words, allowing a single token to correspond to a single knowledge document within a knowledge document. Consequently, the corresponding semantic value of each token is typically much higher than the semantic values ​​that characterize many prior art practices.

[0055] At block 206, the process 200 provides for retraining the task-specific decoder using, at least in part, the corresponding human-verified suggestions in combination with the corresponding textual description input. This retraining can be performed on a case-by-case basis, or the results provided for multiple different prompts / candidate suggestions can be batched for this purpose. These teachings will also be adapted to update the retriever vector index if necessary, thereby facilitating continuous improvement in evidence retrieval and prompt generation. The retraining can be initiated in whole or in part by a human and / or non-human entity (e.g., control circuitry 101) and / or performed in other ways.

[0056] More details consistent with these teachings will now be presented. It should be understood that the specific details of these examples are intended for illustrative purposes and are not intended to suggest any particular limitations on these teachings.

[0057] Figure 3An illustrative example of a retrieval enhancement model 300 is presented, which is configured to assist in maintaining one or more parts of a jet turbine engine. This particular example assumes that the aviation context encoder and the task-specific decoder are trained separately. By one approach, the control circuit 101 described above is configured as the retrieval enhancement model 300.

[0058] In this example, a question (in the form of a text question 301) is input to an encoder retriever 302, which utilizes a semantic document index 303 through a method and outputs corresponding retrieved knowledge documents 304, which are grouped by knowledge type and provided to a task-specific prompt creation function 305 and a subject matter expert 306. The task-specific prompt creation function 305 generates corresponding task-specific prompts 307, which are input to a task-specific decoder 308.

[0059] The task-specific decoder 308 outputs corresponding candidate suggestions to the subject matter expert 306. In this example, the task-specific decoder 308 uses a seq2seq technique as part of generating the suggestions. Seq2seq (short for sequence to sequence) is a class of machine learning algorithms that can transform one sequence of data into another sequence of data. Seq2seq can be used in natural language processing, particularly for tasks such as language translation, text summarization, and conversation modeling. The example shown uses a pair of neural networks: an encoder that processes an input sequence and encodes it into a fixed-dimensional context vector, and a decoder that uses this context vector to generate an output sequence.

[0060] According to this illustrative example, subject matter expert 306 provides a final answer / recommendation 309 that can be used to facilitate real-world maintenance of real-world equipment (e.g., the jet turbine engine described above). In this example, subject matter expert 306 also provides an explanation 310 as to why, for example, the candidate recommendation is correct, partially correct, or completely incorrect. Either or both of these deliverables 309, 310 can then be fed back to, for example, update a retriever index corresponding to a knowledge document 311 that serves as the basis for the corresponding candidate recommendation.

[0061] Figure 4 Illustrative examples of training and inference workflows consistent with these teachings are presented.

[0062] Generally speaking, the function of semantic lookup index 404 is to output decoder prompts with explanation information. The data flow from decoder prompts 405 to answer / suggestion generation decoder 408 represents the training phase, while the data flow from decoder prompts 406 and explanations 407 to answer / suggestion generation decoder 408 and then to user 412 represents the inference and periodic model update phase.

[0063] By one approach, the control circuit 101 is configured to execute the actions of the entire workflow using, for example, corresponding instructions stored in the aforementioned memory 102. In this example, the maintenance context encoder 401 receives a maintenance problem description 402 and a knowledge document 403. In this example, the knowledge document 403 is extracted from relevant design documents, engine shop manuals, and historical maintenance case information (which may include maintenance problem training examples).

[0064] The semantic lookup index 404 receives the output passed by the maintenance context encoder 401 and outputs generated decoder hints 405 for training or 406 for inference, and also outputs corresponding interpretation information representing the knowledge documents that informed these hints.

[0065] The answer / suggestion generation decoder 408 generates a corresponding candidate suggestion 409, which is provided to a subject matter expert 410 (along with the aforementioned interpretation information 407, where applicable). The subject matter expert 410 reviews the material and, upon being able to validate the candidate suggestion 409, transmits a corresponding validation response 411 to one or more users 412, who will implement the recommendation regarding maintenance of the device in question. The validation response 411 can also be utilized by updating the relevant retriever index to reflect the validation.

[0066] When the subject matter expert 410 does not verify the candidate suggestion, but rather modifies the suggestion to provide a more correct suggestion, the modified suggestion can be provided to the user 412 and can also be used for retraining.

[0067] Figure 5 An illustrative example of a semantic search and hint creation process 500 is presented. Figure 2 When optional box 202 is described in the description of , the process can be executed by the above-mentioned control circuit 101. In this example, the semantic search index 501 receives the maintenance problem description context encoding 502 as input. The index 501 then outputs the corresponding K most similar historical maintenance problems 503, related engine workshop manual sections 504 and related component design sections 505 to the decoder hint generator 506. (In the context of data analysis or algorithm design, "K most" refers to identifying the top K elements with the highest frequency in a given data set. The algorithm to solve this problem typically involves creating a frequency distribution of all elements and then using a data structure such as a heap, hash map, or tree to efficiently track and sort these frequencies to extract the top K elements. These data structures may use, for example, the original maintenance problem description, suggestions from the K most similar historical maintenance problems, tolerance limits and optimal component conditions from the related engine workshop manual sections, and engineering design principles from the related component design sections.

[0068] Further aspects of the present disclosure are provided by the subject matter of the following clauses:

[0069] Item 1. A method for facilitating the construction of a knowledge retrieval-enhanced large-scale language model pipeline to automate maintenance, repair, and overhaul recommendations for parts of an apparatus, comprising: receiving, by a control circuit: a textual description of a problem associated with at least a portion of the apparatus as input; accessing at least one data store and retrieving a plurality of knowledge documents based at least in part on information corresponding to the input; generating a language generation prompt based at least in part on the input and the plurality of knowledge documents; outputting the language generation prompt to a task-specific decoder, the task-specific decoder generating at least one candidate suggestion based at least in part on the language generation prompt to resolve the problem associated with at least a portion of the apparatus, the at least one candidate suggestion being output to at least one human reviewer, the at least one human reviewer reviewing the at least one candidate suggestion based at least in part on at least a portion of the plurality of knowledge documents and providing a corresponding human-verified suggestion to resolve the problem associated with at least a portion of the apparatus; wherein the task-specific decoder is retrained at least in part using the corresponding human-verified suggestion in combination with the corresponding textual description input.

[0070] Clause 2. The method of any preceding clause, wherein the device comprises a jet turbine engine.

[0071] Clause 3. The method of any preceding clause, wherein the parts of the equipment include at least some of a compressor, a heat exchanger, a turbine, and an exhaust nozzle.

[0072] Clause 4. The method of any preceding clause, wherein the control circuitry comprises, at least in part, retrieving a portion of an enhanced generative model.

[0073] Clause 5. The method of any preceding clause, wherein outputting the language generation prompt to the task-specific decoder comprises outputting the language generation prompt and at least some of the plurality of knowledge documents to the task-specific decoder.

[0074] Clause 6. The method of any preceding clause, wherein outputting at least some of the plurality of knowledge documents to the task-specific decoder comprises outputting the at least some of the plurality of knowledge documents without specifying any length restriction.

[0075] Clause 7. A method according to any preceding clause, wherein outputting at least some of the plurality of knowledge documents to the task-specific decoder includes outputting each of the at least some of the plurality of knowledge documents as a single large language model knowledge item.

[0076] Item 8. A method according to any preceding item, further comprising: extracting semantic context information at least in part from the input to provide extracted semantic context information; and wherein accessing the at least one data store and retrieving a plurality of knowledge documents based at least in part on the information corresponding to the input comprises accessing the at least one data store and retrieving the plurality of knowledge documents based at least in part on the extracted semantic context information.

[0077] Item 9. A method for facilitating construction of a knowledge retrieval-enhanced large-scale language model pipeline to automate maintenance, repair, and overhaul recommendations for parts of an apparatus, comprising: by a control circuit: receiving as input a textual description of a problem associated with at least a portion of the apparatus; accessing at least one data store and retrieving a plurality of knowledge documents based at least in part on information corresponding to the input; generating a language generation prompt based at least in part on the input and the plurality of knowledge documents; outputting the language generation prompt to a task-specific decoder, the task-specific decoder generating at least one candidate suggestion based at least in part on the language generation prompt to resolve the problem associated with at least a portion of the apparatus; by a human reviewer: accessing the at least one candidate suggestion and reviewing the at least one candidate suggestion based at least in part on at least a portion of the plurality of knowledge documents; providing a corresponding human-verified suggestion to resolve the problem associated with at least a portion of the apparatus; and retraining the task-specific decoder at least in part using the corresponding human-verified suggestion in combination with the corresponding textual description input.

[0078] Clause 10. The method of any preceding clause, wherein the device comprises a jet turbine engine.

[0079] Clause 11. The method of any preceding clause, wherein the control circuitry comprises, at least in part, retrieving a portion of an enhanced generative model.

[0080] Clause 12. The method of any preceding clause, wherein outputting the language generation prompt to the task-specific decoder comprises outputting the language generation prompt and at least some of the plurality of knowledge documents to the task-specific decoder.

[0081] Clause 13. The method of any preceding clause, wherein outputting at least some of the plurality of knowledge documents to the task-specific decoder comprises outputting the at least some of the plurality of knowledge documents without specifying any length restriction.

[0082] Clause 14. The method of any preceding clause, wherein outputting at least some of the plurality of knowledge documents to the task-specific decoder comprises outputting each of the at least some of the plurality of knowledge documents as a single large language model knowledge item.

[0083] Item 15. The method according to any preceding item further includes: by the control circuit: extracting semantic context information at least in part from the input to provide extracted semantic context information; and wherein accessing the at least one data store and retrieving a plurality of knowledge documents at least in part based on the information corresponding to the input includes accessing the at least one data store and retrieving the plurality of knowledge documents at least in part based on the extracted semantic context information.

[0084] Item 16. A device for facilitating construction of a knowledge retrieval-enhanced large-scale language model pipeline to automate maintenance, repair, and overhaul recommendations for parts of an apparatus, comprising: a control circuit configured to: receive as input a textual description of a problem associated with at least a portion of the apparatus; access at least one data store and retrieve a plurality of knowledge documents based at least in part on information corresponding to the input; generate a language generation prompt based at least in part on the input and the plurality of knowledge documents; output the language generation prompt; a task-specific decoder configured to receive the language generation prompt and responsively generate at least one candidate suggestion based at least in part on the language generation prompt to resolve the problem associated with at least a portion of the apparatus, the at least one candidate suggestion being output to at least one human reviewer, the at least one human reviewer reviewing the at least one candidate suggestion based at least in part on at least a portion of the plurality of knowledge documents and providing a corresponding human-verified suggestion to resolve the problem associated with at least a portion of the apparatus; wherein the task-specific decoder is retrained at least in part using the corresponding human-verified suggestion in combination with the corresponding textual description input.

[0085] Clause 17. The apparatus of any preceding clause, wherein the control circuitry is configured, at least in part, to retrieve at least a portion of an enhanced generative model.

[0086] Item 18. An apparatus according to any preceding item, wherein the control circuit is configured to output the language generation prompt in combination with at least some of the plurality of knowledge documents, and wherein the task-specific decoder is configured to receive the language generation prompt in combination with at least some of the plurality of knowledge documents.

[0087] Item 19. An apparatus according to any preceding item, wherein the control circuit is configured to output at least some of the plurality of knowledge documents to the task-specific decoder by at least partially outputting at least some of the plurality of knowledge documents without specifying any length limit.

[0088] Item 20. An apparatus according to any preceding item, wherein the control circuit is configured to output at least some of the plurality of knowledge documents to the task-specific decoder by, at least in part, outputting at least some of the plurality of knowledge documents each as a single large language model knowledge item.

[0089] Item 21. A non-transitory computer-readable storage medium comprising instructions that, when executed, cause control circuitry to receive as input a textual description of a problem associated with at least a portion of the device; access at least one data store and retrieve a plurality of knowledge documents based at least in part on information corresponding to the input; generate a language generation prompt based at least in part on the input and the plurality of knowledge documents; output the language generation prompt; a task-specific decoder configured to receive the language generation prompt and, based at least in part on the language generation prompt, responsively generate at least one candidate suggestion to resolve the problem associated with at least a portion of the device, the at least one candidate suggestion being output to at least one human reviewer, the at least one human reviewer reviewing the at least one candidate suggestion based at least in part on at least a portion of the plurality of knowledge documents and providing a corresponding human-verified suggestion to resolve the problem associated with at least a portion of the device; wherein the task-specific decoder is retrained at least in part using the corresponding human-verified suggestion in combination with the corresponding textual description input.

[0090] Clause 22. The non-transitory computer-readable storage medium of any preceding clause comprising instructions that, when executed, cause the control circuitry to be configured, at least in part, to retrieve at least a portion of an enhanced generative model.

[0091] Item 23. A non-transitory computer-readable storage medium according to any preceding item comprising instructions, which, when executed, cause the control circuit to be configured to output the language generation prompt in combination with at least some of the multiple knowledge documents, and wherein the task-specific decoder is configured to receive the language generation prompt in combination with at least some of the multiple knowledge documents.

[0092] Item 24. A non-transitory computer-readable storage medium according to any preceding item comprising instructions that, when executed, configure the control circuit to output at least some of the plurality of knowledge documents to the task-specific decoder by at least partially outputting at least some of the plurality of knowledge documents without specifying any length restrictions.

[0093] Item 25. A non-transitory computer-readable storage medium according to any preceding item comprising instructions that, when executed, configure the control circuit to output at least some of the plurality of knowledge documents to the task-specific decoder by at least partially outputting at least some of the plurality of knowledge documents as a single large language model knowledge item.

Claims

1. A method for facilitating the construction of a large language model pipeline enhanced with knowledge retrieval to automate maintenance, repair, and overhaul recommendations for parts of equipment, characterized in that include: Through the control circuit: receiving as input a textual description of a problem associated with at least a portion of the device; accessing at least one data store and retrieving a plurality of knowledge documents based at least in part on information corresponding to the input; generating a language generation prompt based at least in part on the input and the plurality of knowledge documents; as well as outputting the language-generated prompt to a task-specific decoder, the task-specific decoder generating at least one candidate suggestion for resolving the problem with at least a portion of the device based at least in part on the language-generated prompt, the at least one candidate suggestion being output to at least one human reviewer, the at least one human reviewer reviewing the at least one candidate suggestion based at least in part on at least a portion of the plurality of knowledge documents and providing a corresponding human-verified suggestion for resolving the problem with at least a portion of the device; wherein the task-specific decoder is retrained at least in part using the corresponding human-verified suggestions in combination with the corresponding textual description input.

2. The method according to claim 1, characterized in that The equipment includes a jet turbine engine.

3. The method according to claim 2, characterized in that The parts of the equipment include at least some of a compressor, a heat exchanger, a turbine and an exhaust nozzle.

4. The method according to claim 1, wherein wherein the control circuitry comprises, at least in part, a portion of a retrieval enhancement generative model.

5. The method according to claim 4, characterized in that Wherein outputting the language generation prompt to the task-specific decoder includes outputting the language generation prompt and at least some of the plurality of knowledge documents to the task-specific decoder.

6. The method according to claim 5, characterized in that Wherein outputting at least some of the plurality of knowledge documents to the task-specific decoder comprises outputting the at least some of the plurality of knowledge documents without specifying any length restriction.

7. The method according to claim 5, characterized in that Wherein outputting at least some of the plurality of knowledge documents to the task-specific decoder comprises outputting each of the at least some of the plurality of knowledge documents as a single large language model knowledge item.

8. The method according to claim 1, characterized in that Further including: extracting, at least in part, semantic context information from the input to provide extracted semantic context information; And wherein accessing the at least one data store and retrieving a plurality of knowledge documents based at least in part on the information corresponding to the input comprises accessing the at least one data store and retrieving the plurality of knowledge documents based at least in part on the extracted semantic context information.

9. A method for facilitating the construction of a large language model pipeline enhanced with knowledge retrieval to automate maintenance, repair, and overhaul recommendations for parts of equipment, characterized in that include: Through the control circuit: receiving as input a textual description of a problem associated with at least a portion of the device; accessing at least one data store and retrieving a plurality of knowledge documents based at least in part on information corresponding to the input; generating a language generation prompt based at least in part on the input and the plurality of knowledge documents; as well as outputting the language generation prompt to a task-specific decoder, the task-specific decoder generating at least one candidate suggestion to resolve the problem associated with at least a portion of the device based at least in part on the language generation prompt; By human reviewers: accessing the at least one candidate suggestion and reviewing the at least one candidate suggestion based at least in part on at least a portion of the plurality of knowledge documents; providing a corresponding human-verified suggestion to resolve the problem with at least a portion of the device; as well as The task-specific decoder is retrained using, at least in part, the corresponding human-verified suggestions in combination with the corresponding textual description input.

10. The method according to claim 9, characterized in that The equipment includes a jet turbine engine.

Images