Artificially intelligent assistant for work protocols

Abstract

To provide systems and methods for aggregating disparate documentation and metadata from multiple programs and sources into centralized large language models.SOLUTION: A computing system 100 comprises a storage device 102 for storing at least technical documents 104. The technical documents include at least work protocols 112 and technical data files tagged with metadata. One or more large language models 134 are trained on at least the work protocols, and text and metadata 122, 124 extracted from the technical documents. An AI assistant 142 is commissioned for the large language models, and at least retrieves text and metadata in response to receiving a prompt. In a digital work environment 140 including an interface for the AI assistant, the AI assistant provides a contextual response via the digital work environment based on the received prompt.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001]

[0001] The present disclosure relates generally to using artificial intelligence assistants for the purpose of authoring, delivering, and executing engineering and manufacturing work protocols. [Background technology]

[0002]

[0002] Large engineering and manufacturing companies rely on critical documentation in the form of design specifications, procedures, policies, and computer-aided drafting (CAD) metadata to enable daily operations throughout the product lifecycle, from the planning and design stages to manufacturing and post-delivery support.

[0003]

[0003] To create and / or use model-based definitions (MBD) and model-based instructions (MBI), such as work protocols, engineers and technicians must manually access and read technical documentation, specifications, policies, and procedures stored in multiple locations. Accessing and reviewing such information typically requires manual effort, requiring engineers to aggregate disparate documentation and metadata from multiple programs and sources. Furthermore, accessing and using each system and its associated documentation requires a significant amount of tacit knowledge and programming expertise. Few employees will possess all of the institutional knowledge necessary to produce comprehensive coverage. Summary of the Invention

[0004] A computer-implemented method includes storing technical documentation in a storage device. The technical documentation includes at least a work protocol and a technical data file tagged with metadata. One or more large-scale language models are trained with at least the work protocol, text extracted from the technical documentation, and the metadata. An artificial intelligence assistant is commissioned to the large-scale language model, and the artificial intelligence assistant is configured to retrieve at least the text and metadata in response to receiving a prompt. A digital work environment is provided that includes an interface for the artificial intelligence assistant. A prompt related to a first work protocol is received at the artificial intelligence assistant. Text and metadata related to the first work protocol are retrieved via the large-scale language model based on the received prompt. A contextual response is provided via the digital work environment based on the retrieved text and metadata. [Brief explanation of the drawings]

[0005] [Figure 1]

[0005] An exemplary computing system with a storage device and a large language model is shown. [Figure 2]

[0006] 1 shows a flow diagram of an exemplary computer-implemented method for delegating an artificial intelligence assistant to a large-scale language model. [Figure 3]

[0007] 1 illustrates an exemplary digital work environment featuring an artificial intelligence assistant. [Figure 4]

[0008] 1 shows a flow diagram of an exemplary computer-implemented method for operating an artificial intelligence assistant. [Figure 5]

[0009] 1 illustrates an example scenario of an artificial intelligence assistant providing responses to text prompts. [Figure 6]

[0010] 1 illustrates an exemplary scenario of an artificial intelligence assistant providing responses to contextual progress through a work protocol. [Figure 7]

[0011] 1 illustrates an example scenario of an artificial intelligence assistant that provides contextual prompts and responses. [Figure 8]

[0012] 1 illustrates an exemplary scenario of an artificial intelligence assistant dynamically reordering work protocols. [Figure 9]

[0013] 1 shows a user equipped with an augmented reality device in a work environment. [Figure 10]

[0014] 1 illustrates an exemplary augmented reality digital work environment featuring an artificial intelligence assistant. [Figure 11]

[0015] 1 shows a flow diagram of an exemplary computer-implemented method for generating low-grained work protocols using an artificial intelligence assistant. [Figure 12]

[0016] 1 illustrates an exemplary scenario for generating a low-grained working protocol. [Figure 13]

[0017] 1 illustrates an exemplary scenario for adapting a low-granularity work protocol. [Figure 14]

[0018] 1 shows a flow diagram of an exemplary computer-implemented method for contextual retraining of large-scale language models. [Figure 15]

[0019] 1 illustrates an example scenario of an artificial intelligence assistant providing responses to text prompts. [Figure 16]

[0020] 1 illustrates an example scenario of an artificial intelligence assistant providing responses to text prompts after retraining a large-scale language model. [Figure 17]

[0021] 1 illustrates a schematic diagram of an exemplary computing system. DETAILED DESCRIPTION OF THE INVENTION

[0006]

[0022] Provided herein are systems and methods for aggregating disparate documents and metadata from multiple programs and sources into a centralized large-scale language model (LLM). An artificial intelligence (AI) assistant is commissioned to provide an interface between the user and the LLM. Such an AI assistant can quickly and efficiently inform engineers involved in design and authoring decisions with relevant, contextual information, while also facilitating manufacturing by intelligently informing engineers of work order processes and contextual procedures. The AI assistant and LLM intelligently aggregate data, provide access to the data, and deliver the data to users of the work order. This offers advantages over traditional methods that require users of the work order to manually search for the called-out information.

[0007]

[0023] The disclosed method utilizes an LLM built from existing and evolving enterprise and engineering documentation. The documentation is then made queryable from within the digital work protocol. Input can be provided via user input in the form of text entries and contextual progressions made within the digital work instruction application, which is then sent to an AI assistant that interacts with the LLM. Responses from the AI assistant are provided in the digital work instruction application and displayed to the user through text within the application, translation of virtual objects, and highlighting of applicable object features.

[0008]

[0024] By providing an intelligent AI assistant within digital work instructions, this tacit knowledge is made explicit and easily accessible to the entire organization. Other similar "chatbots" can typically only be trained on public data, rather than private and proprietary data, and are not applied in the context of work instructions or complex systems visualization.

[0009]

[0025] Technical advantages of implementing the disclosed systems and methods include substantial cost and time savings by eliminating training, design, and manufacturing time. Furthermore, the AI assistant and LLM can be used to generate low-granularity work protocols that can be used as a cross-enterprise standard. Furthermore, the LLM can be retrained whenever documentation is updated, either by the user or the AI assistant itself. The disclosed systems and methods can reduce errors in procedural planning, design, and execution during production, while also allowing engineers to more easily understand their work in context, reference necessary documentation, and reduce rework due to errors.

[0010]

[0026] FIG. 1 illustrates an exemplary computing system 100. The computing system 100 includes a storage device 102 configured to store at least technical documentation 104. The storage device 102 may be a network-accessible storage drive, such as a locally or remotely accessible storage drive. The storage device 102 may also include one or more local storage devices. For example, a subset of the technical documentation 104 may be downloaded to a local storage drive for local retrieval and / or offline use. When configured as a network-accessible storage drive, the storage device 102 may be located behind a firewall and / or other level of security and may not be accessible outside the local network and / or without authentication information. The technical documentation 104 includes CAD files 106, bills of materials 108, current and projected inventory 110, work protocols 112, reference documents 114, and specifications 116. The work protocols 112 may include both engineering and manufacturing work protocols, as well as training protocols, repair protocols, inspection protocols, etc. The work protocols may include work instructions and other materials used for pre-production, in-production, and / or post-production of one or more programs. Specifications 116 may include local specifications, customer specifications, etc. for systems, subsystems, and components. Technical documentation 104 may further include analytical models, geometric models, behavioral models, analytical models, electrical models, compliance models, requirements models, functional models, design models, and / or integrated simulation models, code, verification and validation reports, 2D schematics, 3D schematics, unstructured documentation, and / or other suitable information related to the design and / or manufacture of a tangible article, such as an aircraft, spacecraft, and / or other aeronautical component, thereby enabling a person skilled in the art to build or assemble the tangible article given the technical data file(s).As used herein, technical documentation broadly refers to documents that may contain information related to executing a set of instructions, for example, documents containing information that can be used by engineers in executing engineering and / or manufacturing work protocols. In some examples, technical documentation associated with different projects may be stored in different sub-repositories.

[0011]

[0027] Extracted data 120 may be extracted from technical documents 104. The extracted data may include text data 122, metadata 124, and image data 126. The extracted data 120 may be fed to one or more machines, such as a natural language processor 130 and / or a machine learning module 132. The natural language processor 130 and the machine learning module 132 may be configured to train one or more large-scale language models 134 with the extracted data 120. A digital work environment 140 may be provided to users who wish to interact with the large-scale language models 134. An artificial intelligence assistant 142 may be commissioned to enable users to submit prompts and queries to the large-scale language models 134.

[0012]

[0028] In this manner, the LLM 134 can be constructed from existing and evolving enterprise and engineering documentation. The documentation is then made searchable and retrievable from the digital work environment 140 via an artificial intelligence assistant 142. In particular, the system enables the aggregation of disparate documents and metadata from multiple programs and sources into a centralized LLM and implements an interface for accessing the embedded AI assistant. This AI assistant quickly and efficiently informs engineers of relevant contextual information involved in design and authoring decisions, while also facilitating manufacturing through intelligently informing engineers of contextual work instruction processes and procedures. The LLM 134 can be stored on a network-accessible storage device, such as a cloud computer, and / or on a local storage device. In some embodiments, a lightweight version of the LLM 134, such as a version that handles one or more specific programs, can be downloaded to local storage.

[0013]

[0029] The machine may be implemented using any suitable combination of state-of-the-art and / or future machine learning (ML), artificial intelligence (AI), and / or natural language processing (NLP) techniques. Non-limiting examples of techniques that may be incorporated into one or more machine implementations include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos, temporal convolutional neural networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural network configured to convolve and pool features across one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural tubes, etc.), and the like. These techniques include neural networks (e.g., neural random access memories and / or neural network algorithms), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases), and / or natural language processing techniques (e.g., tokenization, stemming, syntactic parsing, and / or dependency analysis, and / or intent recognition, segmental models, and / or super-segmental models (e.g., hidden dynamic models, etc.)).

[0014]

[0030] In some examples, the methods and processes described herein may be implemented using one or more differentiable functions. In that case, the gradient of the differentiable function may be calculated and / or estimated with respect to the inputs and / or outputs of the differentiable function (e.g., with respect to training data and / or with respect to an objective function). Such methods and processes may be determined at least in part by a set of trainable parameters. Thus, the trainable parameters for a particular method or process may be adjusted through any suitable training procedure to continuously improve the performance of the method or process.

[0015]

[0031] Non-limiting examples of training procedures for adjusting trainable parameters include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on classes derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning) and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consensus), contextual bandit methods, maximum likelihood, and / or expectation-maximization. In some embodiments, multiple methods, processes, and / or components of the systems described herein may be trained simultaneously with respect to an objective function measuring the performance of a collective function of the multiple components (e.g., with respect to reinforcement feedback and / or with respect to labeled training data). Training multiple methods, processes, and / or components simultaneously can improve such collective function. In some embodiments, one or more methods, processes, and / or components may be trained independently of other components (e.g., offline training on historical data).

[0016]

[0032] A language model can utilize lexical features to guide word sampling / retrieval for speech recognition. For example, a language model can be defined at least in part by a statistical distribution of words or other lexical features. For example, a language model can be defined by a statistical distribution of n-grams and define transition probabilities between candidate words according to lexical statistics. A language model can also be based on any other suitable statistical features and / or the results of processing the statistical features with one or more machine learning and / or statistical algorithms (e.g., confidence values obtained from the results of such processing). In some embodiments, a statistical model can constrain what words can be recognized for a speech signal, for example, based on an assumption that the words in the speech signal come from a particular vocabulary.

[0017]

[0033] Alternatively, or in addition, the language model can represent the speech input and words in a shared latent space, e.g., a vector space learned by one or more speech and / or word models (e.g., wav2letter and / or word2vec), based on one or more previously trained neural networks. Thus, finding candidate words can include searching the shared latent space based on vectors encoded by the speech models for the speech input to find candidate word vectors for decoding by the word models. The shared latent space can be used to assess, for one or more candidate words, a confidence that the candidate word is contained in the speech.

[0018]

[0034] The language model may be used in conjunction with an acoustic model configured to evaluate, for a candidate word and the speech signal, a confidence that the candidate word is included in a sound in the speech signal based on acoustic features of the word (e.g., Mel-frequency Cepstral coefficients, formants, etc.). Optionally, in some embodiments, the language model may incorporate the acoustic model (e.g., evaluation and / or training of the language model may be based on the acoustic model). The acoustic model defines a mapping between the acoustic signal and basic sound units such as phonemes, for example, based on labeled speech. The acoustic model may be based on any suitable combination of state-of-the-art or future ML and / or AI models, for example, deep neural networks (e.g., long short-term memory, temporal convolutional neural networks, restricted Boltzmann machines, deep belief networks), hidden Markov models (HMMs), conditional random fields (CRFs) and / or Markov random fields, Gaussian mixture models, and / or other graphical models (e.g., deep Bayesian networks). The speech signal processed by the acoustic model may be pre-processed in any suitable manner, such as encoding at any suitable sampling rate, Fourier transforming, bandpass filtering, etc. The acoustic model may be trained to recognize a mapping between the acoustic signal and sound units based on training with labeled speech data. For example, the acoustic model may be trained based on labeled speech data including speech and modified text to learn a mapping between the speech signal and the sound units indicated by the modified text. Thus, the acoustic model may be continuously refined to improve its usefulness for correctly recognizing speech.

[0019]

[0035] In some examples, in addition to statistical models, neural networks, and / or acoustic models, the language model can incorporate any suitable graphical model, e.g., an HMM or a CRF. The graphical model can utilize statistical features (e.g., transition probabilities) and / or confidence values to determine the probability of recognizing a word given speech and / or other previously recognized words. Thus, the graphical model can utilize statistical features, previously trained machine learning models, and / or acoustic models to define transition probabilities between states represented in the graphical model.

[0020]

[0036] 2 illustrates a flow diagram of an exemplary computer-implemented method 200 for delegating an artificial intelligence assistant to a large-scale language model. FIG. 2 may be implemented by one or more computing systems, such as computing system 100.

[0021]

[0037] At 210, method 200 includes storing the technical document in a storage device. The technical document includes at least a work protocol and a technical data file tagged with metadata. The storage device may include one or more local storage devices and / or one or more network-accessible storage devices. In some embodiments, a plurality of images are extracted from the technical document. The extracted images may also be used to train a large-scale language model. In some embodiments, the stored technical document includes CAD engineering files and / or drawing files and associated metadata.

[0022]

[0038] As shown in FIG. 1 , technical documentation may further include engineering documentation, specifications, policies, procedural design specifications, CAD metadata, inventory, model-based rules, model-based instructions, etc. Technical documentation may be stored in multiple locations and / or various network-accessible storage devices. Technical documentation may further include organization-wide policies, such as clearance policies and security policies. Technical documentation may aggregate documentation and metadata from multiple programs and sources. In some examples, technical documentation may include unstructured documentation, such as free-form input from users and notes from exit interviews.

[0023]

[0039] At 220, method 200 includes extracting text and metadata from the technical documents. In some embodiments, the technical documents may include contextual links to other technical documents via reciprocal or cross-links. Such links may be preserved during extraction. The technical documents may evolve and / or be edited over time. The LLM may be continuously retrained periodically or whenever the documents are updated.

[0024]

[0040] At 230, method 200 includes training one or more large-scale language models with at least the work protocol, the extracted text, and the metadata, as described with respect to LLM 134 and FIG. 1 . The training may further include supporting documents and / or required specifications, such as those referenced but not included in the work protocol itself. In some embodiments, one or more large-scale language models are trained to cross-link the extracted text and metadata for each work protocol. In some embodiments, technical documentation is stored per program for multiple programs. In such embodiments, one or more large-scale language models may be trained per program. A program may include a particular manufacturing endpoint (e.g., an aircraft) or a subset of that endpoint (e.g., a jet engine). In some embodiments, an LLM may be trained for each program or subset of programs using program-specific documents. Procedure and policy documents may affect multiple programs. For example, a clearance document may specify the level of clearance required for employees to work on specific projects. Other documents may be documents specific to a particular project (e.g., a parts list) or subproject.

[0025]

[0041] At 240, method 200 includes delegating an artificial intelligence assistant to one or more large-scale language models. The artificial intelligence assistant is configured to at least search text and metadata in response to receiving a prompt. Delegating the AI assistant may include training the AI assistant to respond to the prompt using information stored in the LLM. The AI assistant can assist technical personnel and / or customers in accessing hard-to-find and difficult-to-understand data through a centralized location, effectively converting tacit knowledge into explicit, accessible knowledge.

[0026]

[0042] At 250, method 200 includes providing a digital work environment including an interface for an artificial intelligence assistant. The trained AI assistant may be queriable from within the digital work environment, which may allow a user to access multiple aspects of assembly orders, assembly procedures, policies, specifications, and metadata from the storage device.

[0027]

[0043] FIG. 3 illustrates a computing device 300 operating an exemplary digital work environment 302 featuring an artificial intelligence assistant 304. Digital work environment 302 may be an example of digital work environment 140, and AI assistant 304 may be an example of AI assistant 142. Digital work environment 302 is provided merely as an exemplary interface; any number of interfaces are possible. In this example, digital work environment 302 includes a navigation pane 306. An exemplary work protocol 308 is presented as a series of steps 310, 312, 314, 316, and 318. Steps 310 and 312 are indicated as completed with checkmarks. The user is then engaged in step 15.6, a substep of the protocol for securing the wire harness. Further information about step 15.6 is presented to the user at 320. AI assistant 304 includes at least a prompt box 322 and a response box 324. The user may ask questions such as, but not limited to, questions related to step 15.6 via prompt box 322. For example, in step 312, the user may ask how long it will take to secure the wire harness.

[0028]

[0044] The digital work environment may enable contextual deployment of somewhat generalized work instructions at runtime. For example, the digital work environment may work with an AI assistant to inform a manufacturer of contextual work instruction processes and procedures. The AI assistant may be configured to search for contextual information related to work protocols and provide that contextual information to the user through the digital work environment.

[0029]

[0045] 4 shows a flow diagram of an exemplary computer-implemented method 400 for operating an artificial intelligence assistant. Method 400 may be implemented by one or more computing systems, such as computing systems 100 and 300. Method 400 may be performed in conjunction with method 200, as indicated by the circled A.

[0030]

[0046] At 410, method 400 includes receiving, at an artificial intelligence assistant, a prompt for a first work protocol. Alternative embodiments of this AI assistant may be used in a web browser or outside of a particular digital work environment as a means to query and access aggregated data. The prompt may be derived from text input entered by a user, such as via a keyboard or via speech-to-text conversion. The prompt may be provided by a user performing installation, repair, maintenance, training, inspection, or design (e.g., contextual visualization). The prompt may relate to a current work protocol, such as by invoking procedural information, specifications, and / or policy references within the work protocol. In other examples, the prompt may be derived from the AI assistant itself. In some examples described further herein, the prompt may be a visual prompt. As used herein, a “first work protocol” simply refers to a particular work protocol among multiple potential work protocols. “First” does not imply timing or order, except where expressly stated. It is contemplated that a single user may be involved in multiple work protocols over the course of method 400 and / or its iterations.

[0031]

[0047] As an example, Figure 5 illustrates a scenario 500 executed on computing system 300. As shown at 316, a user is executing step 15.6 of work protocol 308. The user enters a text prompt 502 asking AI assistant 304 how much torque should be applied to tighten the bolt.

[0032]

[0048] Referring back to FIG. 4 , at 420, method 400 includes retrieving text and metadata related to the first work protocol through a large-scale language model based on the received prompt. For example, the text and metadata may be retrieved through an LLM. The retrieved text and metadata may conform to the work protocol while being sensitive to the specific context encountered while the technician was performing the work protocol. For example, if the user provided a prompt regarding inventory for a particular part, the AI assistant would retrieve text and / or metadata from a corresponding inventory spreadsheet or from communications with external inventory management system(s) via API calls. This may be facilitated through contextually linked documents, enabling rapid retrieval of pertinent contextual information.

[0033]

[0049] At 430, the method 400 includes providing a contextual response via the digital workspace based on the retrieved text and metadata. The response from the AI assistant is received in the digital workspace application and displayed to the user via text within the application, translation of virtual objects, and highlighting of applicable features.

[0034]

[0050] As an example, FIG. 5 shows the AI assistant 304 searching text and metadata related to the user prompt 502 to provide a text response 504 (target 50 inch-pounds). This essentially requires the AI assistant to know which bolt the user is referring to. Presumably, there is a single bolt that will be tightened during step 15.6. In some embodiments, if the AI assistant cannot find the answer, the AI assistant can query the user for further or more specific information.

[0035]

[0051] In some examples, in addition to or instead of text responses, contextual responses may include visual responses to the user via app images, highlights, and / or animated responses within the 3D model, where applicable, to aid in visualizing complex design integration in context for both the engineer and supporting technicians in understanding and executing complex procedures. When a user asks the AI assistant to highlight a particular part, they may also be provided with access to metadata indicating when that part was installed, who installed the part, the inventory status of the associated part, which other steps the part is involved in, which control points this part is involved in, etc.

[0036]

[0052] In some examples, the prompt includes a contextual progression through a first work protocol entered via the digital work environment. For example, FIG. 6 shows an exemplary scenario 600 executed on computing device 300. As shown at 316, the user has completed step 15.6 and moved to step 15.7, as shown at 318. At 602, detailed instructions are provided to the user within digital work environment 302. Here, the user has not entered a prompt, as shown at 604. Rather, AI assistant 304 receives the prompt that the user has progressed to step 15.7 and presents a contextual hint at 606 regarding detailed instructions for this step. This contextual hint may be the result of a previous user creating a nonconformance report during this step or a later related step.

[0037]

[0053] In some examples, the prompts include contextual prompts from the artificial intelligence assistant. The resulting response may be contextual hints or information relevant to the task at hand. For example, an engineer may be informed of a design that violates required build specifications, or an accessibility issue may be highlighted. For example, overlapping or adjacent systems may be considered. Contextual "hints" and information may be displayed within the digital work environment to aid in visualizing future procedures, fault / error detection, and virtual buildup to nonconformance reports (NCRs). This may give the user the feeling that the flow of the work protocol is being controlled by the AI assistant. For example, a contextual prompt from the artificial intelligence assistant may relate to one or more nonconformance reports.

[0038]

[0054] As one example, Figure 7 shows an exemplary scenario 700 executed on computing device 300. The user is at step 15.7, as shown at 318. The user has not entered a prompt, as shown at 704. The AI assistant 304 searches inventory data for step 15.7 and suggests replacement parts that can be used in place of the unavailable part, at 706.

[0039]

[0055] In some examples, an artificial intelligence assistant may be configured to dynamically reallocate resources based on contextual prompts from the artificial intelligence assistant. For example, a technician may be nearing the end of their shift. The AI assistant may instruct them to perform simple tasks that can be completed during their shift, rather than embarking on a lengthy procedure that would require them to stop mid-shift.

[0040]

[0056] In some examples, the artificial intelligence assistant may be configured to dynamically reorder the first work protocol based on the received text and metadata. For example, if inventory is not on hand for a current step in the work protocol, the technician may be instructed to skip one or more steps.

[0041]

[0057] The AI assistant may provide instructions to intelligently skip steps, thereby avoiding rework or creating future problems. The intelligent reordering of steps may be based on one or more pieces of information (e.g., inventory). The AI assistant may consider the technician's qualifications and / or the technician's non-conformance reports for similar work in selective reordering. Thus, the AI assistant can dynamically change resource allocation and reordering based on the technician's qualifications.

[0042]

[0058] As one example, Figure 8 shows an exemplary scenario 800 executed on computing device 300. The user is at step 15.7, as shown at 318. The user has not entered a prompt, as shown at 804. The AI assistant 304 retrieves inventory data for step 15.7 and indicates at 806 that the user should skip to step 17.0.

[0043]

[0059] In some embodiments, the digital work environment is an augmented reality (AR) environment, and in such embodiments, the prompt may be a visual prompt of the physical work environment received from one or more cameras.

[0044]

[0060] 9 illustrates a user 900 equipped with an augmented reality device 902 in a work environment 904. The augmented reality device 902 may be one embodiment of a computing device 300. In this example, the augmented reality device 902 is configured as a tablet computer, but may take the form of any suitable device having a camera and a display, such as a phone or a wearable computing device (e.g., a head-mounted display device). The work environment 904 illustrates a portion of a workstation 910 having three similar, but non-identical, substations (911, 912, 913). The user 900 is training the augmented reality device 902 at substation 912.

[0045]

[0061] Computer vision may be used for object recognition and / or image and object classification in the work environment, enabling work in AR. Thus, the user has information about where they are in the context. In one example of Figure 9, substations 911, 912, and 913 have some, but not complete, overlap. Contextual hints and / or visual cues may be provided to guide the worker to the correct one of multiple possibilities for a step in the work protocol.

[0046]

[0062] In some embodiments, the contextual response is therefore a visual response presented via the augmented reality environment. FIG. 10 illustrates, at 1000, an exemplary augmented reality digital work environment 1002 featuring an artificial intelligence assistant 1004. The augmented reality device 902 is illustrated, including a navigation pane 1006. An exemplary work protocol 1008 is presented as a series of steps 1010, 1012, 1014, 1016, and 1018. Step 1010 is indicated as completed with a checkmark. The user is then engaged in step 17.5, a substep of the protocol for securing the wire splice. At 1020, a picture of a portion of the substation 912 is illustrated. At 1022, the user provides a text prompt to highlight the correct wire splice for step 17.5. The AI assistant 1004 retrieves the appropriate information and provides a text response to the user at 1024, highlighting the wire splice as illustrated at 1026.

[0047]

[0063] Thus, the AR subsystem receives input in the form of visual information about the environment. The AI assistant receives contextual information based on the visual input. The digital work environment can then output text, visuals, images, etc., as well as text prompts, to guide the AI assistant's prompts.

[0048]

[0064] As one example, the AI assistant can align the camera image with the retrieved digital CAD image. The user can then ask the AI assistant prompts, such as "where is what" or "how do I perform this action?" Metadata, particularly CAD metadata, can be retrieved from the LLM using the AI assistant. If the user trains the camera on a portion of the built environment, the AI assistant can recognize what the next step in the work instructions is and then instruct the digital work environment to highlight a specific part. The digital work environment can also be instructed to highlight possible NCR issues.

[0049]

[0065] The AI assistant may have the same capabilities as a user, such as clicking on an AR interface or providing contextual hints to objects, giving the AI assistant a human-like interface and capabilities. The digital work environment can further use the AR subsystem to provide information to supervisors (e.g., executives, floor managers). For example, the AI assistant can be used to generate heat maps and other output from worker activity and present this information visually in context.

[0050]

[0066] The trained LLM and AI assistant can be further utilized in generating new work protocols. For example, an AI assistant can be provided to an engineer creating a work protocol. Furthermore, an AI assistant can be utilized to generate low-granularity and high-level work protocols based on training on existing instructions and a knowledge database. For example, an AI assistant can generate instructions for "routing and installing a wire harness." Such instructions can be verified by a human, and then additional contextually relevant content can be searched for that can be incorporated into the new work protocol. In this manner, the LLM and a human user can collaborate to generate usable work instructions. Using the aggregated data on which the LLM was trained, a low-granularity work protocol can be generated for an engineer to use in authoring a new MBI. In other examples, the AI assistant can generate a high-granularity work protocol, which can then be collaborated with a human user. For example, a revised protocol can be high-granular in nature based on the level of detail contained in the original protocol.

[0051]

[0067] 11 shows a flow diagram of an exemplary computer-implemented method 1100 for generating low-granularity work protocols using an artificial intelligence assistant. Method 1100 may be implemented by one or more computing systems, such as computing systems 100 and 300. Method 1100 may be performed in conjunction with method 200, as indicated by the circled B.

[0052]

[0068] At 1110, the method 1100 includes receiving, at an artificial intelligence assistant, a prompt for constructing a new work protocol. The prompt may be generated by a human or may be generated from an application programming interface (API) call from a work protocol construction script. At 1120, the method 1100 includes receiving text and metadata contextually related to the new work protocol. At 1130, the method 1100 includes generating a coarse-grained work protocol for the new work protocol based on the received prompt and the received text and metadata.

[0053]

[0069] As one example, FIG. 12 shows an exemplary scenario 1200 for generating a coarse-grained working protocol. A prompt 1202 requesting a new working protocol is presented to an AI assistant 1204. The AI assistant 1204 may be an example of an AI assistant 142, 304, or 1004. The AI assistant 1204 then receives information about the prompt 1202 from a large-scale language model 1206. The LLM 1206 may be an example of an LLM 134. The AI assistant 1204 then uses the received contextual information to generate a coarse-grained working protocol 1208. In this example, the coarse-grained working protocol 1208 includes four broad steps: step 1 1210, step 2 1212, step 3 1214, and step 4 1216.

[0054]

[0070] In some examples, method 1100 includes receiving input from a user to compile a low-granularity work protocol, searching for contextually relevant content based on the received input, and presenting at least a portion of the retrieved contextually relevant content to the user, such as when the user edits the low-granularity work protocol, indicates further information that can be used to assemble a high-granularity work protocol, etc.

[0055]

[0071] As an example, Figure 13 shows an exemplary scenario 1300 for adapting a low-granularity work protocol. A prompt 1302 for step 1 (1210) is received at the AI assistant 1204. The AI assistant 1204 then receives information about the prompt 1302 from the large-scale language model 1206. The AI assistant 1204 then uses the retrieved contextual information to generate an adapted work protocol 1304. Steps 1-4 are preserved, but step 1 1210 now has additional granularity in the form of step 1.1 1306 and step 1.2 1308.

[0056]

[0072] Generating and adapting work protocols in this manner can enable an AI assistant to inform relevant and contextual information to engineers involved in design and authoring decisions. For example, a long document may only have one table related to a particular work protocol. Rather than forcing the engineer to review the entire document, the AI assistant can point the engineer to the relevant passages and provide context as to why they are relevant.

[0057]

[0073] In some examples, method 1100 may further include receiving input from a user compiling the low-granularity work protocol, searching for contextually relevant content based on the received input, and indicating to the user potential incompatibility issues with the received input based on the searched contextually relevant content. In some examples, the potential incompatibility issues are based on global policies governing stored technical documentation. In some examples, the potential incompatibility issues are based on different work protocols, such as work protocols from different programs. In this manner, the generated work protocol may create fewer problems for manufacturing engineers implementing the work protocol.

[0058]

[0074] The LLM and AI assistant can provide contextual, dynamic, and organic responses by periodically or constantly retraining the LLM. Figure 14 shows a flow diagram of an exemplary computer-implemented method 1400 for contextual retraining of large-scale language models. Method 1400 may be implemented by one or more computing systems, such as computing systems 100 and 300. Method 400 may be performed in conjunction with method 200, as indicated by the circled C.

[0059]

[0075] At 1410, the method 1400 includes generating, in the artificial intelligence assistant, a contextual response to the prompt for the first work protocol. As an example, FIG. 15 shows a scenario 1500 executed on the computing device 300. As shown at 318, a user is performing step 15.7 of the work protocol 308. The user enters a text prompt 1504 asking the AI assistant 304 how tight the clamp should be. At 1506, the AI assistant 304 receives the text and metadata for the user prompt 1504 and provides a text response 1506 (tighten to 25 inch-pounds).

[0060]

[0076] At 1420, the method 1400 includes retraining one or more large-scale language models in response to changes in content of one or more of the technical documents. The one or more technical documents may be editable and / or "live." The retraining may be performed based on updates to one or more static components, such as parts lists, procedures, guide documents, or training resources.

[0061]

[0077] Retraining may be performed in response to inventory changes, non-compliance updates, new or updated documents, human text submissions, work order progress, skilled labor availability, environmental changes, etc. In some embodiments, one or more large-scale language models are retrained in response to a user's progress through a first work protocol. For example, a user may report an issue, update inventory, etc. Completion of a step in a work protocol may affect whether the initiation of other steps from the same or other work protocols is compliant or non-compliant. The user's progress through the first work protocol may be input by the user via the digital work environment and / or by an AI assistant. Technician behavior and / or activities related to the work environment may also be used as training data. Additional context, such as inventory, active or available resource allocations, available technicians, human resource (HR) allocations, etc., may be edited or input on the fly.

[0062]

[0078] At 1430, method 1400 includes generating, in the artificial intelligence assistant, an updated contextual response to the prompt for the first work protocol. As an example, FIG. 16 shows a scenario 1600 executed on computing device 300. As shown at 318, a user is performing step 15.7 of work protocol 308. The user enters a text prompt 1604 asking AI assistant 304 whether the clamp should be tightened to 25 inch-pounds, as in scenario 1600. At 1606, AI assistant 304 searches the text and metadata related to user prompt 1604 and provides text response 1606 (the clamp needs to be removed in a later step (step 17.1) and is tightened by 15 inch-pounds).

[0063]

[0079] In some examples, updated contextual responses to prompts for a second work protocol are generated in the artificial intelligence assistant. For example, changes in an adjacent build may affect the response for the current build. An electrician's completion of a work protocol may enable a structural work protocol to proceed. An engineering work protocol may be updated based on inputs to a manufacturing work protocol. For example, ongoing nonconformance reports during a manufacturing work protocol may result in retraining of the associated LLM, thus triggering changes to a low-granularity engineering work protocol.

[0064]

[0080] In some examples, a work protocol may be performed redundantly for a single build, or multiple adjacent builds may use the same or similar work protocol. Thus, if a problem occurs during the execution of one work protocol, the LLM can be retrained and contextual hints can be presented to other technicians performing the same work protocol. For example, if drilling a particular hole results in hitting something on the other side of the plate, the AI assistant can provide contextual hints to others attempting to drill the same hole, alerting them to the low tolerance. A heat map of such events can be provided as input to the engineering work protocol. This may result in reordering multiple steps, as described with respect to FIG. 8.

[0065]

[0081] The methods and processes described herein may be coupled to the computing system of one or more computing devices. In particular, such methods and processes may be implemented as an executable computer application program, a network-accessible computing service, an API, a library, or a combination of the above and / or other computing resources.

[0066]

[0082] 17 schematically illustrates a simplified representation of an exemplary computing system 1700 configured to provide any or all of the computing functionality described herein. The computing system 1700 may take the form of one or more personal computers, network-accessible server computers, tablet computers, home entertainment computers, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), visual / augmented / mixed reality computing devices, wearable computing devices, Internet of Things (IoT) devices, embedded computing devices, and / or other computing devices.

[0067]

[0083] Computing system 1700 includes a logic subsystem 1710 and a storage subsystem 1720. Computing system 1700 may optionally include a display subsystem 1730, an input subsystem 1740, a communication subsystem 1750, and / or other subsystems not shown in Figure 17. Computing systems 100 and 300 may be examples of computing system 1700.

[0068]

[0084] The logic subsystem 1710 includes one or more physical devices configured to execute instructions. For example, the logic subsystem may be configured to execute instructions. The instructions are part of one or more applications, services, programs, or other logical structures. The logic subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processors of the logic subsystem may be single-core or multi-core, and the instructions executed by the processors may be configured for sequential, parallel, and / or distributed processing. Individual components of the logic subsystem may optionally be distributed across two or more separate devices. These devices may be remotely located and / or configured for coordinated processing. Aspects of the logic subsystem may be virtualized and executed by remotely accessible networked computing devices configured as a cloud computing configuration.

[0069]

[0085] Storage subsystem 1720 includes one or more physical devices configured to temporarily and / or permanently store computer information, such as data and instructions, executable by the logic subsystem. When the storage subsystem includes two or more devices, these devices may be co-located and / or remotely located. Storage subsystem 1720 may include volatile devices, non-volatile devices, dynamic devices, static devices, read / write devices, read-only devices, random access devices, sequential access devices, location-addressable devices, file-addressable devices, and / or content-addressable devices. Storage subsystem 1720 may include removable and / or internal devices. When the logic subsystem executes instructions, the state of storage subsystem 1720 may be transformed, for example, to hold different data.

[0070]

[0086] Logic subsystem 1710 and storage subsystem 1720 may be integrated into one or more hardware logic components, which may include program and application specific integrated circuits (PASICs / ASICs), program and application specific standard products (PSSPs / ASSPs), systems on a chip (SOCs), and complex programmable logic devices (CPLDs).

[0071]

[0087] The logic subsystem and storage subsystem may cooperate to instantiate one or more logic machines. As used herein, the term “machine” collectively refers to a combination of hardware, firmware, software, instructions, and / or any other components that cooperate to provide computer functionality. In other words, a “machine” is never an abstract idea but always has a concrete form. A machine may be instantiated by a single computing device, or a machine may include two or more subcomponents instantiated by two or more different computing devices. In some embodiments, a machine includes a local component (e.g., a software application executed by a computer processor) that cooperates with a remote component (e.g., a cloud computing service provided by a network of server computers). The software and / or other instructions that give a particular machine its functionality may optionally be stored as one or more unexecuted modules on one or more suitable storage devices.

[0072]

[0088] When included, the display subsystem 1730 can be used to present a visual representation of the data maintained by the storage subsystem 1720. This visual representation can take the form of a graphical user interface (GUI). The display subsystem 1730 can include one or more display devices utilizing virtually any type of technology. In some implementations, the display subsystem can include one or more virtual, augmented, or mixed reality displays.

[0073]

[0089] When included, input subsystem 1740 may include or interact with one or more input devices. Input devices may include sensor devices or user input devices. Examples of user input devices include a keyboard, a mouse, a touchscreen, or a game controller. In some embodiments, the input subsystem may include or interact with selected natural user input (NUI) components. Such components may be integrated or peripheral, and input act transmission and / or processing may be handled on-board or off-board. Exemplary NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition.

[0074]

[0090] If included, communications subsystem 1750 may be configured to communicatively couple computing system 1700 with one or more other computing devices. Communications subsystem 1750 may include wired and / or wireless communication devices compatible with one or more different communications protocols. Communications subsystem may be configured for communication over personal, local, and / or wide area networks.

[0075]

[0091] The methods and processes disclosed herein can be configured to give users and / or any other human beings control over all private and / or potentially sensitive data. Whenever data is stored, accessed, and / or processed, it is handled in accordance with privacy and / or security standards. When user data is collected, the user or other interested party can specify how the data should be used and / or stored. All potentially sensitive data can optionally be encrypted and / or anonymized where possible to further protect the user's privacy. The user can specify that portions of the data, metadata, or data statistics / processing results be made public to other parties, for example, for further processing. Private and / or confidential data can be kept completely private, for example, only temporarily decrypted for processing or decrypted for processing on the user device, and otherwise stored in encrypted form. The user can retain and manage the encryption keys for encrypted data. Alternatively or additionally, the user may designate a trusted third party to hold and manage the encryption keys for the encrypted data, for example, to provide the user with access to the data in accordance with an appropriate authentication protocol.

[0076]

[0092] When the methods and processes described herein incorporate ML and / or AI components, the ML and / or AI components can make decisions based at least in part on training of the components on training data. Accordingly, the ML and / or AI components can and should be trained on diverse and representative datasets that include data that is sufficiently relevant to diverse users and / or groups of users. In particular, the training datasets should be comprehensive with respect to various human individuals and groups. As the ML and / or AI components are trained, their performance improves with respect to the user experience of users and / or groups of users.

[0077]

[0093] The ML and / or AI components can be further trained to make decisions to minimize potential bias against human individuals and / or groups. For example, when an AI system is used to evaluate any qualitative and / or quantitative information about human individuals or groups, the AI system can be trained to be invariant to differences between individuals or groups that are not intended to be measured by the qualitative and / or quantitative evaluation, e.g., so that any decisions are not unintentionally influenced by differences between individuals and groups.

[0078]

[0094] The present disclosure is presented by way of example and with reference to the associated drawings. Components, process steps, and other elements that may be substantially the same in one or more of the drawings are identified collectively and described with minimal repetition. It should be noted, however, that collectively identified elements may also differ to some extent. It should be further noted that some of the drawings are schematic and not to scale. Various drawing scales, aspect ratios, and numbers of elements shown in the drawings may be intentionally distorted to more clearly show particular features or relationships.

[0079]

[0095] It will be understood that the configurations and / or approaches described herein are exemplary in nature, and that these specific embodiments or examples should not be considered limiting, as numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various operations illustrated and / or described may be performed in the order illustrated and / or described, in other orders, concurrently, or omitted. Similarly, the order of processes described above may be changed.

[0080]

[0096] Furthermore, the present disclosure includes multiple configurations according to the following clauses.

[0081]

[0097] Article 1. 1. A computer-implemented method comprising: storing technical documentation in a storage device, the technical documentation including at least a work protocol and a technical data file tagged with metadata; extracting text and metadata from the technical documentation; training one or more large scale language models with at least the work protocol, the extracted text, and the metadata; commissioning an artificial intelligence assistant to the one or more large scale language models, the artificial intelligence assistant being configured to at least search the text and the metadata in response to receiving a prompt; providing a digital work environment including an interface for the artificial intelligence assistant; receiving a prompt regarding a first work protocol at the artificial intelligence assistant; searching for text and metadata regarding the first work protocol via the large scale language model based on the received prompt; and providing a contextual response via the digital work environment based on the searched text and the metadata.

[0082]

[0098] Article 2. 10. The computer-implemented method of claim 1, wherein the technical documentation is stored for each program for a plurality of programs, and the one or more large scale language models are trained for each program.

[0083]

[0099] Article 3. 3. The computer-implemented method of claim 1 or 2, wherein the one or more large-scale language models are trained to cross-link the extracted text and metadata for each working protocol.

[0084]

[0100] Article 4. 4. The computer-implemented method of any one of clauses 1 to 3, wherein a plurality of images are extracted from the technical document, and the extracted images are used to train the large-scale language model.

[0085]

[0101] Article 5. 5. The computer-implemented method of any one of clauses 1 to 4, wherein the prompt includes text input by a user.

[0086]

[0102] Article 6. 6. The computer-implemented method of any one of clauses 1 to 5, wherein the prompts include contextual progress through the first work protocol entered via the digital work environment.

[0087]

[0103] Article 7. 7. The computer-implemented method of any one of clauses 1 to 6, wherein the prompts include contextual prompts from the artificial intelligence assistant.

[0088]

[0104] Article 8. 8. The computer-implemented method of any one of clauses 1 to 7, wherein the contextual prompt from the artificial intelligence assistant relates to one or more nonconformance reports.

[0089]

[0105] Article 9. 9. The computer-implemented method of any one of clauses 1 to 8, wherein the artificial intelligence assistant is configured to dynamically reallocate resources based on contextual prompts from the artificial intelligence assistant.

[0090]

[0106] Article 10. 10. The computer-implemented method of any one of clauses 1 to 9, wherein the artificial intelligence assistant is configured to dynamically reorder the first work protocol based on the retrieved text and the metadata.

[0091]

[0107] Article 11. 11. The computer-implemented method of any one of clauses 1 to 10, wherein the retrieved text and metadata includes inventory data related to the first work protocol.

[0092]

[0108] Article 12. 12. The computer-implemented method of any one of clauses 1 to 11, wherein the stored technical documentation includes computer-aided engineering and / or drafting files and associated metadata.

[0093]

[0109] Article 13. 13. The computer-implemented method of any one of clauses 1 to 12, wherein the digital work environment is an augmented reality environment and the prompt is a visual prompt of a physical work environment received from one or more cameras.

[0094]

[0110] Article 14. 14. The computer-implemented method of any one of clauses 1 to 13, wherein the contextual response is a visual response presented via the augmented reality environment.

[0095]

[0111] Article 15. 1. A computer-implemented method comprising: storing technical documentation in a storage device, the technical documentation including at least a work protocol and a technical data file tagged with metadata; extracting text and metadata from the technical documentation; training one or more large scale language models with at least the work protocol, the extracted text, and the metadata; commissioning an artificial intelligence assistant to the one or more large scale language models, the artificial intelligence assistant being configured to at least search the text and the metadata in response to receiving a prompt; and receiving, at the artificial intelligence assistant, a prompt to construct a new work protocol, searching for text and metadata that are contextually relevant to the new work protocol, and generating a coarse-grained work protocol for the new work protocol based on the received prompt and the searched text and metadata.

[0096]

[0112] Article 16. 16. The computer-implemented method of claim 15, further comprising: receiving input from a user editing the low-granularity work protocol; searching for contextually relevant content based on the received input; and presenting at least a portion of the searched contextually relevant content to the user.

[0097]

[0113] Article 17. 17. The computer-implemented method of claim 15 or 16, further comprising receiving input from a user editing the low-granularity work protocol, searching for contextually relevant content based on the received input, and indicating to the user potential incompatibility issues with the received input based on the searched contextually relevant content.

[0098]

[0114] Article 18. 1. A computer-implemented method comprising: storing technical documentation in a storage device, the technical documentation including at least a work protocol and a technical data file tagged with metadata; extracting text and metadata from the technical documentation; training one or more large scale language models with at least the work protocol, the extracted text, and the metadata; commissioning an artificial intelligence assistant to the one or more large scale language models, the artificial intelligence assistant being configured to search at least the text and the metadata in response to receiving a prompt; generating, at the artificial intelligence assistant, contextual responses to prompts associated with a first work protocol; retraining the one or more large scale language models in response to changes in content of one or more of the technical documentation; and generating, at the artificial intelligence assistant, updated contextual responses to the prompts associated with the first work protocol.

[0099]

[0115] Article 19. 19. The computer-implemented method of claim 18, further comprising retraining the one or more large-scale language models in response to a user's progress through the first task protocol.

[0100]

[0116] Article 20. 20. The computer-implemented method of claim 18 or 19, further comprising generating, in the artificial intelligence assistant, updated contextual responses to prompts related to a second work protocol.

[0101]

[0117] It will be understood that the configurations and / or approaches described herein are exemplary in nature, and that these specific embodiments or examples should not be considered limiting, as numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various operations illustrated and / or described may be performed in the order illustrated and / or described, in other orders, concurrently, or omitted. Similarly, the order of processes described above may be changed.

[0102]

[0118] The subject matter of the present disclosure includes all novel and non-obvious combinations and subcombinations of the various processes, systems, and configurations, as well as other features, functions, operations, and / or properties disclosed herein, and any and all equivalents thereof. [Explanation of symbols]

[0103] 100 Exemplary Computing System 102 Storage Devices 104 Technical Documentation 106 CAD files 108 Parts List 110 Current and Projected Inventories 112 Working Protocol 114 Reference Documents 116 Specifications 120 Extracted Data 122 Text Data 124 Metadata 126 Image Data 130 Natural Language Processor 132 Machine Learning Module 134 Large-scale language models 140 Digital Work Environment 142 Artificial Intelligence Assistant 200 Computer-Implemented Methods 210, 220, 230, 240, 250 Method Steps 300 computing devices 302 Digital Work Environment 304 Artificial Intelligence Assistant 306 Navigation Pane 308 Example Work Protocol 310, 312, 314, 316, 318 steps 320 More Information 322 Prompt Box 324 Response Box 400 Computer-Implemented Methods 410, 420, 430 Method steps 500 Scenarios 502 Text Prompt 504 Text Response 600 scenarios 602 Detailed Instructions 604 Prompt Box 606 Contextual Hints 700 Scenarios 704 Prompt Box 706 display 800 Scenarios 804 Prompt Box 806 display 900 users 902 Augmented Reality Devices 904 Working Environment 910 Workstation 911, 912, 914 Substation 1000 Scenarios 1002 Augmented Reality Digital Work Environment 1004 Artificial Intelligence Assistant 1006 Navigation Pane 1008 Exemplary Working Protocol 1010, 1012, 1014, 1016, 1018 steps 1020 Picture 1022 Prompt Box 1024 text responses 1026 emphasis 1100 Computer-Implemented Methods 1110, 1120, 1130 Method steps 1200 Scenarios 1202 prompt 1204 AI Assistant 1206 Large-scale language models 1208 Low-Grained Working Protocol 1300 Scenario 1302 prompt 1304 Adapted Work Protocols Steps 1306 and 1308 1400 Computer-Implemented Methods 1410, 1420, 1430 Method steps 1500 Scenarios 1504 Text Prompt 1506 Text Responses 1600 Scenarios 1604 Text Prompt 1606 Text Responses 1700 Computing System 1710 Logic Subsystem 1720 Storage Subsystem 1730 Display Subsystem 1740 Input Subsystem 1750 Communications Subsystem

Claims

1. A computer-implemented method (400) comprising: Storing (210) technical documentation (104) in a storage device (102), the technical documentation (104) including at least a work protocol (112) and a technical data file tagged with metadata (124); extracting (220) text (122) and metadata (124) from the technical document (104); training (230) one or more large-scale language models (134) with at least the working protocol (112), the extracted text (122), and the metadata (124); delegating (240) an artificial intelligence assistant (142) to the one or more large-scale language models (134), the artificial intelligence assistant (142) being configured to at least search the text (122) and the metadata (124) in response to receiving a prompt (502); providing (250) a digital work environment (140) including an interface for said artificial intelligence assistant (142); receiving (410) a prompt (502) regarding a first work protocol (308) at the artificial intelligence assistant (142); retrieving (420) text (122) and metadata (124) related to the first work protocol (308) through the large language model (134) based on the received prompt (502); and A computer-implemented method (400) comprising providing (430) a contextual response (504) via the digital work environment (140) based on the retrieved text (122) and the metadata (124).

2. 10. The computer-implemented method of claim 1, wherein the technical documentation is stored for each program for multiple programs, and the one or more large-scale language models are trained for each program.

3. 2. The computer-implemented method of claim 1, wherein the one or more large-scale language models are trained to cross-link the extracted text and metadata for each work protocol.

4. 2. The computer-implemented method of claim 1, wherein a plurality of images are extracted from the technical document, and the extracted images are used to train the large-scale language model.

5. The computer-implemented method of claim 1 , wherein the prompt comprises a text input by a user.

6. 2. The computer-implemented method of claim 1, wherein the prompt includes a contextual progression through the first work protocol entered via the digital work environment.

7. The computer-implemented method of claim 1 , wherein the prompt comprises a contextual prompt from the artificial intelligence assistant.

8. 8. The computer-implemented method of claim 7, wherein the contextual prompts from the artificial intelligence assistant relate to one or more non-conformance reports.

9. 8. The computer-implemented method of claim 7, wherein the artificial intelligence assistant is configured to dynamically reallocate resources based on contextual prompts from the artificial intelligence assistant.

10. 10. The computer-implemented method of claim 9, wherein the artificial intelligence assistant is configured to dynamically reorder the first work protocol based on the retrieved text and the metadata.

11. 11. The computer-implemented method of claim 10, wherein the retrieved text and metadata include inventory data related to the first operating protocol.

12. 10. The computer-implemented method of claim 1, wherein the stored technical documentation includes computer-aided engineering and / or drafting files and associated metadata.

13. 10. The computer-implemented method of claim 9, wherein the digital work environment is an augmented reality environment and the prompt is a visual prompt of a physical work environment received from one or more cameras.

14. 11. The computer-implemented method (400) of claim 10, wherein the contextual response is a visual response (1026) presented via the augmented reality environment (1002).

15. A computer-implemented method (1100) comprising: Storing (210) technical documentation (104) in a storage device (102), the technical documentation (104) including at least a work protocol (112) and a technical data file tagged with metadata (124); extracting (220) text (122) and metadata (124) from the technical document (104); training (230) one or more large-scale language models (134) with at least the working protocol (112), the extracted text (122), and the metadata (124); delegating (240) an artificial intelligence assistant (142) to the one or more large-scale language models (134), the artificial intelligence assistant (142) being configured to at least search the text (122) and the metadata (124) in response to receiving a prompt (1202); and In the artificial intelligence assistant (1204), Receiving (1110) a prompt (1202) to build a new work protocol; retrieving (1120) text (122) and metadata (124) contextually relevant to the new work protocol; and A computer-implemented method (1100) comprising generating (1130) a low-granularity working protocol (1208) for the new working protocol based on the received prompt (1202) and the retrieved text (122) and the metadata (124).

16. receiving input (1302) from a user editing the low-granularity working protocol (1208); retrieving contextually relevant content based on the received input (1302); and 16. The computer-implemented method (1100) of claim 15, further comprising presenting at least a portion of the retrieved contextually relevant content to the user.

17. receiving input (1302) from a user editing the low-granularity working protocol (1208); retrieving contextually relevant content based on the received input (1302); and 16. The computer-implemented method (1100) of claim 15, further comprising indicating to a user potential incompatibility issues with the received input (1302) based on the retrieved contextually relevant content.

18. A computer-implemented method (1400) comprising: Storing (210) technical documentation (104) in a storage device (102), the technical documentation (104) including at least a work protocol (112) and a technical data file tagged with metadata (124); extracting (220) text (122) and metadata (124) from the technical document (104); training (230) one or more large-scale language models (134) with at least the working protocol (112), the extracted text (122), and the metadata (124); delegating (240) an artificial intelligence assistant (142) to the one or more large-scale language models (134), the artificial intelligence assistant (142) being configured to at least search the text (122) and the metadata (124) in response to receiving a prompt (1504); generating (1410) in the artificial intelligence assistant (304) a contextual response (1506) to a prompt (1504) associated with a first work protocol (308); retraining (1420) the one or more large-scale language models (134) in response to changes in the content of one or more of the technical documents (104); and A computer-implemented method (1400) including generating, in the artificial intelligence assistant (304), an updated contextual response (1606) to the prompt (1504) associated with the first work protocol (308).

19. 20. The computer-implemented method (1400) of claim 18, further comprising retraining the one or more large language models (134) in response to a user's progress through the first task protocol (308).

20. 20. The computer-implemented method of claim 19, further comprising generating, in the artificial intelligence assistant (304), updated contextual responses to prompts associated with a second work protocol.

Images