Generative ai based augmented reality artifact creation

Abstract

Embodiments include systems and methods for creating augmented reality (AR) artifacts using a tuned generative AI system. The system receives legacy training materials and an AR artifact template, then prompts the generative AI system to create a structured list of AR artifacts based on the legacy training materials and the AR artifact template. The structured list of AR artifacts is transmitted to an AR device, which renders and displays the AR artifacts to the user during task performance. The system supports multiple AR platforms and formats, ensuring compatibility and ease of use across various devices. The generative AI system processes multimodal data, including CAD models, images, text, and tabular data, to generate comprehensive and accurate AR artifacts. The AR device includes cameras and sensors to monitor the user's actions and provide responsive feedback, enhancing the overall user experience and ensuring accurate task execution.

Description

GENERATIVE Al BASED AUGMENTED REALITY ARTIFACT CREATIONBACKGROUND

[0001] The present generally relates to augmented reality (AR) technologies, and more specifically the automated creation of AR artifacts using generative artificial intelligence (Al).

[0002] Modem industrial manufacturing sites are increasingly incorporating more automation on the shop floor but still have evolving tasks for human workers that cannot be automated. Both humans and machines require sets of procedural steps and protocols to follow for success; however, there are still situations where human workers may have difficulties executing tasks and interpreting written or verbal directions safely, efficiently, and without errors. Instructions often require memorization, are missing nuance, originate from multiple unaligned data sources or experiential knowledge, and may not be readily available to human workers in real time, which exacerbates the existing challenges. Advances in using semantically aligned information combined with advanced technologies are especially useful in factories and warehouses to improve safety, increase user productivity, avoid product and equipment damage, and enable workers directly.

[0003] Creating work instructions is still a manual process, with an engineer having to synthesize existing work instructions, integrate the various data on which they rely, understand the hardware and software parameters of the device, and manually author the scene to deploy to the device. The current process is not scalable and is an obstacle in the widespread adoption of advanced technologies in the manufacturing industry. There is a need to develop a scalable automation pipeline which can quickly create work instructions from existing digital instructions with minimal human intervention. Challenges in creating related artifacts arise from the largely manual editing of content and siloed data formation programs or code.SUMMARY

[0004] According to one aspect of the present invention, a computer-implemented method for creating AR artifacts for a task includes receiving one or more legacy training materials for the task; obtaining an AR artifact template; prompting a tuned generativeartificial intelligence system to create a structured list of augmented reality artifacts based on the one or more legacy training materials; transmitting the structured list of augmented reality artifacts to an augmented reality device of a user performing the task; and rendering one or more augmented reality artifacts based on the structured list of augmented reality artifacts via a display of the augmented reality device to the user during performance of the task.

[0005] According to another aspect, a system for creating augmented reality artifacts for a task includes a processing system configured to obtain one or more legacy training materials for the task and an AR artifact template; a tuned generative artificial intelligence system configured to receive one or more legacy training materials for the task and the AR artifact template from the processing system; and an augmented reality device configured to receive a structured list of augmented reality artifacts and render one or more augmented reality artifacts via a display to a user during performance of the task, wherein the tuned generative artificial intelligence system is configured to create the structured list of augmented reality artifacts based on the one or more legacy training materials and the AR artifact template.

[0006] According to yet another aspect, a computer program product, the computer program product having a non-transitory tangible storage device having program code embodied therewith, the program code executable by a processing system to perform operations including receiving one or more legacy training materials for the task; obtaining an AR artifact template; prompting a tuned generative artificial intelligence system to create a structured list of augmented reality artifacts based on the one or more legacy training materials; transmitting the structured list of augmented reality artifacts to an augmented reality device of a user performing the task; and rendering one or more augmented reality artifacts based on the structured list of augmented reality artifacts via a display of the augmented reality device to the user during performance of the task.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 depicts a schematic block diagram illustrating a system for creating augmented reality artifacts using a tuned generative Al system in accordance with an embodiment.

[0008] FIG. 2 shows a block diagram of an augmented reality device in accordance with an embodiment.

[0009] FIG. 3 illustrates a flow chart diagram depicting a method for creating and displaying augmented reality artifacts using a generative Al system in accordance with an embodiment.

[0010] FIG. 4 illustrates a block diagram of a processing system in accordance with an embodiment.

[0011] In the accompanying figures and following detailed description of the disclosed embodiments, the various elements illustrated in the figures are provided with three digit reference numbers. In some instances, the leftmost digits of each reference number corresponds to the figure in which its element is first illustrated.DETAILED DESCRIPTION

[0012] Modem industrial manufacturing sites are increasingly incorporating more automation on the shop floor but still have evolving tasks for human workers that cannot be automated. Both humans and machines require sets of procedural steps and protocols to follow for success. However, there are still situations where human workers may have difficulties executing tasks and interpreting written or verbal directions safely, efficiently, and without errors. Instructions often require memorization, are missing nuance, originate from multiple unaligned data sources or experiential knowledge, and may not be readily available to human workers in real time, which exacerbates the existing challenges.

[0013] Creating work instructions is still a manual process, with an engineer having to synthesize existing work instructions, integrate the various data on which they rely, understand the hardware and software parameters of the device, and manually author the scene to deploy to the device. The current process is not scalable and is an obstacle in the widespread adoption of advanced technologies in the manufacturing industry. Challenges in creating related artifacts arise from the largely manual editing of content and siloed data formation programs or code.

[0014] The disclosed system and method address these challenges by automating the creation of augmented reality (AR) artifacts using a tuned generative Al system. The system receives legacy training materials and an AR artifact template, then prompts the generative Al system to create a structured list of AR artifacts based on the legacy training materials and an AR artifact template. This structured list of AR artifacts is transmitted to an AR device, which renders and displays the AR artifacts to the user during task performance. The system supports multiple AR platforms and formats, ensuring compatibility and ease of use across various devices. In exemplary embodiments, the generative Al system is a large language model (LLM) that is configured to generate multimodal data.

[0015] Referring now to FIG. 1, a schematic block diagram illustrating a system 100 for creating augmented reality artifacts using a tuned generative Al system in accordance with an exemplary embodiment is shown. The system 100 integrates various components to automate the creation and deployment of AR artifacts, ensuring compatibility across multiple AR platforms and formats. In exemplary embodiments, the system 100 includes a processing system that receives legacy training materials and an AR artifact template, processes these inputs through a tuned generative Al system, and generates a structured list of AR artifacts. This structured list of AR artifacts is then transmitted to an AR device, which renders the AR artifacts to the user during task performance.

[0016] In exemplary embodiments, the processing system 102 is responsible for managing the flow of data and instructions within the system 100. The processing system 102 receives legacy training materials 108 and an augmented reality artifact template 110, and coordinates the interaction with the tuned generative Al system 104. The processing system 102 ensures that the data is correctly formatted and prepared for processing by the generative Al system. Additionally, the processing system 102 handles the transmission of the structured list of AR artifacts to the augmented reality device 106, facilitating the rendering of the AR artifacts to the user.

[0017] In exemplary embodiments, the tuned generative Al system 104 is configured to create the structured list of AR artifacts based on the legacy training materials 108 and the augmented reality artifact template 110. The tuned generative Al system 104 leverages advanced machine learning models, including large language models (LLMs) and multimodalgenerative Al, to process and synthesize data from various sources. The generative Al system 104 can handle multimodal data, such as CAD models, images, text, and tabular data, ensuring comprehensive and accurate AR artifact creation. The generative Al system 104 also performs data validation and ensures that the generated AR artifacts are compatible with the target AR platforms and devices.

[0018] In exemplary embodiments, the tuned generative Al system 104 is created by tuning a base, or general purpose, generative Al system. The tuning of a base generative Al system involves adapting the system to accurately process and generate AR artifacts based on a set of training data. This training data includes a set of legacy training materials and corresponding AR artifacts that were created based on the set of legacy training materials. The tuning process ensures that the generative Al system can effectively synthesize and create new AR artifacts that are consistent with the provided training data.

[0019] The set of legacy training materials may include various types and formats of data, such as training manuals, training videos, CAD models, images, textual documents, tabular data, and recordings of task performances. Training manuals are textual documents that provide detailed instructions and procedural steps for performing specific tasks, often including diagrams, flowcharts, and other visual aids to enhance understanding. Training videos are multimedia files that demonstrate the execution of tasks through visual and auditory means, including voice-over explanations, annotations, and step-by-step demonstrations. CAD models are computer-aided design files that provide detailed 3D representations of parts, assemblies, and tools used in the tasks, including geometric data, material properties, and assembly instructions. Images are static visual representations that capture specific aspects of the tasks, such as equipment setup, tool usage, and intermediate steps, often including annotations, labels, and highlights to emphasize important details. Textual documents provide additional context, background information, and reference materials related to the tasks, including technical specifications, safety guidelines, and troubleshooting tips. Tabular data are structured data tables that organize information in rows and columns, including parts lists, tool inventories, and parameter settings required for the tasks. Recordings of task performances are multimedia files that capture real-time execution of tasks by human workers, including video footage, audio commentary, and sensor data to provide a comprehensive view of the task performance.

[0020] During the tuning process, the generative Al system is trained on this diverse set of legacy training materials and their corresponding AR artifacts. The generative Al system learns to recognize patterns, relationships, and dependencies within the data, enabling it to generate accurate and relevant AR artifacts for new tasks. The tuning process involves iterative training, validation, and refinement to ensure that the generative Al system can handle various types and formats of input data and produce high-quality AR artifacts that meet the desired standards.

[0021] In exemplary embodiments, the augmented reality device 106 is the endpoint where the AR artifacts in the structured list of AR artifacts are rendered and displayed to the user. The augmented reality device 106 can include various types of AR hardware, such as head-mounted displays and tablets. The augmented reality device 106 receives the structured list of AR artifacts from the processing system 102 and uses the display capabilities of the augmented reality device 106 to present the AR artifacts to the user during task performance. The augmented reality device 106 may also include cameras and sensors to monitor the user's actions and provide responsive feedback, enhancing the overall user experience and ensuring accurate task execution.

[0022] In exemplary embodiments, the legacy training materials 108 serve as the source data for creating the AR artifacts. These materials can include training manuals, training videos, and recordings of task performances. The legacy training materials 108 provide the necessary information and context for the generative Al system 104 to create accurate and relevant AR artifacts. The processing system 102 ensures that these materials are correctly formatted and prepared for processing by the generative Al system 104.

[0023] The augmented reality artifact template 110 defines the structure and format of the AR artifacts to be created. The augmented reality artifact template 110 provides a standardized framework that the generative Al system 104 uses to generate the structured list of AR artifacts. The augmented reality artifact template 110 ensures that the AR artifacts are compatible with multiple AR platforms and devices, facilitating seamless integration and deployment. The processing system 102 obtains the augmented reality artifact template 110 and uses the augmented reality artifact template 110 in conjunction with the legacy training materials 108 to guide the generative Al system 104 in creating the AR artifacts.

[0024] Referring now to FIG. 2, a block diagram of an augmented reality device 106 in accordance with an exemplary embodiment is shown. In exemplary embodiments, the augmented reality device 106 serves as the endpoint where the AR artifacts in the structured list of AR artifacts are rendered and displayed to the user. The augmented reality device 106 can include various types of AR hardware, such as head-mounted displays and tablets. The augmented reality device 106 receives the structured list of AR artifacts 210 from the processing system 102 and uses the display capabilities of the augmented reality device 106 to present the AR artifacts to the user during task performance. The augmented reality device 106 may also include cameras and sensors to monitor the user's actions and provide responsive feedback, enhancing the overall user experience and ensuring accurate task execution.

[0025] In exemplary embodiments, the processor(s) 202 within the augmented reality device 106 are responsible for executing the instructions and processing the data necessary for rendering and displaying the AR artifacts. The processor(s) 202 handle the computational tasks required to interpret the structured list of AR artifacts 210 and generate the corresponding visual and interactive elements on the display 208. The processor(s) 202 also manage the communication between the augmented reality device 106 and other components of the system 100, ensuring seamless data flow and synchronization.

[0026] In exemplary embodiments, the memory 204 in the augmented reality device 106 stores the structured list of AR artifacts 210, as well as any additional data and instructions needed for rendering and displaying the AR artifacts. The memory 204 provides temporary storage for the data being processed by the processor(s) 202, enabling quick access and efficient execution of tasks. The memory 204 may include various types of storage, such as RAM, flash memory, and other non-volatile storage, to accommodate the different data requirements of the augmented reality device 106.

[0027] In exemplary embodiments, the camera(s) 206 integrated into the augmented reality device 106 capture real-time images and video of the user's environment and actions. The camera(s) 206 provide the necessary visual input for the augmented reality device 106 to overlay the AR artifacts onto the user's view. The camera(s) 206 also enable the augmented reality device 106 to monitor the user's actions and provide responsive feedback, ensuringaccurate task execution and enhancing the overall user experience. The camera(s) 206 may include various types of imaging sensors, such as RGB cameras, depth sensors, and infrared cameras, to capture different aspects of the user's environment and actions.

[0028] In exemplary embodiments, the display 208 of the augmented reality device 106 presents the AR artifacts to the user during task performance. The display 208 can include various types of screens and projection systems, such as LCD, OLED, and microLED displays, to provide high-quality visual output. The display 208 is responsible for rendering the AR artifacts in a way that is easily visible and understandable to the user, ensuring that the user can follow the instructions and perform the task accurately. The display 208 may also support interactive elements, such as touchscreens and gesture recognition, to enable the user to interact with the AR artifacts and receive real-time feedback.

[0029] In exemplary embodiments, the structured list of augmented reality artifacts 210 is a comprehensive and organized collection of AR artifacts generated by the tuned generative Al system 104. The structured list of augmented reality artifacts 210 includes a plurality of steps of the task and at least one AR artifact associated with each of the plurality of steps. The structured list of augmented reality artifacts 210 ensures that the AR artifacts are presented to the user in a logical and sequential manner, guiding the user through the task step-by-step. The structured list of augmented reality artifacts 210 is formatted to be compatible with multiple AR platforms and devices, facilitating seamless integration and deployment across various types of augmented reality hardware.

[0030] In exemplary embodiments, the augmented reality device 106 includes an artifact rendering module 212 that is responsible for interpreting and rendering the AR artifacts from the structured list of augmented reality artifacts 210 onto the display 208 of the AR device. The artifact rendering module 212 ensures that the AR artifacts are accurately and effectively presented to the user, enhancing the overall user experience and task performance. The artifact rendering module 212 interprets the structured list of augmented reality artifacts 210, which includes various types of AR content such as textual overlays, graphical icons, 3D models, and video clips. The artifact rendering module 212 decodes this information and prepares it for rendering on the display 208. Once the AR artifacts areinterpreted, the artifact rendering module 212 renders them onto the display 208. This involves generating the visual and interactive elements of the AR artifacts in a way that is easily visible and understandable to the user. The artifact rendering module 212 ensures that the AR artifacts are displayed in the correct sequence and format, guiding the user through the task step-by-step.

[0031] The artifact rendering module 212 works in conjunction with other components of the AR device, such as the processor(s) 202, memory 204, and camera(s) 206. The processor(s) 202 execute the instructions and process the data necessary for rendering the AR artifacts, while the memory 204 provides temporary storage for the data being processed. The camera(s) 206 capture real-time images and video of the user's environment, which can be used to overlay the AR artifacts onto the user's view. The artifact rendering module 212 can also handle real-time feedback from the user and the environment. For example, if the camera(s) 206 detect changes in the user's actions or the environment, the module can adjust the rendering of the AR artifacts accordingly. This ensures that the AR artifacts remain relevant and useful throughout the task performance.

[0032] In exemplary embodiments, the artifact rendering module 212 is configured to communicate with the generative Al system 104, which can assist the artifact rendering module 212 in rendering of AR artifact. This interaction between the artifact rendering module and the generative Al system can occur in several ways to enhance the accuracy and relevance of the AR artifacts displayed to the user. One example of this interaction is when the artifact rendering module 212 encounters a complex AR artifact that requires additional processing or contextual information. In such cases, the artifact rendering module 212 can send a request to the generative Al system 104, providing details about the specific AR artifact and the context in which it will be displayed. The generative Al system 104 processes this request, leveraging its LLM models and multimodal data processing capabilities to generate a refined and contextually accurate AR artifact. The refined AR artifact is then transmitted back to the artifact rendering module 212, which renders it on the display 208 for the user.

[0033] Another example is when the artifact rendering module 212 needs to dynamically adjust the AR artifacts based on real-time feedback from the user's actions orchanges in the environment. For instance, if the camera(s) 206 detect that the user has deviated from the prescribed steps or if there are changes in the task environment, the artifact rendering module 212 can communicate this information to the generative Al system 104. The generative Al system 104 can then generate corrective guidance or supplementary AR artifacts to help the user get back on track. These updated AR artifacts are sent to the artifact rendering module 212, which promptly renders them on the display 208, ensuring that the user receives timely and relevant assistance.

[0034] Additionally, the artifact rendering module 212 can interact with the generative Al system 104 to handle user-specific customization of AR artifacts. For example, if the user has specific preferences or requirements for how the AR artifacts should be displayed (e.g., preferred language, visual style, or level of detail), the artifact rendering module 212 can communicate these preferences to the generative Al system 104. The Al system can then generate AR artifacts that are tailored to the user's preferences, enhancing the overall user experience. The customized AR artifacts are transmitted back to the artifact rendering module 212 for rendering on the display 208.

[0035] Furthermore, the artifact rendering module 212 can leverage the generative Al system 104 to validate the accuracy and consistency of the AR artifacts before rendering them. For instance, the artifact rendering module 212 can send the generated AR artifacts to the generative Al system 104 for validation, ensuring that they align with the original training materials and task requirements. The Al system can perform cross-referencing and errorchecking, providing feedback to the artifact rendering module 212. If any discrepancies or errors are detected, the Al system can generate corrected AR artifacts, which are then rendered by the artifact rendering module 212.

[0036] In exemplary embodiments, the display 208 of the augmented reality device 106 can present a variety of AR artifacts to the user during task performance, enhancing the user's ability to follow instructions and complete tasks accurately. One example of an AR artifact is a textual overlay that provides step-by-step instructions for the task at hand. This textual information can include detailed descriptions of each step, safety guidelines, and troubleshooting tips, ensuring that the user has all the necessary information readily available. Another example is graphical icons that represent tools, parts, or specific actions required forthe task. These icons can be overlaid onto the user's view to indicate the exact location and orientation of tools and parts, helping the user to correctly position and use them.

[0037] Additionally, the display 208 can show 3D models of parts and assemblies, allowing the user to visualize the components in a detailed and interactive manner. These 3D models can be rotated, zoomed in, and examined from different angles, providing a comprehensive understanding of the parts and their assembly. The display 208 can also present video clips that demonstrate specific steps of the task, offering a visual and auditory guide to the user. These video clips can include voice-over explanations, annotations, and real-time demonstrations, making it easier for the user to grasp complex procedures.

[0038] Furthermore, the display 208 can utilize augmented reality to highlight specific areas or components in the user's environment. For instance, it can use visual markers or color-coded highlights to draw attention to critical parts, connection points, or areas that require special attention. This feature ensures that the user focuses on the right elements and reduces the likelihood of errors. The display 208 may also support interactive elements, such as touchscreens and gesture recognition, enabling the user to interact with the AR artifacts. For example, the user can tap on an icon to get more information, swipe to navigate through steps, or use gestures to manipulate 3D models.

[0039] In exemplary embodiments, the AR artifacts that will be displayed via the display 208 may be extracted from training materials, such as pictures and videos, that list the steps of a task and include illustrations of parts and tools used in each step through a systematic process involving advanced Al and machine learning techniques. The generative Al system 104, integrated with large language models (LLMs), is configured to perform such an extraction process.

[0040] Initially, the training materials, which may include pictures and videos, are fed into the generative Al system 104. These materials provide a visual and auditory representation of the task, detailing each step and illustrating the parts and tools required. The generative Al system 104 processes various data formats with generated multimodal knowledge as inputs using Al techniques such as LLMs, computer vision, natural language processing (NLP), and other multimodal processing to identify and extract relevant information. For pictures, the Al system employs image recognition algorithms to detect andclassify objects within the images. These algorithms can identify parts, tools, and specific actions depicted in the pictures. The system can also recognize text annotations and labels within the images, which provide additional context and details about the task. By analyzing the spatial relationships and visual cues in the pictures, the Al system can determine the sequence of steps and the correct usage of parts and tools. For videos, the Al system utilizes video analysis techniques to process the visual and auditory content. The system can segment the video into individual frames and analyze each frame to identify parts, tools, and actions. Additionally, the Al system can extract audio information, such as voice-over explanations and annotations, to gain a comprehensive understanding of the task. By combining visual and auditory data, the Al system can accurately interpret the sequence of steps and the specific requirements for each step.

[0041] Once the relevant information is extracted from the pictures and videos, the generative Al system 104 synthesizes this data to create AR artifacts. These artifacts include textual overlays, graphical icons, 3D models, and video clips that correspond to each step of the task. The generative Al system 104 ensures that the AR artifacts are logically organized and presented in a sequential manner, guiding the user through the task step-by-step. In exemplary embodiments, the generative Al system 104 also validates the extracted information to ensure accuracy and consistency. This validation process involves crossreferencing the extracted data with the original training materials and checking for any discrepancies or errors. The validated AR artifacts are then formatted to be compatible with multiple AR platforms and devices, ensuring seamless integration and deployment.

[0042] In one embodiment, the structured list of augmented reality (AR) artifacts is formatted to be compatible with multiple AR platforms, including head-mounted displays and tablets, by utilizing a flexible data schema that can be dynamically adjusted based on the specific requirements of each platform. For instance, the system can generate AR artifacts in a JSON format that is easily adaptable to various rendering engines, such as Unity or Unreal Engine, ensuring seamless integration across different devices. In another embodiment, the system can employ a modular approach where the AR artifacts are created as independent modules that can be selectively activated or deactivated depending on the capabilities and constraints of the target AR platform. This modularity allows for efficient resource management and ensures that the AR artifacts are optimized for performance on both high-end and low-end devices. Additionally, the system can include a configuration management component that automatically detects the type of AR device being used and adjusts the display settings, such as resolution and frame rate, to provide an optimal user experience. For example, on a head-mounted display, the system might prioritize immersive 3D models and interactive elements, while on a tablet, the system might focus on clear textual instructions and 2D overlays. Furthermore, the system can support various input methods, such as touch, voice commands, and gesture recognition, to enhance user interaction with the AR artifacts across different platforms. This adaptability ensures that the AR artifacts are not only compatible with multiple AR platforms but also provide a consistent and intuitive user experience regardless of the device being used.

[0043] Referring now to FIG. 3, a flow chart diagram depicting a method 300 for creating and displaying augmented reality artifacts using a generative Al system in accordance with an embodiment is shown. The method 300 is designed to automate the creation of AR artifacts from legacy training materials, ensuring compatibility across multiple AR platforms and devices. As shown at block 302, the first step in the method 300 is to obtain one or more legacy training materials for a task. The legacy training materials may include a variety of types of data that may be in multiple different formats, such as documents, videos, and the like. The processing system 102 is responsible for obtaining these legacy training materials, which may include training manuals, training videos, and recordings of task performances. The legacy training materials provide the necessary information and context for the generative Al system 104 to create accurate and relevant AR artifacts. The processing system 102 ensures that these materials are correctly formatted and prepared for processing by the generative Al system 104.

[0044] The method 300 also includes obtaining an augmented reality artifact template, as shown at block 304. The augmented reality artifact template 110 defines the structure and format of the AR artifacts to be created. The augmented reality artifact template 110 provides a standardized framework that the generative Al system 104 uses to generate the structured list of AR artifacts. The augmented reality artifact template 110 ensures that the AR artifacts are compatible with multiple AR platforms and devices, facilitating seamless integration and deployment. The processing system 102 obtains the augmented reality artifact template 110 and uses the augmented reality artifact template 110 in conjunction withthe legacy training materials 108 to guide the generative Al system 104 in creating the AR artifacts.

[0045] Next, as shown at block 306, the method 300 includes prompting a tuned generative Al system to create a structured list of augmented reality artifacts based on the one or more legacy training materials. The tuned generative Al system 104 is configured to create the structured list of AR artifacts based on the legacy training materials 108 and the augmented reality artifact template 110. The tuned generative Al system 104 leverages advanced machine learning models, including large language models (LLMs) and multimodal generative Al, to process and synthesize data from various sources. The generative Al system 104 can ingest and create multimodal data, such as CAD models, images, text, and tabular data, ensuring comprehensive and accurate AR artifact creation. The generative Al system 104 also performs data validation and ensures that the generated AR artifacts are compatible with the target AR platforms and devices.

[0046] Once the structured list of augmented reality artifacts is created, the next step is to transmit the structured list of augmented reality artifacts to an augmented reality device of a user performing the task, as shown at block 308. The processing system 102 handles the transmission of the structured list of AR artifacts to the augmented reality device 106, facilitating the rendering of the AR artifacts to the user. The structured list of augmented reality artifacts includes a plurality of steps of the task and at least one AR artifact associated with each of the plurality of steps. The structured list of augmented reality artifacts is formatted to be compatible with multiple AR platforms and devices, ensuring seamless integration and deployment across various types of augmented reality hardware.

[0047] The method 300 concludes at block 310 by rendering one or more augmented reality artifacts based on the structured list of augmented reality artifacts via a display of the augmented reality device to the user during performance of the task. The augmented reality device 106 serves as the endpoint where the AR artifacts in the structured list of AR artifacts are rendered and displayed to the user. The augmented reality device 106 can include various types of AR hardware, such as head-mounted displays and tablets. The augmented reality device 106 receives the structured list of AR artifacts from the processing system 102 and uses the display capabilities of the augmented reality device 106 to present the AR artifacts tothe user during task performance. The augmented reality device 106 may also include cameras and sensors to monitor the user's actions and provide responsive feedback, enhancing the overall user experience and ensuring accurate task execution.

[0048] In exemplary embodiments, the augmented reality (AR) device 106 is designed to communicate with the generative Al system 104, either directly or through the processing system 102. This communication enables the AR device to interact dynamically with the generative Al system to generate and display AR artifacts in real-time, enhancing the user's experience and ensuring accurate task execution.

[0049] The AR device 106 is configured to prompt the generative Al system 104 to generate an AR artifact that is suitable for display by the AR device. This process involves the AR device sending a request to the generative Al system, specifying the type of AR artifact needed based on the current task or user interaction. For instance, if the user is performing a specific step in a task, the AR device may request an AR artifact that provides detailed instructions, visual aids, or interactive elements relevant to that step.

[0050] To facilitate this interaction, the AR device 106 may transmit one or more entries from the structured list of augmented reality artifacts to the generative Al system 104. These entries contain information about the task, the required AR artifacts, and any contextual data needed to generate the appropriate content. The generative Al system 104 processes this information and generates the requested AR artifact, ensuring it is tailored to the specific requirements of the task and the capabilities of the AR device.

[0051] Once the generative Al system 104 generates the AR artifact, it is transmitted back to the AR device 106 for display. The AR device then renders the artifact on its display, providing the user with the necessary guidance, visual aids, or interactive elements to perform the task accurately. This real-time interaction between the AR device and the generative Al system ensures that the user receives up-to-date and contextually relevant information, enhancing the overall efficiency and effectiveness of the task execution.

[0052] Additionally, the AR device 106 may include sensors and cameras to monitor the user's actions and provide feedback to the generative Al system 104. This feedback loop allows the Al system to adjust the AR artifacts dynamically based on the user's progress andany deviations from the prescribed steps. For example, if the user makes an error or requires additional assistance, the AR device can prompt the generative Al system to generate corrective guidance or supplementary information to help the user get back on track.

[0053] In an example usage scenario, a worker is tasked with assembling a complex mechanical item on the shop floor. The worker is equipped with an augmented reality (AR) device, such as a head-mounted display, which is connected to a system that uses generative Al to create AR artifacts from legacy training materials, including a detailed assembly manual. As the worker begins the assembly task, the AR device receives a structured list of AR artifacts generated by the tuned generative Al system. These artifacts are based on the assembly manual, which includes step-by-step instructions, illustrations of parts, and tools required for each step. The AR device's display presents the first step of the assembly process to the worker.

[0054] The display shows a textual overlay with detailed instructions for the initial step, such as “Attach the base plate to the main frame using four screws.” Alongside the textual instructions, the display highlights the specific tools needed for this step, such as a screwdriver and the screws. The AR device uses visual markers to draw the worker's attention to the exact location of the tools within their field of view. For instance, the screwdriver and screws are highlighted with a glowing outline or color-coded markers, making it easy for the worker to identify and pick up the correct tools.

[0055] As the worker proceeds with the assembly, the AR device continues to guide them through each step. For example, when the worker needs to attach a component to the main frame, the display shows a 3D model of the component and its correct orientation. The AR device may also provide visual cues, such as arrows or alignment lines, to help the worker position the component accurately. If the worker needs to use a specific torque wrench to tighten bolts, the AR device highlights the torque wrench in their field of view and displays the required torque setting. Throughout the assembly process, the AR device monitors the worker's actions using integrated cameras and sensors. If the worker deviates from the instructions or makes an error, the AR device provides real-time feedback and corrective guidance. For instance, if the worker uses the wrong tool or places a componentincorrectly, the display alerts them with a warning message and visual indicators to correct the mistake.

[0056] In addition to textual instructions and tool highlights, the AR device can display video clips demonstrating specific assembly steps. These video clips include voiceover explanations and annotations, offering a visual and auditory guide to the worker. The worker can interact with the AR artifacts using gestures, such as swiping to navigate through steps or tapping on icons to get more information. By using the AR device, the worker can efficiently and accurately complete the assembly task with minimal errors. The AR artifacts extracted from the assembly manual provide comprehensive and interactive guidance, ensuring that the worker has all the necessary information readily available.

[0057] In one embodiment, the augmented reality (AR) device is equipped with advanced computer vision capabilities, utilizing multiple high-resolution cameras and depth sensors to accurately monitor the user's actions and the surrounding environment. This configuration allows the device to provide real-time feedback and guidance, ensuring precise task execution. The cameras can capture detailed images and videos, which are processed by the device's onboard Al to detect and correct any deviations from the prescribed steps. In another embodiment, the AR device includes a combination of RGB and infrared cameras to enhance the ability to function in various lighting conditions, ensuring consistent performance in both well-lit and dim environments. Additionally, the device may feature a robust gesture recognition system that interprets user gestures to interact with the AR artifacts, such as swiping to navigate through instructions or pinching to zoom in on 3D models. This system can be further enhanced with haptic feedback mechanisms, providing tactile responses to user interactions, thereby improving the overall user experience. In yet another embodiment, the AR device is designed to be lightweight and ergonomically optimized for prolonged use, featuring adjustable head straps and cushioned padding to ensure comfort during extended periods of task performance. The device's display can be a high-definition OLED screen, offering vibrant colors and sharp visuals, which are important for accurately following detailed instructions and visual cues. Furthermore, the AR device may include a built-in audio system with noise-canceling capabilities, allowing the user to receive clear auditory instructions and feedback even in noisy industrial environments.

[0058] In exemplary embodiments, the processing system may also include an automatic scene creator module that interacts with the generative Al system by serving as an isolated processing / translator block that generates data in a repeatable format for use in AR devices. The automatic scene creator module takes input data, which can be in various formats such as JSON, text, CSV, or schema, and processes it to create a unified model, or AR artifact template, that the generative Al system can use. This unified model is derived from variable source data formats, including proprietary processing from source industry settings, and is mapped into a common knowledge structure that the automatic scene creator module can store.

[0059] In exemplary embodiments, pre-processing for data used in AR scene creation may be performed to ensure that the input data is cohesively aligned and useful for generating AR artifacts. The pre-processing includes collecting various types of input data, which can include training manuals, training videos, CAD models, images, textual documents, tabular data, and recordings of task performances. This diverse set of data provides the necessary information and context for creating AR artifacts. The collected data is then formatted into a repeatable format that can be used as input for the generative Al system, such as JSON, text, CSV, schema, or other standardized data structures.

[0060] The pre-processing phase also involves cleaning and integrating the data from multiple unaligned data sources, addressing issues such as missing data, inconsistencies, and redundancies. The data is combined into a cohesive dataset that accurately represents the task and its associated steps, parts, and tools. During pre-processing, metadata and additional information regarding the AR device, engine, and target hardware settings are extracted, providing context and parameters essential for generating AR artifacts compatible with the target platforms and devices. The pre-processed data undergoes validation to ensure its accuracy and consistency, involving cross-referencing the data with the original sources and checking for any discrepancies or errors. The validated data is then ready for use by the generative Al system.

[0061] The pre-processed data is mapped into a common knowledge structure that the Automatic Scene Creator (ASC) can store, creating a unified model that serves as the input for the generative Al system. This unified model enables the Al system to create AR artifactsbased on the standardized data format. The generative Al system is then tuned using the pre- processed data and the unified model, involving training the Al system on the diverse set of input data to recognize patterns, relationships, and dependencies within the data. The tuned Al system can generate accurate and relevant AR artifacts for new tasks.

[0062] In exemplary embodiments, metadata extraction is performed during the preprocessing phase, where additional information regarding the AR device, artifact rending module, and target hardware settings is extracted. This metadata provides context and parameters that are essential for generating AR artifacts that are compatible with the target platforms and devices. During metadata extraction, the processing system identifies and collects various types of metadata from the input data sources. This metadata can include information about the AR device, such as its model, specifications, and capabilities. For example, the system may extract details about the display resolution, field of view, and supported input methods (e.g., touch, voice commands, gesture recognition) of the AR device. This information is critical for ensuring that the generated AR artifacts are optimized for the specific hardware being used.

[0063] Additionally, the processing system extracts metadata related to the artifact rending module, which is the software platform responsible for rendering and displaying the AR content. This metadata can include the version of the artifact rending module, supported file formats, rendering capabilities, and any specific requirements or constraints imposed by the engine. By understanding the capabilities and limitations of the artifact rending module, the system can generate AR artifacts that are compatible and perform well within the given software environment. Furthermore, the processing system gathers metadata about the target hardware settings, which can include information about the processing power, memory, and storage capacity of the device. This metadata helps the system to tailor the AR artifacts to the performance characteristics of the target hardware, ensuring that the content is rendered smoothly and efficiently. For instance, if the target hardware has limited processing power, the system may generate simplified AR artifacts that require less computational resources.

[0064] By following these pre-processing steps, the processing system ensures that the input data is cohesively aligned, accurately represented, and ready for use in AR scene creation. This pre-processing phase is essential for generating high-quality AR artifacts thatare compatible with multiple platforms and devices, facilitating seamless integration and deployment in various industrial settings.

[0065] FIG. 4 illustrates an example of a processing system 400 that can be used to implement the computer-based components described herein. The processing system 400 includes an exemplary computing device (“computer”) 402 configured for performing various aspects of the operations described herein in accordance aspects of the invention. In addition to computer 402, exemplary processing system 400 includes network 414, which connects computer 402 to additional systems (not depicted) and can include one or more wide area networks (WANs) and / or local area networks (LANs) such as the Internet, intranet(s), and / or wireless communication network(s). Computer 402 and additional system are in communication via network 414, e.g., to communicate data between them.

[0066] Exemplary computer 402 includes processor cores 404, main memory (“memory”) 410, and input / output component(s) 44, which are in communication via bus 403. Processor cores 404 includes cache memory (“cache”) 406 and controls 408, which include branch prediction structures and associated search, hit, detect and update logic, which will be described in more detail below. Cache 406 can include multiple cache levels (not depicted) that are on or off-chip from processor 404. Memory 410 can include various data stored therein, e.g., instructions, software, routines, etc., which, e.g., can be transferred to / from cache 406 by controls 408 for execution by processor 404. Input / output component s) 412 can include one or more components that facilitate local and / or remote input / output operations to / from computer 402, such as a display, keyboard, modem, network adapter, etc. (not depicted).

[0067] A cloud computing system 420 is in wired or wireless electronic communication with the processing system 400. The cloud computing system 420 can supplement, support or replace some or all of the functionality (in any combination) of the processing system 400. Additionally, some or all of the functionality of the processing system 400 can be implemented as a node of the cloud computing system 420.

[0068] For the sake of brevity, conventional techniques related to making and using the disclosed embodiments may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs to implement thevarious technical features described herein are well known. Accordingly, in the interest of brevity, many conventional implementation details are only mentioned briefly or are omitted entirely without providing the well-known system and / or process details.

[0069] The various components / modules / models of the systems illustrated herein are depicted separately for ease of illustration and explanation. In embodiments of the invention, the functions performed by the various components / modules / models can be distributed differently than shown without departing from the scope of the various embodiments of the invention describe herein unless it is specifically stated otherwise.

[0070] Aspects of the invention can be embodied as a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0071] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, element components, and / or groups thereof.

[0072] While the present invention has been described with reference to an exemplary embodiment or embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the present invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from the essential scope thereof. Therefore, it is intended that the present invention not be limited to the particular embodiment disclosed as the best mode contemplated for carrying out this present invention, but that the present invention will include all embodiments falling within the scope of the claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for creating augmented reality (AR) artifacts for a task, the method comprising: receiving one or more legacy training materials for the task; obtaining an AR artifact template; prompting a tuned generative artificial intelligence (Al) system to create a structured list of augmented reality artifacts based on the one or more legacy training materials; transmitting the structured list of augmented reality artifacts to an augmented reality device of a user performing the task; and rendering one or more augmented reality artifacts based on the structured list of augmented reality artifacts via a display of the augmented reality device to the user during performance of the task.

2. The computer-implemented method of claim 1, wherein the one or more legacy training materials includes one or more a training manual for the task, a training video for the task, video of a performance of the task.

3. The computer-implemented method of claim 1, wherein the tuned generative artificial intelligence (Al) system is configured to process multimodal data including one or more CAD models, images, text, and tabular data.

4. The computer-implemented method of claim 1, wherein the structured list of augmented reality artifacts includes a plurality of steps of the task and at least one AR artifact associated with each of the plurality of steps of the task.

5. The computer-implemented method of claim 1, wherein the tuned generative Al system is configured to extract a plurality of steps of the task, parts and tools associatedwith the steps of the task and to generate AR artifacts for each of the plurality of steps of the task.

6. The computer-implemented method of claim 5, wherein the AR artifacts include one or more of text, icons, and a video to be displayed to the user during performance of the task.

7. The computer-implemented method of claim 1, wherein the structured list of augmented reality artifacts has a format that is compatible with multiple AR platforms including head mounted displays and tablets.

8. The computer-implemented method of claim 7, wherein the format of the structured list of augmented reality artifacts is determined based on the AR artifact template.

9. The computer-implemented method of claim 1, wherein the augmented reality device is configured to monitor the performance of the task by the user via one or more cameras and to responsive render the one or more augmented reality artifacts.

10. A system for creating augmented reality (AR) artifacts for a task, comprising: a processing system configured to obtain one or more legacy training materials for the task and an AR artifact template; a tuned generative artificial intelligence (Al) system configured to receive one or more legacy training materials for the task and the AR artifact template from the processing system; and an augmented reality device configured to receive a structured list of augmented reality artifacts and render one or more augmented reality artifacts via a display to a user during performance of the task, wherein the tuned generative artificial intelligence (Al) system is configured to create the structured list of augmented reality artifacts based on the one or more legacy training materials and the AR artifact template.

11. The system of claim 10, wherein the one or more legacy training materials includes one or more a training manual for the task, a training video for the task, video of a performance of the task.

12. The system of claim 10, wherein the tuned generative artificial intelligence (Al) system is configured to process multimodal data including one or more CAD models, images, text, and tabular data.

13. The system of claim 10, wherein the structured list of augmented reality artifacts includes a plurality of steps of the task and at least one AR artifact associated with each of the plurality of steps of the task.

14. The system of claim 10, wherein the tuned generative Al system is configured to extract a plurality of steps of the task, parts and tools associated with the steps of the task and to generate AR artifacts for each of the plurality of steps of the task.

15. The system of claim 14, wherein the AR artifacts include one or more of text, icons, and a video to be displayed to the user during performance of the task.

16. The system of claim 10, wherein the structured list of augmented reality artifacts has a format that is compatible with multiple AR platforms including head mounted displays and tablets.

17. The system of claim 16, wherein the format of the structured list of augmented reality artifacts is determined based on the AR artifact template.

18. The system of claim 10, wherein the augmented reality device is configured to monitor the performance of the task by the user via one or more cameras and to responsive render the one or more augmented reality artifacts.

19. A computer program product, the computer program product comprising a non-transitory tangible storage device having program code embodied therewith, the program code executable by a processing system to perform operations comprising: receiving one or more legacy training materials for performing a task;obtaining an AR artifact template; prompting a tuned generative artificial intelligence (Al) system to create a structured list of augmented reality artifacts based on the one or more legacy training materials; transmitting the structured list of augmented reality artifacts to an augmented reality device of a user performing the task; and rendering one or more augmented reality artifacts based on the structured list of augmented reality artifacts via a display of the augmented reality device to the user during performance of the task.

20. The computer program product of claim 19, wherein the one or more legacy training materials includes one or more a training manual for the task, a training video for the task, video of a performance of the task.

Images