Guided visual diagnostic system and method for equipment

Abstract

The present invention provides a guided visual diagnostic system and method for use with medical devices. [Solution] When a technician discovers and reports a device malfunction, the diagnostic system automatically generates a diagnostic plan detailing the necessary parts, areas of interest, and actions to be taken. The perceptual module analyzes the technician's actions and recommends the next steps, guiding the technician using augmented reality (AR) indicators. The planning component creates the diagnostic plan using a knowledge graph (KG) and a large-scale language model (LLM), eliminating the need for extensive data labeling or model training. The tracking component enhances 3D detection using perceptual sensors and a 2D nested object detection model. The guiding component simplifies the process for novice technicians by integrating 2D models and AR interactions to ensure efficient and accurate diagnosis.

Description

[Technical Field]

[0001] This disclosure generally pertains to computer vision applications, and more specifically to guided visual diagnostic systems and methods for vision-based monitoring applications such as failure inspection. [Background technology]

[0002] As the success of AI technology in computer vision is demonstrated, many industries are beginning to implement vision-based systems to automate processes such as inspection, quality control, and equipment monitoring. Existing solutions are limited to acquiring images and having AI-based models identify objects and defects. However, little consideration has been given to automating visual root cause analysis for on-site diagnosis. Current support service tools are heavily focused on remote assistance, and remote experts still need to guide on-site technicians. These remote systems are often ineffective due to network latency, which limits interactivity, coordination, and collaboration. Automated solutions for effective support services, such as fault diagnosis and repair, are crucial for improving customer satisfaction, retention, and loyalty. However, providing consistent, high-quality, and timely support is a challenging task. In practice, customer support typically requires technicians to perform on-site diagnosis, but support quality is often negatively impacted by limited availability of skilled technicians, high turnover rates, and minimal automation tools.

[0003] When building automated guidance solutions, three main challenges arise. First, a knowledge base is necessary to provide reasoning and scalability, but traditional methods require vast amounts of data and labels. Second, understanding the scene is crucial to ensuring the quality of visual guidance, but existing methods are insufficient to handle changing customer environments. Third, advanced user interfaces need to be intuitive and useful, but AR using 3D methods is slow for rich human interaction. The recent trend of exploring new methods for industrial services is influenced by breakthroughs in large language models (LLMs) and augmented / virtual reality, which offer new ways and opportunities for enhancement.

[0004] The embodiments of this specification, advantageously, unlike conventional remote solutions, provide a smart system that generates diagnostic plans for troubleshooting and guides field technicians regardless of their experience level. This makes the diagnostic process more efficient and effective, reduces the need for highly skilled technicians, and ensures high-quality support at the customer site.

[0005] Visual inspection systems that analyze images to detect defects in various types of equipment are widely incorporated. Deep learning-based methods have demonstrated acceptable performance in automating business pipelines related to quality control and streamlining maintenance and repair. Common applications of visual inspection systems relate to surface defects such as roads, welds, building cracks, and damage to railways or vehicles. Common deep learning models used in these systems include classifiers such as ResNet, MobileNet, and Swin Transformer, real-time object detectors such as the Yolo series, and instance segmentation models such as MaskRCNN, DeepLab, and Yolov7s. These detectors are primarily trained to detect defects or anomalies from isolated images. However, little consideration has been given to identifying the root cause of failures from multiple correlated views and objects.

[0006] Common support systems such as chatbots are widely used in customer support. However, visual support systems remain limited to messages and action instructions. Advanced support systems typically include some form of knowledge base (KB) to provide information on a wide range of support tasks. However, traditional KB development requires large amounts of data, making the KB development process slow and complex. The main task of building a knowledge graph focuses on entity and relationship extraction (RE) to enable reasoning based on graph semantics. Common approaches to entity and relationship extraction include custom seq2seq models and REBEL. With the emergence of LLMs such as ChatGPT®, entity tagging and relationship extraction are being re-examined to evaluate the possible performance of domain-specific knowledge. It has been demonstrated that high performance can be achieved with minimal training data in relationship extraction. Embodiments herein use LLMs to extract industrial entities such as parts, components, systems, and diagnostic tasks.

[0007] Scene understanding has received considerable attention, particularly in fields such as autonomous robotics and driving systems. Scene understanding involves analyzing and interpreting the functional context and semantics of objects in relation to 3D space and layout. Methods of scene understanding can be classified into object-oriented and space-oriented approaches. A graph representation of a scene captures objects and their relationships within a physical layout, such as a room or garden. Traditional 2D and 3D approaches suffer from low accuracy and stability in situations involving the transparency and high reflectivity of objects. Our focus is on objects of interest (parts) and associated key objects, representing spatial relationships defined as an area of ​​interest.

[0008] With the proliferation of augmented reality (AR) libraries such as Apple ARKit (Apple and ARKit are registered trademarks, respectively), Google ARCore (Google and ARCore are registered trademarks, respectively), and WebAR, several AR approaches have been designed for navigation guidance, assembly tracking, and repair assistance. However, diagnostic guidance remains minimal, with technicians simply taking makeshift images for reference or documentation. Most AR applications integrate overlay annotations for user interaction, using either 1) physical markers (e.g., lines on the floor or barcodes on objects) or 2) 3D object recognition to identify target objects. To identify objects, these solutions involve three steps: First, the 3D scanner learns the environment. Second, a 3D object representation is generated, such as a CAD object or point cloud. Third, the application recognizes the scanned object using the 3D representation. However, the accuracy of the 3D model is still heavily influenced by environmental differences (background, area, lighting, layout, etc.). In practice, creating a 3D environment using conventional methods is difficult, and in some cases, impossible. Furthermore, 3D recognition remains a challenge for real-time systems due to latency (over 1 second). Unlike such approaches, the embodiments herein utilize a 2D object recognition model that surpasses 3D detection accuracy and response time. [Overview of the Initiative]

[0009] The systems and methods described herein enable a process for diagnosing equipment failures. Upon reporting a fault discovered by a technician, various embodiments automatically generate a diagnostic plan including necessary parts, areas of interest, diagnostic tasks, and actions to be taken. Using this plan, a perceptual module analyzes and tracks the technician's actions to recommend the next steps, while guiding the technician using AR indicators. In embodiments, this is achieved through three main components: planning, tracking, and guiding. The planning component automates the creation of the diagnostic plan by querying a knowledge graph (KG) built using an LLM to accelerate the extraction of parts, components, tasks, and relationships from manuals. The system leverages the LLM with few-shot prompting and requires neither extensive data labels nor model training. The tracking component is designed to enhance 3D detection by using perceptual sensors in conjunction with a 2D nested object detection model. The guiding component reduces process complexity for novice technicians by integrating 2D models and AR interactions to ensure an efficient and accurate diagnostic process.

[0010] In some aspects of this disclosure, a guided visual diagnostic method for equipment failures includes using an LLM to construct a KG used to generate a diagnostic plan in response to receiving information about a failure related to the equipment. The KG is constructed by a KG builder. The KG builder reduces the need for manual data labeling or model training by extracting information from one or more documents and constructing a graph that includes at least one of the parts, components, spatial areas, or diagnostic tasks related to the text source. The visual diagnostic method also includes using a diagnostic plan and real-time AR indicators to guide the user through the diagnostic process. The AR is configured to overlay visual indicators on visual data captured by the user. The visual diagnostic method also includes using an image builder to analyze and track user actions to enhance 3D scene understanding. The image builder constructs an image-based model for detecting areas and parts for scene understanding, and the image builder uses a perception module that includes one or more perception sensors. The visual diagnostic method also includes generating an indicator database containing overlay annotations extracted from at least one of 2D images, paths, markers, or messages; utilizing a 2D nested object detection model and 3D positioning to determine at least one of the object size, orientation, position, or coverage of objects in a scene based on a diagnostic checklist; processing motion sensor data to ensure correct orientation and alignment during diagnosis; and using an AR tracking and interaction module to manage visual indicators and guide the user along the diagnostic checklist until all checkpoints on the checklist are met.

[0011] In some aspects, the text source may include manuals, parts lists, texts, or fault reports.

[0012] In some embodiments, 2D nested object detection models use training datasets containing a relatively small set of training samples to improve detection accuracy.

[0013] In some embodiments, the perception module provides feedback and recommendations based on the analysis and tracking of actions, for example, in real time, and the perception sensor includes a camera and / or motion sensor.

[0014] In some embodiments, the initial calibration involves mapping the user and device into 3D space using 2D images and AR tracking, where each of the 2D images may include at least a portion of the object of interest.

[0015] In some embodiments, a non-temporary computer-readable medium for storing instructions for performing a process, the instructions include a description for causing a computer to use an LLM to construct a KG used to generate a diagnostic plan, in response to receiving information about a fault related to the equipment. The KG is constructed by a KG builder. The KG builder reduces the need for manual data labeling or model training by constructing a graph containing at least one of the parts, components, spatial areas, or diagnostic tasks related to the text source, by extracting information from one or more documents using an LLM. The instructions include a description for causing a computer to use a diagnostic plan and real-time AR indicators to guide the user through the diagnostic process. The AR is configured to overlay visual indicators on visual data captured by the user. The instructions also include a description for causing a computer to use an image builder to analyze and track user actions to enhance 3D scene understanding. The image builder constructs an image-based model for detecting areas and parts for scene understanding, and the image builder uses a perception module that includes one or more perception sensors. The instruction includes a description to cause a computer to generate an indicator database containing overlay annotations extracted from at least one of 2D images, paths, markers, or messages; a description to cause a computer to utilize a 2D nested object detection model and 3D positioning to determine at least one of the object size, orientation, position, or coverage of objects in a scene based on a diagnostic checklist, and to process motion sensor data to ensure correct orientation and alignment during diagnosis; and a description to cause a computer to use an AR tracking and interaction module to manage visual indicators and guide the user along the diagnostic checklist until all checkpoints on the diagnostic checklist are met.A computer program having the command may be implemented. That is, the computer program can cause a computer to execute the command.

[0016] In some embodiments, the text source includes a manual, a parts list, text, or a trouble report.

[0017] In some embodiments, the 2D nested object detection model uses a training dataset that includes a relatively small set of training samples to improve detection accuracy.

[0018] In some embodiments, the perception module provides feedback and recommendations based on, for example, real-time analysis and tracking of actions, and the perception sensors include cameras and / or motion sensors.

[0019] In some embodiments, the initial calibration includes mapping the user and the device into a 3D space using 2D images and AR tracking, and each of the 2D images may include at least a portion of an object of interest.

[0020] In some embodiments, the technology described herein relates to a device comprising a processor. The processor uses an LLM to construct a KG used to generate a diagnostic plan in response to receiving information about a fault related to the device. The LLM reduces the need for manual data labeling or model training by realizing the KG. The processor uses the diagnostic plan and real-time AR indicators to guide the user through the diagnostic process. AR is configured to overlay visual indicators on visual data captured by the user. The processor uses an image constructor to analyze and track user actions to enhance 3D scene understanding. The image constructor constructs an image-based model for detecting areas and parts for scene understanding, and the image constructor uses a perception module including one or more perception sensors. The processor generates an indicator database containing overlay annotations extracted from at least one of 2D images, paths, markers, or messages. The processor utilizes a 2D nested object detection model and 3D positioning to determine at least one of the object size, orientation, position, or coverage of objects in a scene based on a diagnostic checklist, and processes motion sensor data to ensure correct orientation and alignment during diagnosis. The processor uses an AR tracking and interaction module to manage visual indicators and guide the user along the diagnostic checklist until all checkpoints on the checklist are met.

[0021] Aspects of this disclosure may include a system, which comprises means for performing steps including using an LLM to construct a KG used to generate a diagnostic plan in response to receiving information about a fault related to equipment. The LLM reduces the need for manual data labeling or model training by realizing the KG. The system comprises means for using the diagnostic plan and real-time AR indicators to guide the user through the diagnostic process. The AR is configured to overlay visual indicators on visual data captured by the user. The system comprises means for using an image constructor to analyze and track user actions to enhance 3D scene understanding. The image constructor constructs an image-based model for detecting areas and parts for scene understanding, and the image constructor uses a perception module including one or more perception sensors. The system comprises means for generating an indicator database including overlay annotations extracted from at least one of 2D images, paths, markers, or messages; means for utilizing a 2D nested object detection model and 3D positioning to determine at least one of the object size, orientation, position, or coverage of objects in a scene based on a diagnostic checklist, and for processing motion sensor data to ensure correct orientation and alignment during diagnosis; and means for using an AR tracking and interaction module to manage visual indicators and guide the user along the diagnostic checklist until all checkpoints on the diagnostic checklist are met. [Brief explanation of the drawing]

[0022] [Figure 1] This figure shows exemplary areas of interest and viewpoints in various embodiments of the present disclosure. [Figure 2] This is an exemplary system flow according to various embodiments of the present disclosure. [Figure 3]This figure shows the core technology architecture and interaction for AI and AR-guided diagnostics for systems for guided diagnostics according to various embodiments of the present disclosure. [Figure 4] This figure shows an indicator for guiding engineers according to various embodiments of the present disclosure. [Figure 5] This is an exemplary workflow illustrating a guided diagnostic process according to various embodiments of the present disclosure. [Figure 6] This is an exemplary workflow illustrating a guided diagnostic process according to various embodiments of the present disclosure. [Figure 7] This figure shows a knowledge base building module and a plan generator according to various embodiments of the present disclosure. [Figure 8] This figure shows exemplary expected outputs according to various embodiments of the present disclosure. [Figure 9] This figure shows exemplary viewpoint graphs according to various embodiments of the present disclosure. [Figure 10] This is an exemplary workflow for model training and inference according to various embodiments of the present disclosure. [Figure 11] This figure shows exemplary object labels for a 2D nested object model according to various embodiments of the present disclosure. [Figure 12] This figure shows exemplary 2D nested object detection according to various embodiments of the present disclosure. [Figure 13] This figure shows an example of detected rotation according to various embodiments of the present disclosure. [Figure 14] This flowchart shows a process for guided visual diagnosis of equipment failure according to various embodiments of the present disclosure. [Figure 15] This figure shows an exemplary computing environment using exemplary computer devices according to various embodiments of the present disclosure. [Modes for carrying out the invention]

[0023] The following detailed description provides further information regarding the figures and exemplary implementations of this application. For clarity, the reference numerals and descriptions of overlapping elements between figures have been omitted. The terminology used throughout this description is provided as an example and is not intended to be limiting. For example, the use of the term “automatic” may include fully automatic or semi-automatic implementations that include user or administrator control over specific aspects of the implementation, depending on the desired implementation of one of the persons skilled in the art practicing the implementation of this application. The selection may be made by the user via a user interface or other input means, or implemented through a desired algorithm. The exemplary implementations described herein may be used individually or in combination, and the functions of the exemplary implementations may be implemented by any means depending on the desired implementation.

[0024] In this document, the terms “engineer” and “inspector” are used interchangeably. Similarly, the terms “diagnostic plan,” “checklist plan,” “checklist,” and “plan” may be used interchangeably. Headings are used for structural purposes only and are not used to limit the scope of this description or the claims. Each reference / document cited in this patent document is incorporated herein by reference in its entirety.

[0025] overview

[0026] Embodiments of this specification extend the previous framework to provide a guide for diagnostic cases. Technicians need to know which parts to check to identify the root cause. Due to limited space or the need for resolution when capturing visual input, it is necessary to inspect the equipment from various areas and perspectives, including close-up views of specific areas and parts within the viewpoint. In such embodiments, multipoint inspection is divided into small areas of interest (e.g., controller area or refrigeration) within the viewpoint of the equipment (front or left side). Technicians are required to evaluate these areas of interest by collecting visual records that serve as evidence of the equipment's condition. This process involves determining the condition of parts within these areas of interest and recommending repairs as necessary.

[0027] In this situation, the checklist perspective plan V is: Perspective of Interest V = {v j ,...,v n Area of ​​interest within {α} = {α} ij ,...,α mn Redefined as a checklist of}, where AoI∈V and α ij is a partition j of viewpoint v, where i ≤ n and j ≤ m, where n is the number of viewpoints to be examined and m is the number of areas of interest for a given viewpoint j. The new plan is represented as an area of ​​interest plan, where AoI is defined as a complete viewpoint V or a subset of V. Figure 1 shows exemplary areas of interest and viewpoints according to various embodiments of this disclosure. As shown, the complete viewpoint V 102 includes areas of interest AoI 104-108.

[0028] Definition of the problem

[0029] In the context of support services, visual device diagnosis can be defined as the process necessary to find the root cause of a fault reported by a customer. Since a fault is often just a symptom of a main problem, this process requires a more comprehensive visual assessment and troubleshooting of the device. Through this detailed process, a technician can determine appropriate repair recommendations or fault mitigation measures. Therefore, the quality of this work depends greatly on the technician's proficiency. If the technician is not sufficiently proficient, they may need to consult a manual to identify appropriate troubleshooting steps and parts to check, which can be inaccurate and time-consuming.

[0030] According to embodiments of the present specification, a technician can perform a diagnosis regardless of their proficiency, the type of device, or the complexity of the fault. Given a device E and a reported fault f, the following definitions apply.

[0031] Point of interest: A device E having a three-dimensional (3D) structure is considered to be composed of a set of viewpoints V = {v1,..., v n}, where n > 0. V represents the spatial plane of the device and is used for physical navigation. For example, the front, back, sides, etc.

[0032] Area of interest: Given a viewpoint v i , an area A = {a 1,i , …, a m,n} is composed, where m > 0 and i > 0. A represents a set of mutually exclusive divisions within the viewpoint v i . For example, (upper, front), (center, front), (lower, front).

[0033] Part-component-system: A part P = {p1, …, p k}, where k > 0. P represents the available parts arranged in a specific area a j,i and viewpoint v i (spatial plane). For example, filters, holders, handles, etc.

[0034] Diagnostic task: T={t1,…,t k Let} be the case, where k≧0 represents the tasks that the technician should perform to identify the root cause of the failure. For example, opening a door, turning it on, moving up, etc.

[0035] Diagnostic plan: D={s1,…,s n Let} be a series of steps s for diagnosing a fault f, where n>0 is a series of steps s n This represents each step s i is s i =(p i ,t i ) is defined as, where part p i This is a part that should be checked, i This is a task to be performed, and here it can be expressed as follows: Each step s i This includes 1) visual evaluation, 2) requirements verification, and 3) part p i This includes automatic recording (video or images). For example, steps s i In this scenario, the technician needs to 1) open the door and evaluate the engine parts visible in the central area of ​​the front view, and 2) meet the size, orientation, and coverage requirements to trigger automatic recording of the parts.

number

[0036] Diagnostic requirements: R={r1,…,r k The} represents the expected requirements for visual documentation that should be met during the diagnosis, such as size, coverage, and orientation.

[0037] Guided Visual Diagnosis (GD): GD represents a system-assisted process based on the diagnostic plan D and requirements R. The system uses text messages, AR instructions, and AR navigation to assess possible causes of the fault f.

[0038] Given a reported fault f for a given device E, we define a general guideline for maximizing the process quality Q of the parts evaluated and recorded in a given diagnostic plan D. Quality is defined as a function of completeness and consistency. Completeness is defined as the completion of the diagnostic plan generated by querying a knowledge graph, where D = {s1,…,s} for fault f. n} and steps i =(p i ,t i ) and p i Area A j,i , viewpoint v i This can be expressed using the formula 1. Consistency is defined as the similarity between the expected visual record (image or video) and the actual capture by the technician. Similarity is expressed by comparing the visual requirement R and the observed state O, where C = sim(image, R, O), and R and O include size, orientation, coverage, etc.

[0039] Challenges and Solutions

[0040] Existing challenges include: (1) Complex fault diagnosis methods: Troubleshooting requires a detailed fault tree table that includes multiple parts and components specific to the product type and their failures. Diagnosing faults requires skilled technicians and can vary depending on the equipment's uptime. (2) Translucent and highly reflective areas: Industrial equipment often includes translucent materials (such as glass) or highly reflective surfaces (such as aluminum). These materials reflect surrounding objects, complicating scene understanding. (3) Visual variability of equipment status at customer sites: Equipment status encounters complex conditions due to the installation environment and daily use. (4) Limited indoor inspection areas: Machinery is installed in confined spaces, and the limited space for technicians to move around increases the likelihood of obstructions that prevent a complete 360-degree inspection. (5) Fast response time methods: User-interactive systems require rapid response times, for example, less than one second, to provide seamless interaction.

[0041] Therefore, it is desirable to have systems and methods that maximize the quality of the visual diagnostic process, regardless of the technician's skill level, the machine's installation location, the type of machine, or the nature of the fault. The main objective is to incorporate useful visual inputs that can be further used for remote defect diagnosis, parts replacement, repair, and degradation tracking.

[0042] To ensure the quality of visual diagnoses by unskilled technicians at the customer's location, embodiments of this specification enable efficient instrumental diagnostics. This system integrates mechanisms for automatically 1) generating diagnostic plans, 2) guiding technicians, and 3) tracking actions to troubleshoot areas potentially causing faults. Various embodiments include online flows and offline tasks. Offline tasks further include three main components.

[0043] 1) A KG builder that constructs a knowledge graph containing parts, components, spatial areas, and tasks to enable the generation of a diagnostic plan.

[0044] 2) A model builder that constructs image-based models for detecting areas and parts, enabling scene understanding for the diagnostic process.

[0045] 3) An indicator builder that generates an indicator database containing overlays extracted from 2D images, paths, markers, and messages.

[0046] Similarly, the online flow includes three main steps: 1) plan creation, 2) plan tracking, and 3) technician guidance. Figure 2 shows exemplary system flows according to various embodiments of this disclosure, which are described in more detail below. The online flow 202 shows the interaction between device 204, user 206, and steps including generating a diagnostic plan 208, tracking visual states and actions 210, and generating guide indicators 212. Conversely, the offline task 230 includes images and annotations 228, as well as interactions between the knowledge graph builder 220, the model builder 222, and the indicator builder 224.

[0047] Figure 3 shows the core technology architecture and interaction for AI and AR-guided diagnostics for a system for guided diagnostics according to various embodiments of the present disclosure. In an embodiment, the system 300 includes three main components: a knowledge-based guide planning module 302, a scene-based guide grasping module 310, and an AR-based guide module 320, each including two sub-modules. As shown in Figure 3, the knowledge-based guide planning module 302 includes an LLM-based extractor 304 and a spatial viewpoint extractor 306. The scene-based guide grasping module 310 includes a 2D nested object detector 312 and a 3D positioning module 314. The AR-based guide module 320 includes an AR tracking module 322 and an AR interaction module 324.

[0048] In one embodiment, the Knowledge Base Guide Planning Module 302

[0049] In an embodiment, the system 300 uses free-text descriptions of equipment problems, fault trees, and other relevant reference text documents to identify diagnostic tasks, component and part relationships, and spatial perspective information. This information determines the steps that the technician should take during inspection. The embodiment utilizes the LLM model in conjunction with retrieval-augmented generation (RAG) technology to extract entities and relationships, enabling the system to construct inspection checklists. This significantly reduces the time required for manual planning from weeks to minutes and facilitates the rapid deployment of new equipment guides with minimal manual effort.

[0050] A parts-component location extractor can identify parts of the equipment being inspected by extracting entities and their relationships from a subset of text associated with those parts, while a viewpoint-parts-component mapping tool can identify the spatial location of parts that need to be evaluated by a technician during inspection by using a knowledge graph built from image datasets of parts and viewpoints to query and create a diagnostic plan that the technician follows during the inspection process.

[0051] The scene-based guide understanding module 310 uses perceptual sensors (camera and motion) and an AI-based model to determine the current objects in the scene and analyze their size, orientation, position, and coverage relative to a checklist plan. Module 310 can determine whether the technician is following the instructions to complete the checklist plan by analyzing the visual environment using a closed feedback loop between the environment and the inspection performed by the technician. As shown in the figure, the scene-based guide understanding module 310 uses a 2D nested object detector 312 and a 3D positioning module 314.

[0052] The 2D nested object detector 312 improves detection accuracy based on the spatial relationships between objects. This method analyzes the scene from a live camera feed to determine the viewpoint-area-parts of interest and their spatial semantics, thereby determining the correct size, 2D horizontal alignment, and coverage. System 300 combines the 2D detection results for fast inference and does not perform 3D object detection (point cloud) and reconstruction because point cloud detection is slow and its accuracy is reduced due to lighting conditions.

[0053] The 3D positioning module 314 uses motion sensors to determine the current alignment and orientation of the 3D object compared to the expected plan. The alignment, orientation, and rotation of the object are determined when the technician examines the equipment. Some areas may require an orthogonal view, while others may require some tilt. The 3D positioning module 314 further processes readings from the motion sensors (pitch and roll), for example, three times per second, to determine whether the orientation and rotation are correct or incorrect.

[0054] The AR-based guide module 320 can interact with technicians in 3D space using ARKit. ARKit provides the foundation for determining physical space and movement using perceptual sensors. This component manages a list of interactions with technicians based on the inspection plan and environmental conditions. After objects are detected using 2D models, they are mapped into the 3D environment using AR markers and notifications that communicate the next steps in the process to the technician, as shown in Figure 4. The AR-based guide module 320 uses the AR tracking module 322 and the AR interaction module 324.

[0055] The AR tracking module 322 uses the (x,y) position, which has been transformed into (x,y,z) coordinates via a starting point calibration mechanism. This initial step maps the machine and the technician into 3D space. As a result, even if the technician moves, the system can display markers and indicators as part of the camera view.

[0056] The AR interaction module 324 manages visual indicators to guide the technician to complete a checklist plan according to expected actions. Markers are displayed in 3D space (x,y,z) mapped to the technician's 3D space, indicating their real-world position on the machine. The markers remain in 3D real space even as the technician moves. The AR interaction module 324 can use navigation, detection, orientation, and alignment indicators to help the user follow the expected plan.

[0057] Figures 5 and 6 are exemplary workflows illustrating a guided diagnostic process according to various embodiments of the present disclosure. The technician may first select the type of checklist plan to execute, after which the camera live feed is activated. The collection process begins with finding a starting point, which allows the camera view to be positioned relative to the technician's real-world 3D coordinates, so that the system can determine how to begin the guided process. Once the starting point is determined, the system displays markers indicating where the technician needs to capture visual recordings, based on the selected checklist plan. A loop of instructions and indicators then guides the technician in finding the checkpoints. The system displays indicators and messages to guide the technician based on the assessment of the requirements. If the requirements are met, the system indicates that the technician can capture data and proceed to the next item on the checklist. The process is completed when all checkpoints have been captured.

[0058] In one embodiment, the system leverages the LLM method to automatically construct a diagnostic plan to plan the inspection points that a technician would need to follow to diagnose equipment failures. The system uses existing fault trees and free text from manuals, etc., to identify elements that need to be inspected when a problem is reported. To accomplish this, two main modules (shown in Figure 7) may be used: (1) a knowledge base construction module 702 and (2) a plan generator 730. As will be discussed in more detail below, the knowledge base construction module 702 may include an ontology design 704 that defines the main concepts for constructing the KG, an LLM extractor 706 that extracts a task-part-component graph, and a viewpoint extractor 708 that extracts viewpoints from image annotations.

[0059] In one embodiment, ontology design 704 enables reasoning using a rich knowledge base and defines an ontology that includes the categories {part P, component, system, task T, area A, viewpoint V, fault F}. Figure 2 shows an example of classes and relationships for creating a diagnostic plan D for fault f. As described below, {task, part, component, system} are categories automatically extracted from text using LLM. {relationships between viewpoint, area, and part} are extracted from an image annotation dataset.

[0060] In embodiments, the LLM extractor 706 extracts entities representing parts or components of equipment, such as a sauce container, motor, or controller. To achieve this, the LLM allows the use of specific data sources to generate output. Prompts may be generated, including instructions and examples of expected extractions, to generate a graph of parts (or components) and relationships. A data source is provided, including fault trees related to the problem, technician descriptions, and other text-free materials. Model generation is required as a completed task to obtain entities and relationships represented as a graph. Furthermore, the location of parts may be extracted using specific prompts. Figure 8 shows exemplary expected outputs according to various embodiments of the present disclosure.

[0061] To extract parts and relationships, LLM may use the following pseudocode for extracting parts and tasks.

[0062] Pseudocode Input: f_desc: Description of the problem Output: Plan: List of (parts, related tasks) 1. Read the text description f_desc of the fault from the previous record or fault tree table. 2. Set up LLM for completion tasks using system prompts and user prompts. 3. Prepare a system prompt that includes instructions on what to extract and the expected format. 4. Add the f_desc example to the user prompt as a small sample of the user's problem to show the data source. 5. Execute the LLM generation request. 6. Process the output as a tuple of parts and tasks as a diagnostic plan and add it to the knowledge graph.

[0063] In embodiments, the viewpoint extractor 708 acquires the spatial (physical) relationships of parts within a machine, guiding the technician to gather information from the machine, enabling the understanding and interaction component to assist in this. To achieve this, a knowledge graph of parts and viewpoints can be constructed using image data labels. Given a set of labeled images representing the relationships between parts, areas of interest, and viewpoints, a knowledge graph is created. This can be achieved by scanning the images, extracting labels, finding object overlaps, and creating relationships. For example, for a part that should belong to a particular area and a particular viewpoint, the object overlap with the viewpoint and area annotations is calculated. To perform the mapping found by the part extractor, the viewpoint extractor 708 queries the viewpoint graph to identify part-viewpoint relationships. Figure 9 shows exemplary viewpoint graphs according to various embodiments of the present disclosure.

[0064] In one embodiment, the diagnostic plan generator 730 (shown in Figure 7) generates a plan that the technician must follow during the diagnostic process. The plan may include a list of checkpoints to be completed, which may be defined as triplets of viewpoint + area, parts, and requirements list to satisfy size and orientation requirements. The list may be sorted by viewpoint to reduce movement during the inspection process.

[0065] Returning to Figure 3, in an embodiment, the scene-based guide understanding module 310 applies a scene understanding method to grasp the state of data acquisition performed by the technician. Module 310 uses perceptual sensors (camera and motion) and an AI-based model to determine visible objects and their size, orientation, and position relative to a checklist plan. To achieve this, module 310 uses a 2D object detection (nested object detection) module 312 and a 3D positioning module 314. Figure 10 shows exemplary workflows for model training and inference according to various embodiments of the present disclosure.

[0066] In this embodiment, the 2D nested object detection module 312 detects viewpoints (e.g., top, center, etc.) and areas of interest (e.g., controller, refrigerator, etc.) to realize a checklist AR guide. As previously mentioned, conventional object detectors often face challenges in achieving high accuracy due to the limited availability of data for new products, as well as highly reflective materials such as aluminum and glass surfaces. For example, in some customer stores, there may be reflections of other objects on the surface of machinery or objects behind glass doors. To address these challenges, embodiments of this specification train a 2D object model with nested object labels to improve confidence scores and overall detection accuracy. The training dataset is designed using the areas of interest (viewpoint + area), key anchor objects (parts), and object relationships (spatial semantics).

[0067] Figure 11 shows exemplary object labels for a 2D nested object model according to various embodiments of the present disclosure. The Area of ​​Interest (AoI) includes a set of objects within a viewpoint V. The objects of interest are labeled as viewpoint + area, for example, front-top, front-center, left-top, etc. In this case, outer objects representing viewpoints of interest such as front and left, and areas of interest such as top and center, may be defined. Key Anchor Objects (KaOs) are clearly defined, such as shape, contrast, light, color, etc. Key Anchor Objects include a set of one or more objects nested within the viewpoint-area of ​​interest, where KaO j,i ∈AoI ii>1 and j>1 (i≧1 and ≧>1 are also acceptable). Key-anchor objects are selected so that the object detector functions with high accuracy and loss activation propagates to outer objects. As a result, the accuracy of outer object detection is also improved. Anchor objects address the issue of reflective and translucent surfaces. This allows the model to learn representations of objects with minimal variation, thereby influencing the learning of other areas with more variation.

[0068] Various embodiments suggest spatial information based on object relationships. It is suggested that outer objects should also be suggested using highly identified inner objects. Figure 11 shows examples of labeling (bounding boxes) for inner and outer objects in an exemplary machine use case. From a front view, areas of interest are shown: top, middle, and bottom. The outer objects are front-top, front-middle, and front-bottom. Exemplary inner objects are a source container, an LCD device, and a controller protector.

[0069] In the embodiment, visual semantics including nested objects (key, anchor, object-parts) and outer target objects (viewpoint + area) are selected and labeled for training the model. A model for object detection training can be built using an object detection model such as a YOLO network that has low latency and acceptable accuracy (>80%).

[0070] During data acquisition, inference may be performed on frames from the camera feed, for example, every 0.33 seconds (at 3 FPS). Detection results with a confidence score of 0.5 or higher may be selected.

[0071] To improve viewpoint-area detection, anchor objects are used to suggest or infer confidence scores when the viewpoint-area is low due to reflection or area transmittance. The detected bounding boxes can be used to determine relative size, 2D position, and coverage from the scene. The output of object detection (bounding boxes of the object of interest) can be used to calculate the object size within the frame and convert it to expected size and expected area coverage. For the 2D spatial semantics of the object, the positioning of the object within the frame can be used to calculate its center position, thereby determining whether the object is properly centered. Figure 12 shows an exemplary 2D nested object detection.

[0072] Figure 14 is a flowchart showing a process for guided visual diagnosis of equipment failures according to various embodiments of the present disclosure. In an embodiment, process 1400 may be initiated in step 1402 when information regarding a failure related to the equipment is received.

[0073] In step 1404, a KG is constructed to generate a diagnostic plan. The KG can be constructed by a KG builder, which reduces the need for manual data labeling or model training by constructing a graph containing at least one of the parts, components, spatial areas, or diagnostic tasks related to the text source, by extracting information from one or more documents using an LLM.

[0074] In step 1406, the diagnostic plan and real-time AR indicators are used to guide the user through the diagnostic process, and the AR is configured to overlay visual indicators onto the visual data captured by the user.

[0075] Step 1408 enhances 3D scene understanding by analyzing and tracking user actions using an image constructor that constructs an image-based model for detecting areas and parts for scene understanding, and which uses a perception module that includes one or more perception sensors.

[0076] In step 1410, an indicator database is generated which includes overlay annotations extracted from at least one of the following: a 2D image, a path, a marker, or a message.

[0077] In step 1412, a 2D nested object detection model and 3D positioning are used to determine at least one of the object size, orientation, position, or coverage of objects in the scene based on a diagnostic checklist, and motion sensor data is processed to ensure correct orientation and alignment during the diagnosis.

[0078] In step 1414, the AR tracking and interaction module is used to manage visual indicators and guide the user through the diagnostic checklist until all checkpoints on the checklist are met.

[0079] Those skilled in the art will recognize that (1) certain steps may be performed arbitrarily, (2) the steps may not be limited to the specific order described herein, (3) certain steps may be performed in a different order, and (4) certain steps may be performed simultaneously.

[0080] Figure 15 shows an exemplary computing environment with exemplary computer devices suitable for use in several exemplary implementations. The computer device 1505 within the computing environment 1500 may include one or more processing units, cores, or processors 1510, memory 1515 (e.g., RAM, ROM, and / or similar), internal storage 1520 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interfaces 1525, all of which may be coupled on a communication mechanism or bus 1530 for transmitting information, or embedded in the computer device 1505. The I / O interface 1525 may also be configured, depending on the desired implementation, to receive images from a camera or provide images to a projector or display.

[0081] Computer device 1505 can be communicatively coupled to input / user interface 1535 and output device / interface 1540. Either or both of input / user interface 1535 and output device / interface 1540 can be wired or wireless interfaces and can be detachable. Input / user interface 1535 may include any physical or virtual device, component, sensor, or interface that can be used to provide input (e.g., buttons, touchscreen interfaces, keyboards, pointing / cursor controls, microphones, cameras, Braille, motion sensors, optical readers, and / or similar). Output device / interface 1540 may include displays, televisions, monitors, printers, speakers, Braille, and the like. In some exemplary implementations, input / user interface 1535 and output device / interface 1540 can be embedded in computer device 1505 or physically coupled to it. In other exemplary implementations, other computer devices may function as or provide input / user interfaces 1535 and output devices / interfaces 1540 for computer device 1505.

[0082] Examples of computer devices 1505 may include highly mobile devices (e.g., smartphones, devices in vehicles and other machines, devices carried by humans and animals), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions, radios, etc., with one or more processors embedded and / or combined).

[0083] Computer device 1505 can communicate with any number of networked components, devices, and systems, including one or more computer devices of the same or different configurations, by being communicatively coupled to external storage 1545 and network 1550 (for example, via I / O interface 1525). Computer device 1505 or any connected computer device may function as, provide, or be called a server, client, thin server, general-purpose machine, dedicated machine, or other labeled service.

[0084] The I / O interface 1525 may include wired and / or wireless interfaces that use any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMAX, modem, cellular network protocol, etc.) to transmit information to and from at least all connected components, devices, and networks within the computing environment 1500. The network 1550 may be any network or combination of networks (e.g., the Internet, local area network, wide area network, telephonic network, cellular network, satellite network, etc.).

[0085] Computer devices 1505 may use and / or communicate using computer-usable or computer-readable media, including temporary and non-temporary media. Temporary media include transmission media (e.g., metal cables, optical fibers), signals, carrier waves, etc. Non-temporary media include magnetic media (e.g., disks and tapes), optical media (e.g., CD-ROMs, digital video discs, Blu-ray® discs), solid-state media (e.g., RAM, ROMs, flash memory, solid-state storage), and other non-volatile storage or memory.

[0086] Computer device 1505 can be used to implement techniques, methods, applications, processes, or computer executable instructions in several exemplary computing environments. Computer executable instructions can be retrieved from temporary media, stored on non-temporary media, and retrieved from non-temporary media. Executable instructions can arise from one or more of any programming languages, scripting languages, and machine code (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0087] The processor 1510 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, including a logic unit 1560, an application programming interface (API) unit 1565, an input unit 1570, an output unit 1575, and an inter-unit communication mechanism 1595 for different units to communicate with each other, with the OS, and with other applications (not shown). The units and elements described are subject to change in design, function, configuration, or implementation, and are not limited to those described. The processor 1510 may take the form of a hardware processor such as a central processing unit (CPU), or a combination of hardware and software units.

[0088] In some exemplary implementations, once information or execution instructions are received by the API unit 1565, they may be propagated to one or more other units (e.g., logic unit 1560, input unit 1570, output unit 1575). In some examples, the logic unit 1560 may be configured to control the flow of information between units and direct the services provided by the API unit 1565, input unit 1570, and output unit 1575 in some exemplary implementations described above. For example, the flow of one or more processes or implementations may be controlled by the logic unit 1560, either alone or in conjunction with the API unit 1565. The input unit 1570 may be configured to take input for the computation described in the exemplary implementation, and the output unit 1575 may be configured to provide output based on the computation described in the exemplary implementation.

[0089] The processor 1510, in response to receiving information about a fault related to the equipment, as described with respect to Figure 2, for example, uses the LLM to construct a KG used to generate a diagnostic plan, the KG being constructed by a KG constructor, which can be configured to execute a method or computer instruction that includes steps including using to extract information from one or more documents and construct a graph containing at least one of the parts, components, spatial areas, or diagnostic tasks related to the text source, thereby reducing the need for manual data labeling or model training.

[0090] The processor 1510 may use a diagnostic plan and real-time AR indicators to guide the user through the diagnostic process, as described with respect to Figure 2, for example, and the AR may be configured to use or execute computer instructions, which may include using visual indicators to overlay visual indicators on visual data captured by the user.

[0091] The processor 1510 may use an image constructor to analyze and track user actions to enhance 3D scene understanding, as described with respect to Figures 3 and 14, for example, the image constructor constructs an image-based model for detecting areas and parts for scene understanding, and the image constructor may be configured to use, or include using, a perceptual module including one or more perceptual sensors, or to execute computer instructions.

[0092] The processor 1510 can be configured to execute a method or computer instruction that may include generating an indicator database containing overlay annotations extracted from at least one of a 2D image, path, marker, or message, as described with respect to Figures 2 and 14, for example.

[0093] The processor 1510 can be configured to execute methods or computer instructions that may include, for example, as described with respect to Figures 2, 3, and 14, utilizing a 2D nested object detection model and 3D positioning to determine at least one of the object size, orientation, position, or coverage of objects in a scene based on a diagnostic checklist, and processing motion sensor data to ensure correct orientation and alignment during diagnosis.

[0094] The processor 1510 may be configured to execute computer instructions in a manner that includes using an AR tracking and interaction module to manage visual indicators and guide the user along the diagnostic checklist until all checkpoints on the diagnostic checklist are met, as described with respect to Figures 2, 3, and 14, for example.

[0095] Some of the detailed explanations are presented in terms of algorithms and their symbolic representations of computer operations. These algorithmic descriptions and symbolic representations are means used by those skilled in the field of data processing technology to communicate the essence of their innovations to others skilled in the field. An algorithm is a set of defined steps that bring about a desired final state or result. In exemplary implementations, the steps performed require the physical manipulation of tangible quantities to achieve a tangible result.

[0096] Unless otherwise noted, as will be evident from this discussion, discussions throughout this explanation using terms such as “processing,” “computing,” “calculating,” “decision,” and “display” may include the operation and processes of a computer system or other information processing device that manipulates data represented as physical (electronic) quantities in the registers and memory of a computer system and converts it into other data similarly represented as physical quantities in the memory or registers of a computer system, or in other information storage, transmission, or display devices.

[0097] Exemplary implementations may also relate to apparatus for performing the operations described herein. Such apparatus may include one or more general-purpose computers, which may be specifically constructed for a required purpose or selectively operated or reconfigured by one or more computer programs. Such computer programs may be stored on computer-readable media, such as computer-readable storage media or computer-readable signal media. Computer-readable storage media may include tangible media such as optical disks, magnetic disks, read-only memory, random-access memory, solid-state devices, drives, or any other type of tangible or non-temporary medium suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and representations presented herein are not inherently related to any particular computer or other apparatus. Computer programs may include pure software implementations containing instructions for performing the operations of a desired implementation.

[0098] Various general-purpose systems may be used with the programs and modules illustrated herein, or it may be advantageous to construct more specialized devices to perform desired method steps. Furthermore, the exemplary implementations are not described with reference to any particular programming language. It will be understood that various programming languages ​​may be used to implement the techniques of the exemplary implementations described herein. Instructions in a programming language may be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.

[0099] As is known in the art, the operations described above can be performed by hardware, software, or any combination of software and hardware. Various embodiments of the exemplary implementations may be implemented using circuits and logic devices (hardware), while other embodiments may be implemented using instructions stored in a machine-readable medium (software) that, when performed by a processor, cause the processor to perform the methods for carrying out the implementations of this application. Furthermore, some exemplary implementations of this application may be performed by hardware alone, while other exemplary implementations may be performed by software alone. Moreover, the various functions described may be performed by a single unit or distributed across several components in any number of ways. When performed by software, these methods may be performed by a processor, such as a general-purpose computer, based on instructions stored in a computer-readable medium. Instructions may be stored on the medium in a compressed and / or encrypted form, if desired.

[0100] Furthermore, other implementations of this application will become apparent to those skilled in the art from examining this specification and practicing the art of this application. Various aspects and / or components of the exemplary implementations described may be used individually or in any combination. This specification and exemplary implementations should be considered merely examples, and the true scope and idea of ​​this application shall be shown by the following claims.

Claims

1. A guided visual diagnostic method for equipment malfunction, In response to receiving information about equipment-related failures, a Large Language Model (LLM) is used to construct a Knowledge Graph (KG) used to generate a diagnostic plan. The aforementioned KG is constructed by a KG constructor. The KG builder reduces the need for manual data labeling or model training by using the LLM to extract information from one or more documents and constructing a graph that includes at least one of the parts, components, spatial areas, or diagnostic tasks related to the text source. The diagnostic plan and real-time augmented reality (AR) indicators are used to guide the user through the diagnostic process, The AR is configured to overlay a visual indicator onto the visual data captured by the user. To enhance understanding of 3D scenes, we use an image constructor to analyze and track user actions, The aforementioned image constructor builds an image-based model for detecting areas and parts for scene understanding, The image constructor uses a perception module that includes one or more perception sensors. To generate an indicator database containing overlay annotations extracted from at least one of 2D images, paths, markers, or messages, This involves using a 2D nested object detection model and 3D positioning to determine at least one of the object size, orientation, position, or coverage of objects in a scene based on a diagnostic checklist, and processing motion sensor data to ensure correct orientation and alignment during diagnosis. To manage visual indicators and guide the user along the diagnostic checklist until all checkpoints on the diagnostic checklist are met, an AR tracking and interaction module is used. Guided visual diagnostic methods, including...

2. The method according to claim 1, wherein the one or more documents include at least one of a manual or a text.

3. The method according to claim 1, wherein the 2D nested object detection model uses a training dataset that includes a relatively small set of training samples in order to improve detection accuracy.

4. The method according to claim 1, wherein the one or more perceptual modules provide feedback and recommendations based on the analysis and tracking of the actions.

5. The method according to claim 4, wherein at least one of the aforementioned feedback or recommendation is provided in real time.

6. The method according to claim 1, wherein the text source includes at least one of a parts list or a failure report.

7. The method according to claim 1, wherein the one or more sensory sensors include a camera and / or a motion sensor.

8. The method according to claim 1, wherein performing initial calibration includes mapping the user and the equipment into a 3D space using 2D images and AR tracking.

9. The method according to claim 1, wherein each of the 2D images includes at least a portion of the object of interest.

10. In response to receiving information about equipment-related failures, a Large Language Model (LLM) is used to construct a Knowledge Graph (KG) used to generate a diagnostic plan. The aforementioned KG is constructed by a KG constructor. The KG builder reduces the need for manual data labeling or model training by extracting information from one or more documents and constructing a graph that includes at least one of the parts, components, spatial areas, or diagnostic tasks related to the text source. The diagnostic plan and real-time augmented reality (AR) indicators are used to guide the user through the diagnostic process, The AR is configured to overlay a visual indicator onto the visual data captured by the user. To enhance understanding of 3D scenes, we use an image constructor to analyze and track user actions, The aforementioned image constructor builds an image-based model for detecting areas and parts for scene understanding, The image constructor uses a perception module that includes one or more perception sensors. To generate an indicator database containing overlay annotations extracted from at least one of 2D images, paths, markers, or messages, This involves using a 2D nested object detection model and 3D positioning to determine at least one of the object size, orientation, position, or coverage of objects in a scene based on a diagnostic checklist, and processing motion sensor data to ensure correct orientation and alignment during diagnosis. To manage visual indicators and guide the user along the diagnostic checklist until all checkpoints on the diagnostic checklist are met, an AR tracking and interaction module is used. A computer program that causes a computer to execute a command.

11. The computer program according to claim 10, wherein the one or more documents include at least one of a manual or a text.

12. The computer program according to claim 10, wherein the 2D nested object detection model uses a training dataset containing a relatively small set of training samples to improve detection accuracy.

13. The computer program according to claim 10, wherein the one or more perceptual modules provide feedback and recommendations based on the analysis and tracking of actions.

14. The computer program according to claim 13, wherein at least one of the aforementioned feedback or recommendation is provided in real time.

15. The computer program according to claim 10, wherein the text source includes at least one of a parts list or a failure report.

16. The computer program according to claim 10, wherein the one or more sensory sensors include a camera and / or a motion sensor.

17. The computer program according to claim 10, wherein performing initial calibration includes mapping the user and the equipment into a 3D space using 2D images and AR tracking.

18. The computer program according to claim 10, wherein each of the 2D images includes at least a portion of the object of interest.

19. A means of using a Large-Scale Language Model (LLM) to construct a knowledge graph (KG) used to generate a diagnostic plan in response to receiving information about equipment-related failures, The aforementioned KG is constructed by a KG constructor. The KG builder reduces the need for manual data labeling or model training by extracting information from one or more documents and constructing a graph that includes at least one of the parts, components, spatial areas, or diagnostic tasks related to the text source. Means for using the diagnostic plan and real-time augmented reality (AR) indicators to guide the user through the diagnostic process, The AR is configured to overlay a visual indicator onto the visual data captured by the user. Means for using an image constructor to analyze and track user actions in order to enhance 3D scene understanding, The aforementioned image constructor builds an image-based model for detecting areas and parts for scene understanding, The image constructor uses a perception module that includes one or more perception sensors. Means for generating an indicator database including overlay annotations extracted from at least one of a 2D image, path, marker, or message, A means for using a 2D nested object detection model and 3D positioning to determine at least one of the object size, orientation, position, or coverage of objects in a scene based on a diagnostic checklist, and for processing motion sensor data to ensure correct orientation and alignment during diagnosis, Means for using an AR tracking and interaction module to manage visual indicators and guide the user along the diagnostic checklist until all checkpoints on the diagnostic checklist are met, A device equipped with the following features.

20. A means of using a Large-Scale Language Model (LLM) to construct a knowledge graph (KG) used to generate a diagnostic plan in response to receiving information about equipment-related failures, The aforementioned KG is constructed by a KG constructor. The KG builder reduces the need for manual data labeling or model training by extracting information from one or more documents and constructing a graph that includes at least one of the parts, components, spatial areas, or diagnostic tasks related to the text source. Means for using the diagnostic plan and real-time augmented reality (AR) indicators to guide the user through the diagnostic process, The AR is configured to overlay a visual indicator onto the visual data captured by the user. Means for using an image constructor to analyze and track user actions in order to enhance 3D scene understanding, The aforementioned image constructor builds an image-based model for detecting areas and parts for scene understanding, The image constructor uses a perception module that includes one or more perception sensors. Means for generating an indicator database including overlay annotations extracted from at least one of a 2D image, path, marker, or message, A means for using a 2D nested object detection model and 3D positioning to determine at least one of the object size, orientation, position, or coverage of objects in a scene based on a diagnostic checklist, and for processing motion sensor data to ensure correct orientation and alignment during diagnosis, Means for using an AR tracking and interaction module to manage visual indicators and guide the user along the diagnostic checklist until all checkpoints on the diagnostic checklist are met, A guided visual diagnostic system equipped with [specific features / features].

Images