Methods and systems for adaptive augmented reality in interactive learning experiences

Abstract

Methods and systems for adaptive augmented reality in interactive learning experiences are provided. According to an aspect, a method for facilitating interactive learning experience through use of an interactive work instruction in a real-world setting is provided. The method includes identifying a position of an anchor in a physical environment of a work subject. The position of the anchor refers to a location and orientation of the anchor in the physical environment. The anchor corresponds to a work instruction related to the work subject. The method further includes generating digital content to be overlaid on the physical environment of the work subject based on the position of the anchor. The digital content corresponds to the work instruction.

Description

METHODS AND SYSTEMS FOR ADAPTIVE AUGMENTED REALITY IN INTERACTIVE LEARNING EXPERIENCESFIELD OF THE INVENTION

[0001] The present invention pertains to the field of augmented reality (AR) systems, and in particular to methods and systems for adaptive augmented reality in interactive learning experiences.BACKGROUND

[0002] In modern sectors, traditional training and education methods are proving inadequate. These methods, which have not evolved at the same pace as technological advancements, often fail to provide the interactive learning experiences necessary to engage today’s learners effectively. There is a disconnect when young learners are introduced to practical skills, which are often not aligned with current industry demands. The prevailing learning styles, which favor interactive and hands-on approaches, are not adequately supported by one-way, lecture-based educational models. Furthermore, the cost of keeping training programs up-to-date is becoming increasingly prohibitive for both individuals seeking skills enhancement and organizations needing to train their workforce. This creates a barrier to the development of essential skills in the workforce, necessitating a reevaluation of how interactive learning experiences are structured and delivered across various industries.

[0004] Therefore, there is a need for methods and systems for adaptive augmented reality in interactive learning experiences that obviates or mitigates one or more limitations of the prior art.

[0005] This background information is provided to reveal information believed by the applicant to be of possible relevance to the present invention. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art against the present invention.SUMMARY

[0006] Methods and systems for adaptive augmented reality in interactive learning experiences are provided. According to an aspect, a method for facilitating interactive learningexperience through use of an interactive work instruction in a real-world setting is provided. The method includes identifying a position of an anchor in a physical environment of a work subject. The position of the anchor refers to a location and orientation of the anchor in the physical environment. The anchor corresponds to a work instruction related to the work subject. The method further includes generating digital content to be overlaid on the physical environment of the work subject based on the position of the anchor. The digital content corresponds to the work instruction.

[0007] In some embodiments, identifying the position of the anchor in the physical environment of the work subject includes identifying a location and an orientation of the anchor in the physical environment of the work subject. In some embodiments, generating the digital content to be overlaid on the physical environment of the work subject based on the position of the anchor includes generating the digital contentto be overlaid on the physical environment of the work subject based on the location and the orientation of the anchor.

[0008] In some embodiments, the digital content when overlaid on the physical environment of the work subject maintains a spatial relationship with the anchor. In some embodiments, the digital content when overlaid on the physical environment of the work subject maintains a spatial relationship with the anchor based on the location and the orientation of the anchor.

[0009] In some embodiments, identifying the position of the anchor in the physical environment of the work subject comprises identifying multiple reference positions each corresponding to a different anchor of multiple anchors. In some embodiments, generating the digital content to be overlaid on the physical environment of the work subject based on the position of the anchor includes generating the digital content to be overlaid on the physical environment based on one or more reference positions of the multiple reference positions.

[0010] In some embodiments, generating the digital content to be overlaid on the physical environment based on one or more reference positions of the multiple reference positions includes generating and overlaying the digital content based one or more thresholds including: a time threshold, a number of reference positions, and an overlay accuracy. In some embodiments, generating the digital content to be overlaid on the physical environment based on one or more reference positions of the multiple reference positions includes comparing an overlay of the digital content corresponding to the one or more reference positions with another overlay of the digital content corresponding to one or more different reference positions of the multiple reference positions.

[0011] In some embodiments, the method further includes selecting the work instruction from a list of work instructions related to the work subject. In some embodiments, the anchor is part of or attaches to the work subject. In some embodiments, the anchor is separate from the work subject.

[0012] According to another aspect, an apparatus is provided. The apparatus includes modules or electronics configured to perform one or more of the methods and systems described herein.

[0013] According to one aspect, an apparatus is provided, where the apparatus includes: a memory, configured to store a program; a processor, configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to perform one or more of the methods and systems described herein.

[0014] According to another aspect, a computer readable medium is provided, where the computer readable medium stores program code executed by a device and the program code is used to perform one or more of the methods and systems described herein.

[0015] According to one aspect, a chip is provided, where the chip includes a processor and a data interface, and the processor reads, by using the data interface, an instruction stored in a memory, to perform one or more of the methods and systems described herein. Aspects may further include the memory.

[0016] Other aspects of the disclosure provide for apparatus, and systems configured to implement the methods according to the first aspect disclosed herein. For example, wireless stations and access points can be configured with machine readable memory containing instructions, which when executed by the processors of these devices, configures the device to perform one or more of the methods and systems described herein.

[0017] Embodiments have been described above in conjunctions with aspects of the present invention upon which they can be implemented. Those skilled in the art will appreciate that embodiments may be implemented in conjunction with the aspect with which they are described, but may also be implemented with other embodiments of that aspect. When embodiments are mutually exclusive, or are otherwise incompatible with each other, it will be apparent to those skilled in the art. Some embodiments are described in relation to one aspect, but may also be applicable to other aspects, as will be apparent to those of skill in the art.BRIEF DESCRIPTION OF THE FIGURES

[0018] Further features and advantages of the present invention will become apparent from the following detailed description, taken in combination with the appended drawings, in which:

[0019] FIG. 1 illustrates a flowchart for an AR-based work instruction development and implementation procedure, according to an embodiment.

[0020] FIG. 2 illustrates a flowchart of an AR-guided work instruction procedure, according to an embodiment.

[0021] FIG. 3 illustrates an apparatus that may perform any or all of operations of the above methods and features explicitly or implicitly described herein, according to different embodiments.

[0022] FIG. 4 illustrates a method for facilitating interactive learning experience, according to an embodiment.

[0023] It will be noted that throughout the appended drawings, like features are identified by like reference numerals.DETAILED DESCRIPTION

[0024] Methods and systems for adaptive augmented reality in interactive learning experiences are provided. According to an aspect, a method is provided for facilitating interactive learning or training experience in real-world settings. Method includes identifying a position of an anchor in a physical environment of a work subject. The position of the anchor includes a location and an orientation of the anchor in the physical environment. The anchor corresponds to a work instruction related to the work subject. The method further includes generating digital content to be overlaid on the physical environment of the work subject based on the position of the anchor, including the location and orientation of the anchor. The digital content corresponds to the work instruction.

[0025] One or more embodiments provide for AR interactive learning experience in various applications across different industries. Some example applications include trade skills (i.e., heavy truck and trailer technician / carpentry, etc.) where technicians or apprentices are guided through complex machinery maintenance. Medical training and healthcare are other potential applications, where a trainee can undergo surgical training in an operations room or anemergency protocol in an emergency room. Emergency response and public safety training can also be potential applications of one or more embodiments. For example, first responders can also be trained in simulated emergency scenarios, where responders navigate through virtual disaster settings, practicing skills like evacuation procedures or first aid, with the system providing situational guidance and performance feedback.

[0026] A “work subject” as used herein refers to an object, entity, environment or context that is related to a work or a task. Correspondingly, a work is performed on or is associated with a work subject. While often it has a physical presence, such as a machine component (like an engine) or a designated space (like an emergency room in a hospital), it can also encompass abstract or virtual spaces.

[0027] A “work instruction” refers to the work or task that is related to the work subject. A work instruction involves a set of steps, directions or guidelines that break down the work into manageable and structured actions. These steps are designed to guide a user through the process of performing the work. For example, a work instruction can be an oil change performed on an engine. A work instruction for an oil change on an engine might include steps such as “drain the old oil,” “replace the oil filter,” and “refill with new oil.”

[0028] In one embodiment, the work instruction is provided as part of a library.

[0029] In one embodiment, the library contains one or more work instructions related to a specific topic or application. For example, in the field of heavy equipment maintenance, in one embodiment, the library may include a group or series of tasks related to maintenance of a specific component or part of the heavy equipment, such as an engine block or a transmission. In another embodiment, the library may include work instructions relating all maintenance tasks typically associated with maintaining a specific type (or model) of vehicle.

[0030] In one embodiment, the library includes one or more learning modules, in which the modules are designed to take the user through range of maintenance tasks.

[0031] In one embodiment, the library includes a series of learning modules, beginning at a basic, learner level module and progressing through increasingly difficult and complex tasks in subsequent advanced modules. The library, in this embodiment, can guide the user through a structured learning path, starting from apprentice / beginner levels and advancing though to attaining a skill level suitable to obtain professional accreditation.

[0032] In one embodiment, access to a library of work instructions is provided to the user on a subscription basis.

[0033] An “anchor” refers to a spatial reference point within the physical environment of the work subject. It serves as a spatial reference point for generating and overlaying digital content, such as holograms or instructions, onto the physical environment. An anchor can manifest in various forms, but it is often a tangible, physical object integrated into the physical environment of the work subject. For instance, it could take the shape of a positioned QR code attached to a specific component of the work subject, such as a component of an engine. In this scenario, the AR (Augmented Reality) system recognizes and utilizes the anchor as a reference point for superimposing digital content related to the ongoing work instruction. The anchor’s position including its location and orientation are used as reference for overlaying the digital content on the physical environment.

[0034] The anchor’s position refers to both its specific location and orientation within the physical environment of the work subject. These positional and orientational details serve as the foundation for accurately overlaying digital content onto the physical environment, aligning it with the work subject, and providing users with a seamless and contextually relevant augmented learning experience.

[0035] Accordingly, the anchor can be anything that serves as the reference point for overlaying digital content. In some instances, the anchor is an inherent part of the work subject itself. For example, it may be a specific component of an engine, such as the engine block or a critical fixture, which possesses a precisely known location and orientation within the subject. This internal anchor remains static relative to the work subject, providing a consistent reference for overlaying digital content. Augmented reality instructions, visual cues, or holographic guides are generated based on this stable anchor, facilitating tasks like maintenance, repairs, or assembly.

[0036] Alternatively, the anchor can exist separately from the work subject. This external anchor can be a specialized tool or fixture designed to establish a fixed position reference (including location and orientation) within the physical environment. The tool or fixture is carefully calibrated to maintain its stability and alignment. The digital content is then generated based on the position of this external anchor.

[0037] A “digital overlay” or “overlay” refers to digital content that is superimposed onto a user's real-world view. This content typically includes text, images, animations, and interactive elements designed to augment and enhance the user's perception of their physical environment. It dynamically updates and adapts in real-time based on changes in the user’s environment, interactions, and tasks at hand. The overlay aims to provide contextual, relevant, and helpful information, thereby enriching the user's experience by seamlessly integrating virtual elements with their real-world surroundings.

[0038] FIG. 1 illustrates a flowchart for an AR-based work instruction development and implementation procedure, according to an embodiment. Flowchart 100 includes, at 102, defining a work instruction based on a work or a task related to a work subject. This step involves analyzing the task to understand its steps, requirements, and objectives. This step may further involve gathering relevant information including existing documentation, guidelines, manuals, or any other reference materials. The work instruction should align with existing processes, protocols, and standards within the organization, and comply with relevant industry regulations and legal requirements, where applicable.

[0039] Analyzing the task may involve breaking down the task into its individual steps or components. Analyzing the task may further involve documenting each step in a sequential order. This analysis should provide a detailed understanding of how the task is performed. This step may further involve determining the objectives and goals of the task, for example, determining what is the purpose of performing the task and what should be achieved by the end of it. This step may further involve determining requirements, which involves determining all the resources and prerequisites necessary to complete the task successfully. This could include tools, equipment, materials, personnel, skills, and time-frames. Where applicable, safety and quality standards are integrated into the task analysis. Potential hazards or quality checkpoints that need to be addressed in the work instruction are identified.

[0040] Flowchart 100 further includes, at 104, identifying a position of the anchor. A position including a location and orientation of the anchor can be chosen that aligns with the work instruction's needs and objectives. The anchor may be positioned in a way that aligns with the steps of the work instruction. For example, the position of the anchor should be logically integrated into the physical workflow. In some embodiments, the anchor enhances the user's ability to perform the task effectively. Ergonomics and safety aspects may be considered in determining the position of the anchor.

[0041] In some embodiments, the anchor's position can be optimized for user convenienceduring the setup phase, making it accessible for the user to interact with to set the reference point accurately. After selecting the position of the anchor, a suitable anchor may be selected for the work instruction. The anchor is selected to ensure that it is recognizable by the AR system for establishing the reference point.

[0042] Flowchart 100 further includes, at 106, testing, calibrating and documenting the anchor’s positioning and recognition. This may involve testing the anchor's recognition in various environmental conditions, including lighting conditions, angles, temperatures, humidity levels, etc. This step may further involve calibrating the system to establish a stable reference frame. Further details about the anchor's position, characteristics, and alignment with the work instruction for future reference and maintenance can be documented.

[0043] Flowchart 100 further includes, at 108, generating digital content and establishing spatial relationship. At this step, each step of the work instruction is mapped to specific digital content requirements. Further, digital content tailored to each step is generated considering the anchor's position and orientation. Further spatial relationship, including the digital content's position and orientation with respect to the anchor is defined. Accordingly, a mapping of digital content to the anchor’s position (location and orientation in the physical environment of the work subject) is developed that associates the anchor to the digital content at each step for the work instruction. This mapping of digital content to the anchor defines what digital content should be displayed for each step of each work instruction. This mapping further defines the relative position and orientation of the digital content with respect to the anchor for each step. Some digital content need not have a spatial relationship with the position of the anchor, and as such these digital content can be displayed on user’s view without a defined mapping with the anchor.

[0044] The digital content can be informative and interactive, allowing user to engage with the content in a more hands-on manner. Digital content includes various forms of media such as text, images, videos, 3D models, holograms and animations to cater to different learning styles and enhance the understanding of the work instruction. Digital content can have options for different font sizes, contrast settings, and support for screen readers if needed. Digital content can dynamically adjust based on the user’s viewpoint and interaction with the environment.

[0045] Flowchart 100 further includes, at 110, integrating the generated digital content into the AR system's database or platform. This integration also includes the mapping information of the digital content to the anchor for each step of each work instruction. This integration mayinvolve structuring and storing the digital content within the AR system's database, ensuring it is categorized effectively for ease of access and retrieval. Types of content such as text, images, videos, and interactive elements can be organized based on their respective steps and linked anchor points.

[0046] Flowchart 100 further includes, at 112, real-time interaction and user guidance. At this step, the AR system is configured to enable the overlay of digital content onto the physical environment related to the work subject in real-time. The digital content is configured to be dynamically displayed as users interact with the digital content through the work instruction. The system is designed to recognize and track the physical environment, as well as the user's position and orientation within it, allowing for the accurate placement of digital content in the user's field of vision.

[0047] Furthermore, the digital content is not static; it is configured to be dynamic and responsive. As users interact with the work instruction, the digital content reacts and updates in real-time to reflect their actions and decisions. This could include changes in content based on the user's progress through the work instruction steps, adjustments in response to user inputs, or even modifications due to changes in the physical environment.

[0048] The AR system's ability to provide real-time user guidance is a key feature of this step. As users navigate through the work instructions, the system offers contextual guidance, tips, and information, enhancing their understanding and performance of the task. This guidance is context-aware, meaning it adjusts based on where the user is within the task sequence and what they are currently viewing or interacting with.

[0049] Depending on the complexity of the work instruction, the AR system enables user interactivity with the digital content. The user can manipulate, resize, or query the content for additional information to enhance their learning experience.

[0050] Flowchart 100 can further include, at 114, documentation and training. This step involves providing documentation and training for users or operators on how to use the AR system effectively, including instructions on how to interact with the anchor and access digital content for different steps within each work instruction.

[0051] The AR-based work instruction development and implementation procedure illustrated in flowchart 100 integrates augmented reality technology into the creation and execution of work instructions thereby providing an immersive and interactive learningexperience. The methodical approach may allow for improved quality and clarity of work instructions and may further ensure they are accessible, interactive, and aligned with the specific needs of the task at hand.

[0052] AR work instructions generated as per flowchart 100 can reduce the need for physical training materials and resources. They can also decrease the time and cost associated with training employees, as AR can simulate various scenarios and environments. AR systems can be tailored to meet the specific needs of different tasks, users, and environments. This customization allows for greater flexibility in how work instructions are presented and interacted with.

[0053] Once the work instruction(s) have been developed and integrated into the AR system, the focus shifts to the user's experience. This is where the AR-guided work instruction procedure comes into play. This procedure is from a user's perspective and outlines how an individual interacts with and navigates through the AR-enhanced work instruction. It provides a step-by-step guide on how users can select, understand, and complete tasks using the augmented reality interface. This user-centric approach may facilitate a deeper understanding of the work subject and may enhance the learning experience through interactive and immersive technology.

[0054] The flowchart 100 is described based on a single reference point according to the position of the anchor. In some embodiments, the AR system utilizes multiple reference points or anchors within the environment. This approach allows for a more accurate overlay of digital content, especially in complex or extensive physical spaces. The digital content can be mapped to each of the multiple reference points when creating the digital content. Each piece of digital content maintains a defined spatial relationship with each reference point, ensuring accuracy in various user positions and perspectives.

[0055] In some embodiments, The AR system dynamically selects one or more reference points for generating digital content, based on factors such as the work instruction, the work subject, the user's position, and their viewing angle. The system can employ algorithms to decide which reference points are most suitable for generating the digital content in any given scenario.

[0056] In some embodiments, the reference points can have a priority, a ranking or an order according to which the AR system generates the digital content. For example, AR system can generate the digital content based on a reference point that is ranked first or has a first priority.For whatever reason (e.g., the user moves or the work subject moves), if the AR system has difficulty in generating or accurately generating the digital content based on the reference point that is ranked first, the AR system then uses the reference point that is ranked second or has a second priority to generate the digital content.

[0057] In some embodiments, the AR system may dynamically select a reference point, among the set of reference point, to use as the reference point for generating the digital content. This dynamic selection may occur, for example, when the user or the work subject moves, and the AR system has to dynamically adjust the digital content. The AR system may select the reference point based on the view and the position of the user for example.

[0058] In some embodiments, the AR system can use a threshold number of reference points for generating the digital content. The threshold number of reference points can be understood as the number of reference points used for generating the overlay which may ensure a level of overlay accuracy.

[0059] In some embodiments, a threshold time may be set as the limit for the time the AR system takes to generate the digital content, beyond which, the AR system dynamically reduces the number of reference points used to generate the digital overlay. This threshold time ensures that the digital content is generated in a real-time manner or without substantial delay from the user’s perspective. By reducing the number of the reference point used to generate the digital content, the AR system performs fewer processing operations to generate the digital content.

[0060] In some embodiments, the AR system may verify the positioning of the digital content to be generated (or the digital overlay) with respect to one or more reference points with the positioning (or mapping or spatial relationship) of the same digital content with respect to one or more different reference points when generating and overlaying the digital content. This verification mechanism may allow for improved accuracy of the digital overlay.

[0061] In some embodiments, a digital overlay generated based on a reference point is considered accurate if the digital overlay maintains its spatial relationship when compared to a second reference point. The same comparison can be done when the digital overlay is generated based on multiple reference points, in which the digital overlay is further compared to one or more reference points different from the multiple reference points used to generate the digital overlay. A digital overlay is considered to be accurate (i.e., maintain its spatial relationship with one or more different reference points) if the error of the digital overlay iswithin an acceptable limit when the digital overlay is compared to an equivalent or another overlay (i.e., the same digital content overlaid) based on one or more different reference points. Various mechanisms and techniques exist to measure such errors. In some embodiments, an overlay accuracy threshold may be set for generating the digital overlay. Accordingly, the AR system may continuously monitor this overlay accuracy by performing appropriate error calculations and dynamically change the one or more reference points used to generate the digital content when the overlay accuracy drops below the overlay accuracy threshold. The change in the one or more reference points to satisfy the overlay accuracy may reduce or increase the number of reference points used.

[0062] In some embodiments, the AR system dynamically selects one or more reference point based on one or more thresholds for generating and overlaying the digital content, the one or more thresholds include a time threshold, a number of reference points, and an overlay accuracy as described herein.

[0063] In some embodiments, when setting up (identifying or establishing) multiple reference points, each reference point may have a defined spatial relationship (a mapping or a positional relationship (location and orientation) with one or more other reference points. This defined spatial relationship includes information about their relative distances, angles, or orientations. These spatial relationship or mapping may be integrated with the AR system’s database. This integration allows the system to understand how reference points are related in the physical space. When a user activates one reference point, the AR system can automatically recognize and activate additional reference points based on their predefined spatial relationships. In addition to automatic recognition of one or more reference point from recognition and activation of a reference point (via the user), the user may further manually active multiple refence points, thereby further improving the accuracy and reliability of the reference points to be used for generating and overlaying digital content. This can be especially useful in large or complex environments where activating additional points can provide better coverage and overlay accuracy.

[0064] FIG. 2 illustrates a flowchart of an AR-guided work instruction procedure, according to an embodiment. The flowchart 200 includes, at 202, selecting a work instruction related to a work subject. The user begins by selecting a specific work instruction from a list of available work instructions. This choice can be based on the task they need to perform on the work subject. The system can aid this decision by providing brief descriptions or visuals of each work instruction. To streamline the process, the system can include search and filterfunctionalities, allowing users to quickly find specific instructions or those that fit certain criteria (e.g., skill level required, estimated completion time). Once a work instruction is selected, the system should confirm the user’s choice and guide them to the next step, which is typically positioning and activating the anchor.

[0065] Flowchart 200 further includes, at 204, positioning and activating the anchor. Once the work instruction is selected, the user is instructed by the AR system to position and activate the anchor in the physical environment of the work subject. The user then positions and activates the anchor. The user may need to place the anchor at a specific location and orientation that aligns with the chosen work instruction. By positioning the anchor, the user is setting up the spatial context for the AR system, allowing it to understand where to display the digital content in the user's field of vision.

[0066] In some embodiments, the AR system provides step-by-step instructions or guidelines overlaid on the user's field of view on how to position and activate the anchor. These instructions may be referred to as anchor-alignment assistance content and can include arrows, outlines, matching shapes, or animated paths to guide the user showing where and how to place the anchor. As the user moves to position the anchor, the system can give realtime feedback, such as visual cues turning from red to green when the anchor is correctly positioned. The system may assist in aligning the anchor correctly. For instance, it could display a virtual image of the anchor that aligns with its physical counterpart when correctly positioned.

[0067] Once positioned, the user activates the anchor. Activation could involve scanning the anchor with the AR device or confirming its placement through a user interface. For virtual anchors, the user aligns the virtual marker to a specific point in the physical environment, often guided by the AR system’s overlay cues. Once aligned, the user confirms the placement, possibly through a gesture, voice command, or interaction with the AR device’s interface.

[0068] In some embodiments, where multiple reference points are established, when the user activates one reference point, the AR system can automatically recognize and activate additional reference points based on their predefined spatial relationships. This creates a cascading effect, where activating one point brings others into the operational framework. Beyond automatic recognition, users can manually activate multiple reference points. This can be especially useful in large or complex environments where activating additional points can provide better coverage and overlay accuracy.

[0069] Flowchart 200 further includes, at 206, recognizing the anchor. The AR system recognizes the anchor's position and orientation within the physical environment. This recognition allows the system to establish a reference point for overlaying digital content. The AR system may use advanced image processing techniques to identify the anchor within the user's field of view. This may involve analyzing the visual data captured by the AR device's camera. For physical anchors, this might involve recognizing specific shapes, patterns, or codes. For virtual anchors, it may involve identifying spatial coordinates or alignment with virtual objects.

[0070] Once the anchor is recognized, the AR system may perform spatial mapping to understand the anchor's location in relation to the surrounding physical environment. This mapping is relevant for accurately overlaying digital content. The system also determines the anchor's orientation. This includes its angle, tilt, and rotation relative to the user and the environment, which is relevant for correctly aligning the AR content. The AR system may involve depth sensing to gauge the distance between the anchor and the AR device. This helps in adjusting the size and perspective of the overlaid digital content.

[0071] Once the anchor is recognized, the user receives a confirmation, which could be a visual cue on the display or an auditory signal, indicating that the system is ready to proceed. If the anchor is not properly recognized or aligned, the system alerts the user and may provide guidance for repositioning the anchor.

[0072] The AR system continuously adjusts its recognition and mapping based on any changes in the anchor's position or the user's viewpoint, maintaining the accuracy of the digital overlay. In some embodiments, the work subject may be mobile, under motion or moved. In such embodiments, the AR system is equipped to track the position of the anchor in real time. This means that if the work subject or the user moves, the system can dynamically adjust its recognition and mapping algorithms. As the work subject or the user moves, the AR system continuously recalibrates the overlay of digital content to maintain its correct position and orientation relative to where the position of the anchor would be. This ensures that the overlaid information remains accurate and useful, even in a dynamic setting. The AR system also accounts for changes in the user’s position and viewpoint. As the user moves around the work subject, the system adjusts the digital content to remain consistent with the user’s perspective.

[0073] Flowchart 200 further includes, at 208, work instruction execution. After the anchor is recognized, the work instruction may begin. The user controls the pace and the sequence of the work instruction. They can start the instruction, pause, proceed to the next step, or revisitprevious steps at their convenience. This flexibility allows users to learn and work at a pace that suits their comfort and understanding levels. User interacts with the digital content to navigate through the work instructions. The user interacts with the AR system to navigate through the work instructions. This interaction could involve gestures, voice commands, touch inputs, or using controls on the AR device.

[0074] Each step of the work instruction is associated with specific digital content, predetermined in the system's mapping. For each step, as the user selects to begin the step, the AR system identifies what digital content should be overlaid based on pre-defined mapping associated with each step and the anchor. The mapping may also include information about the size and scale of the digital content relative to the anchor. Based on the anchor's position and orientation, the AR system renders the associated digital content in real-time. This rendering process involves calculating the appropriate location and orientation for the digital content to ensure it aligns correctly with the anchor in the user's view. The rendered digital content is then overlaid onto the user's view through the AR device's display. The system constantly adjusts the perspective of the digital content based on the user’s position and viewing angle, providing a consistent and immersive AR experience. Users can move around the digital overlay, examining different aspects of the content or viewing it from different angles. Despite the user’s movement, the AR system ensures that the digital content remains stable and correctly positioned relative to the physical environment.

[0075] Flowchart 200 further includes, at 210, user interaction. Users can interact with the overlaid digital content through a variety of interaction modes. For example, users can use hand gestures to interact with the digital content. This could include swiping to navigate through different steps of the instruction, pointing or grabbing to select items, or gestural commands for specific actions. Users can interact with the digital content through touch. This can involve tapping, dragging, or pinching to zoom in on specific areas. Further users can interact with the AR system using spoken commands, which is particularly useful when handsfree operation is needed or in environments where physical interaction is not feasible. In some AR systems, users might use controllers, joysticks, or other peripheral devices to interact with the digital content. These devices can offer a more tactile experience and can be especially useful for detailed or complex interactions.

[0076] The AR system provides real-time feedback in response to user interactions. For example, if a user resizes a digital component, the change is reflected instantly in their view. As users interact with the content, the AR system dynamically adjusts the display. Thisincludes changing the orientation, size, and position of digital elements to align with the user's perspective and actions. The system can recognize the context of the user's actions and provide relevant information or options. For instance, selecting a component might bring up detailed information about it or instructions related to it.

[0077] In some embodiments, users can manipulate digital objects in a 3D space, such as rotating them to view different sides, assembling parts in a virtual simulation, or conducting virtual repairs. Users can further query digital content for more information. This can be done through selecting items or using voice commands to ask questions, with the system providing detailed responses or additional data. Users might have the ability to add annotations or notes to the digital content, which can be saved for future reference or shared with other users.

[0078] Flowchart 200 may further include, at 212, stabilization and tracking updates. To maintain the alignment between the anchor and the digital content, the AR system continues to remember the anchor's position and orientation in real-time. If the work subject moves or the AR device moves, the system updates the position and orientation of the digital content accordingly.

[0079] After the anchor's initial recognition and activation, the AR system retains a virtual memory of its position and orientation. This reference point is relevant for maintaining the alignment of digital content within the user's environment. The AR system continuously references this spatial memory in real time. This capability allows the system to adapt the digital overlay based on the remembered position and orientation of the now-invisible anchor. If the work subject to which the anchor's position was related moves, the AR system updates the digital content's position to maintain alignment with the new location of the work subject. As the user moves around with the AR device, the system recalibrates the digital content relative to the remembered anchor point. This ensures that the content remains correctly oriented and positioned from the user's perspective. The system may employ advanced mapping algorithms to virtually track the remembered anchor point, allowing it to accurately overlay digital content in the physical space.

[0080] The AR system continuously monitors the integrity of the virtual anchor point. If it detects that the reference point has been compromised or lost - for instance, due to significant movement of the work subject or the user - it initiates a notification protocol. In such cases, the system alerts the user that the anchor point needs to be reset. This alert can be a visual prompt, an auditory signal, or a combination of both, depending on the user's settings and the nature of the work environment. Upon receiving the alert, the user is guided through theprocess of resetting the anchor point. This process may resemble the initial setup, where the user is instructed to reposition and reactivate the anchor at a suitable location. Maintaining the integrity of the anchor point is needed forthe accuracy ofthe digital overlay. If the anchor point is lost or significantly displaced, the overlaid digital content may no longer align correctly with the physical environment. The ability to reset the anchor point ensures that the AR system can maintain consistency in the instructional content, even in dynamic or unpredictable environments.

[0081] In the case that multiple reference points are established for a work instruction, the AR system may dynamically select one or more reference points for generating and overlaying the digital content as described herein. With multiple interrelated reference points activated, the digital content overlay can be adjusted and aligned with higher precision. This is particularly beneficial in scenarios where the work subject is extensive or has complex geometrical features. This approach also introduces a level of redundancy. If one reference point fails or becomes less effective (e.g., due to being obscured or moved), the system can rely on other points to maintain the accuracy of the digital overlay. The AR system employs intelligent algorithms to manage these reference points efficiently, deciding which combinations to activate based on the user’s location, task requirements, and environmental factors as described herein.

[0082] Users can confirm the completion 214 of the task through the AR system. This might involve checking off completed steps or using a ‘complete task’ button or command. Confirmation of task completion can also trigger the system to log the completion status, which can be important for tracking progress and maintaining records. Users can directly provide feedback within the AR system interface upon completing the work instruction. This could be facilitated through a feedback form, voice input, or interactive prompts within the AR environment.

[0083] The user can conclude 216 the AR session, and the digital content is removed from the physical environment. Users have the ability to actively end the AR session. This can be done through a clear command or action, such as selecting a ‘end session’ button or giving a voice command. The system can present the user with a list or menu of available work instructions, allowing them to easily choose the next task. The AR system’s interface facilitates easy navigation between different work instructions. Users can browse, preview, and select from various instructions efficiently.

[0084] In some embodiments, the AR system may dynamically manage the execution of thework instructions, including transitions between steps. The AR system may continuously monitor the user’s progress in executing the work instruction. By analyzing the user’s environment and interaction, the AR system can understand when a step is completed, thereby dynamically moving to the next step. The AR system may dynamically update the overlay content and guidance accordingly.

[0085] In some embodiments, the AR system continually tracks and records user actions and interactions at each step through sensors and user interface. This includes monitoring the user's movements, actions performed on physical objects, and interactions with the digital overlay. The AR system can analyze this data to assess the user's progress. It considers one or more factors like the accuracy of completed actions, time spent on each step, response to overlay instructions, and overall task engagement. Based on the analysis, the system may make dynamic adjustments in executing the work instruction. For example, if the user struggles with a particular step, the system might provide additional guidance or repeat instructions. Conversely, if the user is progressing smoothly, the system could offer more advanced tips or accelerate the pace of execution. In some embodiments, the AR system validates the completion by cross-referencing the user's actions with the expected outcomes of the steps of the work instruction. Upon task completion, the AR system can provide a summary of the user's performance. This includes highlights of well-executed steps, areas for improvement, and overall efficiency.

[0086] FIG. 3 illustrates an apparatus that may perform any or all of operations of the above methods and features explicitly or implicitly described herein, according to different embodiments. Apparatus 300 may be capable of performing operations in various AR environments as described herein. This apparatus, which could range from a network-enabled computer to user equipment like smartphones or specialized AR devices, such as an AR headset or smart glasses, is designed to adapt to a multitude of AR scenarios. In some embodiments, apparatus 300 can be viewed as an assembly of hardware components and a dynamic platform capable of integrating and managing various elements of the AR system. It can act as the central unit in scenarios involving complex AR interactions, such as those requiring real-time environmental data processing, anchor recognition, and seamless integration of digital overlays with the user's environment.

[0087] In some embodiments, a computer equipped with network function may be configured as the apparatus 300. In some aspect, apparatus 300 can be a device that connects to the network infrastructure over a radio interface, such as a mobile phone, smartphone or other such device that may be classified as user equipment (UE). In some aspects, apparatus 300 is a Machine Type Communications (MTC) device (also referred to as a machine-to-machine (m2m) device), or another such device that is categorized as a UE despite not providing a direct service to a user. In some aspects, apparatus 300 is component, module, device used to implement one or more components, modules, systems, mechanisms according to one or more aspects described herein.

[0088] As shown, the apparatus 300 may include one or more of: a processor 310, such as a central processing unit (CPU) or specialized processors such as a graphics processing unit (GPU) or other such processor unit, memory 320, non-transitory mass storage 330, inputoutput (I / O) interface 340, network interface 350, and a transceiver 360, all of which are communicatively coupled via bi-directional bus 370.

[0089] The I / O interface includes one or more components including a display system 342, audio system 344, sensor 346, and user interface 348. The display system 342 can project digital content into the user's field of view. This can be through transparent lenses for mixed reality or opaque displays for full AR experiences.

[0090] Display system 342 outputs the digital overlay onto the user's visual field, relevant for displaying work instructions and augmented content. Display system 342 refers to any system of visually presenting digital content to the user. The display system is the output interface through which the user views the augmented reality. Audio system 344 handles all audio-related tasks, both receiving input (voice commands) and delivering output (sounds and instructions).

[0091] Sensors 346 capture real-time environmental data, including images and depth information, relevant for contextual awareness within the AR experience and forming the basis for generating the AR overlay, for environment mapping and anchor tracking. These sensors can be a variety of types including physical controls (buttons, touchpads), optical sensors (including depth sensors) for visual data, auditory sensors for sound, touch sensors, gesture and eye tracking, motion sensors (including accelerometers and gyroscopes) for movement and orientation, environmental sensors for ambient conditions, and proximity sensors for assessing spatial relationships and distances, enabling accurate overlay of digital content onto the real world. Sensors 346 may track user gestures and eye movements for intuitive interaction with AR content and includes advanced sensors for precise tracking and seamless user interaction.

[0092] The user interfaces 348 represent interfaces through which the user interacts with the AR system and include interfaces for gesture control, voice commands, and touchscreen inputs, among others. These interfaces allow the user to interact with and control the AR experience. These interfaces are relevant for ensuring that the AR system is interactive and responsive to the user’s actions. User interactions or inputs through the user interfaces are fed back to the processing, influencing real-time adjustments in the AR content.

[0093] Processor 310 is responsible for processing all operations. This includes interpreting sensor data to identify anchor positions, overlaying digital content based on those positions, and dynamically adjusting content based on user interactions and movements, as described herein. Processor 310 processes and interpreting the sensor data to create a cohesive AR experience, integrating digital content with the real world. The processed and contextualized AR content (the digital overlay content) is then sent to the display system 342, where it is overlaid onto the user’s view of the real world for user interaction. In some embodiments processor 310 may refer to a cloud-based processor where data is sent to a cloud server and the results are sent back to the apparatus.

[0094] Transceiver 360 facilitates wireless communication, essential for sending and receiving data, especially in scenarios where remote assistance or real-time collaboration is involved. Transceiver 360 may include one or multiple antennas. In some embodiments, one or more of the depicted elements may be utilized, or only a subset of the elements. Further, apparatus 300 may contain multiple instances of certain elements, such as multiple processors, memories, or transceivers. Also, elements of the hardware device may be directly coupled to other elements without the bi-directional bus. Additionally, or alternatively to a processor and memory, other electronics or processing electronics, such as integrated circuits, application specific integrated circuits, field programmable gate arrays, digital circuitry, analog circuitry, chips, dies, multichip modules, substrates or the like, or a combination thereof may be employed for performing the required logical operations.

[0095] The memory 320 may store temporary data and instructions necessary for the processor's operations. This includes storing algorithms for anchor recognition, digital content mapping, and real-time user interaction data. Memory 320 may include any type of non- transitory memory such as static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), any combination of such, or the like. The mass storage element 330 may include any type of non-transitory storage device, such as a solid-state drive, hard disk drive, a magnetic disk drive, an opticaldisk drive, USB drive, or any computer program product configured to store data and machine executable program code. According to certain aspects, the memory 320 or mass storage 330 may have recorded thereon statements and instructions executable by the processor 310 for performing any method operations described herein. Memory 320 may store executable instructions and temporary data required for the operation of the apparatus 300 including identifying anchor positions and generating digital content. Mass storage 330 may provide long-term storage of digital content, software, and other data necessary for the apparatus’ functionality, including storing work instructions and digital overlays. For example, mass storage 330 may house long-term data including a library of work instructions, digital overlays, and historical user interaction data. It enables the retrieval of specific work instructions and associated digital content for overlay.

[0096] The processor 310 and memory 320 may function together as a chipset which may be provided together for installation into wireless communication apparatus 300 in order to implement wireless local area network (WLAN) functionality. The chipset may be configured to receive as input data including but not limited to physical protocol data unit (PPDUs) from the network interface 350. The chipset may be configured to output data including but not limited to PPDUs to the network interface 350. Network interface 350 provides connectivity for data synchronization, software updates, and potentially cloud-based processing. This is crucial for maintaining current AR content and enabling collaborative AR experiences. Network interface 350 can include Wi-Fi, Bluetooth, and potentially cellular connectivity for online functionalities.

[0097] In some embodiments, network interface 350 can be used to have multiple users participate in executing a work instruction via a collaborative session. A collaborative session, such as a shared environment, can be shared among multiple users. Each session may have unique identifier that allows for accessing the session. This shared space can be a virtual representation of a real-world environment, and all users can see and interact with this same environment. Each user may access the shared environment via the network interface 350.

[0098] In some embodiments, the apparatus 300 can be implemented via one or multiple devices. In some embodiments the apparatus 300 can integrate blockchain technology for credentialing and performance tracking. The apparatus 300 may include or have access to a blockchain management module. Blockchain technology can be used to securely record and verify the skills and milestones achieved by the user during the execution of the work instruction. This could be beneficial for certification and tracking professional development.This integration involves recording each user’s progress (including user interaction, performance) and achievements securely on the blockchain after completion of each step of a work instruction. User performance can include metrics such as completion time, accuracy, and any specific skills demonstrated. As a result, an immutable and transparent record of milestones can be made, and verifiable digital credentials can be generated. This can be particularly beneficial for professional development, where accurate tracking of skills and competencies is crucial. Beyond credentialing, blockchain can be used to enhance security, user privacy, and enable decentralized control over training content, offering a range of possibilities AR-based learning systems.

[0099] As may be appreciated, integration of blockchain for credentialing can enhance the certification process in various fields, including trades like the Red Seal program, health industry certifications, and other professional accreditations. For certifications like Red Seal, this approach ensures that essential skill and knowledge area is recorded and verified. In the health industry, such a system can track detailed procedural skills or compliance with critical protocols, providing a reliable basis for professional certification. The blockchain's immutable and transparent nature offers a robust and trustworthy platform for credentialing, ensuring that the qualifications gained via these AR training methods are credible and recognized across various sectors. This integration not only adds value to the training process but also to the credentials earned, paving the way for their wide acceptance in industry and professional circles.

[0100] FIG. 4 illustrates a method for facilitating interactive learning experience, according to an embodiment. The method 400 includes identifying 401 a position of an anchor in a physical environment of a work subject. The anchor corresponds to a work instruction related to the work subject. The method 400 further includes 402 generating digital content to be overlaid on the physical environment of the work subject based on the position of the anchor. The digital content corresponds to the work instruction.

[0101] In some embodiments, identifying the position of the anchor in the physical environment of the work subject includes identifying a location and an orientation of the anchor in the physical environment of the work subject. In some embodiments, generating the digital content to be overlaid on the physical environment of the work subject based on the position of the anchor includes generating the digital contentto be overlaid on the physical environment of the work subject based on the location and the orientation of the anchor.

[0102] In some embodiments, the digital content when overlaid on the physical environmentof the work subject maintains a spatial relationship with the anchor. In some embodiments, the digital content when overlaid on the physical environment of the work subject maintains a spatial relationship with the anchor based on the location and the orientation of the anchor.

[0103] In some embodiments, identifying the position of the anchor in the physical environment of the work subject comprises identifying multiple reference positions each corresponding to a different anchor of multiple anchors. In some embodiments, generating the digital content to be overlaid on the physical environment of the work subject based on the position of the anchor includes generating the digital content to be overlaid on the physical environment based on one or more reference positions of the multiple reference positions.

[0104] In some embodiments, generating the digital content to be overlaid on the physical environment based on one or more reference positions of the multiple reference positions includes generating and overlaying the digital content based one or more thresholds including: a time threshold, a number of reference positions, and an overlay accuracy. In some embodiments, generating the digital content to be overlaid on the physical environment based on one or more reference positions of the multiple reference positions includes comparing an overlay of the digital content corresponding to the one or more reference positions with another overlay of the digital content corresponding to one or more different reference positions of the multiple reference positions.

[0105] In some embodiments, the method further includes selecting the work instruction from a list of work instructions related to the work subject. In some embodiments, the anchor is part of or attaches to the work subject. In some embodiments, the anchor is separate from the work subject.

[0106] Aspects of the present disclosure can be implemented using electronics hardware, software, or a combination thereof. In some aspects, this is implemented by one or multiple computer processors executing program instructions stored in memory. In some aspects, the invention is implemented partially or fully in hardware, for example using one or more field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs) to rapidly perform processing operations.

[0107] It will be appreciated that, although specific embodiments of the technology have been described herein for purposes of illustration, various modifications may be made without departing from the scope of the technology. The specification and drawings are, accordingly, to be regarded simply as an illustration of the invention as defined by the appended claims,and are contemplated to cover any and all modifications, variations, combinations or equivalents that fall within the scope of the present invention. In particular, it is within the scope of the technology to provide a computer program product or program element, or a program storage or memory device such as a magnetic or optical wire, tape or disc, or the like, for storing signals readable by a machine, for controlling the operation of a computer according to the method of the technology and / or to structure some or all of its components in accordance with the system of the technology.

[0108] Acts associated with the method described herein can be implemented as coded instructions in a computer program product. In other words, the computer program product is a computer-readable medium upon which software code is recorded to execute the method when the computer program product is loaded into memory and executed on the microprocessor of the wireless communication device.

[0109] Further, each operation of the method may be executed on any computing device, such as a personal computer, server, PDA, or the like and pursuant to one or more, or a part of one or more, program elements, modules or objects generated from any programming language, such as C++, Java, or the like. In addition, each operation, or a file or object or the like implementing each said operation, may be executed by special purpose hardware or a circuit module designed for that purpose.

[0110] Through the descriptions of the preceding embodiments, the present invention may be implemented by using hardware only or by using software and a necessary universal hardware platform. Based on such understandings, the technical solution of the present invention may be embodied in the form of a software product. The software product may be stored in a non-volatile or non-transitory storage medium, which can be a compact disk readonly memory (CD-ROM), USB flash disk, or a removable hard disk. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to execute the methods provided in the embodiments of the present invention. For example, such an execution may correspond to a simulation of the logical operations as described herein. The software product may additionally or alternatively include number of instructions that enable a computer device to execute operations for configuring or programming a digital logic apparatus in accordance with embodiments of the present invention.

[0111] Although the present invention has been described with reference to specific features and embodiments thereof, it is evident that various modifications and combinations can bemade thereto without departing from the invention. The specification and drawings are, accordingly, to be regarded simply as an illustration of the invention as defined by the appended claims, and are contemplated to cover any and all modifications, variations, combinations or equivalents that fall within the scope of the present invention.

Claims

WHAT IS CLAIMED IS1. A method for facilitating interactive learning experience through use of an interactive work instruction, the method comprising: identifying a position of an anchor in a physical environment of a work subject, the anchor corresponding to a work instruction related to the work subject; and generating digital content to be overlaid on the physical environment of the work subject based on the position of the anchor, the digital content corresponding to the work instruction.

2. The method of claim 1 , wherein identifying the position of the anchor in the physical environment of the work subject comprises identifying a location and an orientation of the anchor in the physical environment of the work subject.

3. The method of claim 2, wherein generating the digital content to be overlaid on the physical environment of the work subject based on the position of the anchor comprises generating the digital content to be overlaid on the physical environment of the work subject based on the location and the orientation of the anchor.

4. The method of any one of claims 1 to 3, wherein the digital content when overlaid on the physical environment of the work subject maintains a spatial relationship with the anchor.

5. The method of claim 3, wherein the digital content when overlaid on the physical environment of the work subject maintains a spatial relationship with the anchor based on the location and the orientation of the anchor.

6. The method of any one of claims 1 to 5 further comprising selecting the work instruction from a list of work instructions related to the work subject.

7. The method of any one of claims 1 to 6, wherein the anchor is part of or attaches to the work subject.

8. The method of any one of claims 1 to 6, wherein the anchor is separate from the work subject.

9. The method of any one of claims 1 to 6, wherein: identifying the position of the anchor in the physical environment of the work subject comprises identifying multiple reference positions each corresponding to a different anchor of multiple anchors; and generating the digital content to be overlaid on the physical environment of the work subject based on the position of the anchor comprises generating the digital content to be overlaid on the physical environment based on one or more reference positions of the multiple reference positions.

10. The method of claim 9 wherein generating the digital content to be overlaid on the physical environment based on one or more reference positions of the multiple reference positions comprises generating and overlaying the digital content based one or more thresholds including: a time threshold, a number of reference positions, and an overlay accuracy.11 . The method of claim 9 wherein generating the digital content to be overlaid on the physical environment based on one or more reference positions of the multiple reference positions comprises comparing an overlay of the digital content corresponding to the one or more reference positions with another overlay of the digital content corresponding to one or more different reference positions of the multiple reference positions.

12. An apparatus comprising at least one processor and at least one non-transitory machine- readable medium storing executable instructions which when executed by the at least one processor configure the apparatus to perform any of method claims 1 to 11.

13. An augmented reality system for facilitating an interactive learning experience through use of interactive work instruction related to a work subject, the system comprising: an anchor, the anchor being a spatial reference point within the physical environment of the work subject for superimposition of digital content related to the work instruction; a display system for dynamically projecting in real-time the digital content related to the work instruction to a user’s field of view, the digital content being superimposed onto the user's real-world view of the physical environment related to the work subject based on the position of the anchor; a user interface configured to facilitate user interaction with the work instruction,one or more sensors configured to capture environmental data in real-time, wherein the environmental data includes the position of the anchor within the physical environment of the work subject.

14. The system of claim 13, further comprising an audio system configured to receive audio input including user voice commands and deliver audio outputs including sound prompts and instructions.

15. The system of claim 13 or 14, wherein the anchor is part of the work subject.

16. The system of claim 13 or 14, wherein the anchor is attached to the work subject.

17. The system of claim 13 or 14, wherein the anchor is a QR code attached to a component of the work subject.

18. The system of any one of claims 13 to 17, further comprising one or more sensors configured to track user gestures and eye movements.

19. The system of any one of claims 13 to 18, wherein the user interaction with the work instruction comprises navigation through the work instructions.

20. The system of any one of claims 13 to 18, wherein the user interaction with the work instruction comprises one or more of gestures, voice commands, and touchscreen inputs.21 . The system of any one of claims 13 to 18, wherein the user interaction with the work instruction comprises use of controllers, joysticks, or other peripheral devices to interact with the digital content.

22. The system of any one of claims 13 to 21 , wherein the system provides real-time feedback in response to user interactions.

23. The system of any one of claims 13 to 22, further comprising a mass storage configured to store a library of work instructions.

Images