Method for anchoring and aligning of augmented reality virtual content with the dynamic open-sea environment for maritime and naval operational training

Abstract

The present invention relates to a method for anchoring and aligning of virtual content using augmented reality (AR) technology for the construction of simulators for maritime and naval operations on board of ship bridges. The simulator enables the execution of training scenarios with virtual objects and content while maintaining visual contact with the real environment, whether the ship is sailing in open-sea or moored alongside a berth. In particular, the present invention describes a two-layer anchoring and alignment method, which serves as a technique for accurately synchronizing virtual objects and content with the real environment during vessel movement relative to true north, while also compensating for the user's movements, allowing realistic display through the user's augmented reality glasses.

Description

[0001] METHOD FOR ANCHORING AND ALIGNING OF AUGMENTED REALITY VIRTUAL CONTENT WITH THE DYNAMIC OPEN-SEA ENVIROMENT FOR MARITIME AND NAVAL OPERATIONAL TRAINING

[0002] TECHNICAL FIELD OF THE INVENTION

[0003] The present invention relates generally to augmented reality (AR) technology and in particular to a method for ensuring accurate anchoring, alignment, and stabilization of AR virtual content within dynamic maritime training environments. Further, the present invention provides a method capable of continuously compensating in real-time for any relative motion, including movements of a vessel operating in the open sea, movements of a stationary vessel at anchor subjected to simulated conditions, or movements of an operator whether located on a real vessel or located within a land-based simulator room.

[0004] The method according to the present invention ensures the stable and precise alignment of virtual content, such as ships, aircraft, missiles, coastal features, and other objects and backgrounds, independent of real-time movements of the vessel and / or the operator. As a result, the present invention delivers highly immersive, realistic, and lifelike training experiences directly comparable to real operational maritime and naval scenarios, yet without incurring the significant operational costs and logistical complexities typically associated with deploying actual maritime assets for training purposes.

[0005] BACKGROUND OF THE INVENTION

[0006] Augmented reality (AR) technology is widely recognized and has undergone extensive development in recent decades. Its application commonly involves the use of specialized glasses or display devices, as described in the publication Practical Augmented Reality by Steve Aukstakalnis (2016). Head-mounted augmented reality (AR) devices enhance the user’s perception of the physical environment by overlaying it with digital, computer-generated content. This digital information (referred to collectively as virtual content) may include alphanumeric data, symbolic graphics, or fully rendered objects. The main goal of AR is the alignment, correlation, and stabilization of such virtual content within the user's real-world view, in a manner that is spatially contextual, perceptually coherent, and responsive to environmental and userspecific variables.

[0007] There are primarily two categories of head-mounted AR glasses:

[0008] 1) Optical See-Through: Users perceive the real world directly through monocular or binocular optical elements, such as holographic waveguides, onto which virtual content is projected and superimposed. Notably, Optical See-Through glasses may also be equipped with cameras to support specific tasks such as image recognition, spatial tracking, or photogrammetry, including those processes described in the present method.

[0009] 2) Video See-Through: Said systems capture the external environment via one or more cameras mounted on the front of the display. These captured images are digitally combined with virtual elements and then presented on internal screens to the user.

[0010] A significant contribution to AR anchoring methodology is described in the work of Benjamin Nuemberger, Eyal Ofek, Hrvoje Benko, and Andrew Wilson, Snap To Reality: Aligning Augmented Reality to the Real World (2016). This method outlines a framework for spatial correlation between virtual and physical content, utilizing fixed objects in the real environment as reference points. Such static anchors allow virtual content to be correctly positioned and oriented within indoor or controlled outdoor environments. Moreover, US 9,384,594 B2, describes a system for anchoring virtual images to real-world surfaces in augmented reality (AR) environments using head-mounted devices (HMDs). The anchoring process involves capturing spatial and image data via stereo cameras and sensor arrays embedded in the HMD. Users can designate real- orld surfaces as anchor points using gestures, voice commands, or other input modalities. The system calculates the orientation and distance between the device and the selected surface, then renders the virtual object so that it appears to be naturally affixed to that surface. This anchoring can be either flush with or offset from the surface and can be dynamically updated as the user moves. This technology supports switching anchor surfaces and maintains object coherence across movements.

[0011] The snapping to reality process conceptually works as follows: a. Detection of a Real-World Feature. The AR system begins by scanning and interpreting the real-world environment using a combination of sensing and computer vision techniques. This allows it to detect key reference points like window frames, table edges, or walls. Key detection methods include: Plane Detection (Identifies flat surfaces such as floors or walls), Feature Point Clouds (Uses visual features to track unique points in space), Depth Sensors (Capture spatial geometry with depth accuracy), Semantic Understanding (Al-powered models recognize and label elements, e.g., “this is a window”). b. Anchoring a Virtual Line. Once a reliable physical feature is detected, the AR system creates a virtual line or axis that is mathematically aligned with that feature, such as a vertical line aligned with the edge of a window frame, a horizontal line along the surface of a desk etc. This virtual line becomes a stable reference for placing virtual content. c. Snapping Virtual Objects to the Line. Virtual content (e.g., 3D objects, data overlays) is then snapped to the anchored virtual line. This ensures spatial consistency and realism.

[0012] However, while effective in static settings such as architectural interiors or public spaces (e.g., buildings, rooms, or open squares), this anchoring method is not suitable for dynamic environments like the open sea, where static reference points are inherently absent. This limitation becomes critical in the context of constructing realistic maritime training simulators that rely on AR technologies. Specifically, when using AR glasses or binocular devices on a moving ship, existing methods fail to maintain consistent and realistic positioning of virtual content relative to the actual environment, particularly when the ship or the operator’s head moves or rotates relative to true north.

[0013] In the maritime domain, augmented reality technologies have been explored primarily in the context of navigational enhancement, rather than immersive operational training. According to the study by Nordby, K., Gemez, E., Frydenberg, S., and Eikenes, J. O., titled Augmenting OpenBridge: An Open User Interface Architecture for Augmented Reality Applications on Ship Bridges (2020, The Oslo School of Architecture and Design), AR can play a supportive role for ship navigators by overlaying data from bridge systems directly into their primary field of view. This integration allows the navigator to monitor and interact with various shipboard systems (such as radar, electronic charts, or speed indicators) without diverting attention from the external environment.

[0014] AR applications of this kind enhance situational awareness and operational performance by reducing cognitive load, minimizing the need for task-switching, and seamlessly linking real- world visuals with digital navigation aids. This approach is comparable in function and concept to heads-up display (HUD) systems, which were first developed for aviation and later adopted in automotive industries. A relevant example is Mercedes-Benz’s implementation of augmented reality head-up displays in its vehicle lineup, as described in the 2024 manual for Mercedes-Benz passengercars titled “Function of the Head-Up Display with Augmented Reality”. Such systems demonstrate the value of AR in increasing operational efficiency in transportation sectors. However, they are designed exclusively for navigational support and not for the simulation or training of complex maritime operations. These applications do not address the challenges of realistic content placement in dynamically shifting environments like the open sea, nor do they accommodate the movement of both the platform (ship) and the operator (user wearing the AR system) within immersive training contexts. They remain confined to static overlay zones, often within the physical bridge structure, and are not capable of rendering lifelike external threats or simulated combat scenarios that demand high spatial realism and interaction.

[0015] A notable implementation of augmented reality in the maritime environment is the OpenBridge user interface architecture, which facilitates the integration of maritime applications into headmounted display (HMD) AR systems, such as the Microsoft HoloLens or Magic Leap. OpenBridge focuses on enhancing navigational situational awareness by displaying navigational data in spatial context, aligned with the ship's bridge environment and operational layout.

[0016] Within OpenBridge, AR visualizations are categorized into five primary types of digital information: a. App Display: Provides a full application view, such as Electronic Chart Display and Information Systems (ECDIS) or radar systems. b. Widget Display: Displays individual components of a larger application, including indicators like compasses or speed gauges. c. AR Map: Projects geospatial information on a 2D or 3D map frame that aligns with the real- world environment. d. Annotation: Offers contextual data linked to real-world objects; for instance, providing details about vessels visible from the bridge. e. Ocean Overlay: Superimposes digital information directly onto the ocean’s surface, aligning AR graphics with the real location of points or areas of interest.

[0017] These AR representations are restricted to designated AR Zones within the bridge, which are predefined areas optimized for the display of digital information. Zones are strategically positioned to support the navigator's task workflow and are bound by masked areas where virtual content is deliberately suppressed to avoid interference with critical real-world visibility, such as views through bridge windows or structural blind spot. Virtual content in OpenBridge is generally confined within the boundaries of the bridge interior. Even when annotations and overlays correspond to external objects or environments, their graphical representation often appears spatially tethered to the ship’s interior architecture (such as control consoles or window frames) thereby reducing the sense of external immersion required for realistic operational training.

[0018] Specifically, when virtual information is anchored to real-world objects — as annotations on ships or other external elements — the structural areas of the ship bridge may interfere. When a physical structure obstructs the line of sight, the virtual annotations do not disappear. Instead, they remain visible in the user's field of view, crossing from the external environment into the interior of the bridge. This significantly compromises the realism of virtual content that is meant to appear as part of the outside world, since the information does not obey occlusion rules. Rather than being hidden by real-world structures, the annotations are superimposed over them, making outside elements appear inside the bridge. However, the OpenBridge system lacks of spatial consistency, and therefore, the OpenBridge may serve only for the support of the navigator, and it is not suitable for the creation of a realistic training environment.

[0019] Summarizing, the primary objective of OpenBridge is to improve the functionality of the bridge workspace by enhancing the display and accessibility of navigational aids. The system integrates live data from operational systems such as radar, ECDIS, and the Automatic Identification System (AIS), and represents this information as virtual content associated with real-world targets, positions, or features.

[0020] However, despite its advanced integration capabilities, the OpenBridge architecture remains focused on supplementing routine navigational awareness rather than simulating complex operational or combat scenarios. The system is not intended to, nor capable of, simulating threats, high-tempo operations, or life-like engagement conditions outside of a navigational support context.

[0021] The purpose of the present invention is to address the limitations of existing AR systems in maritime environments by introducing a novel method for the anchoring and alignment of virtual content with the real world. This method enables the development and deployment of maritime and naval operational training simulators directly on the bridge of any vessel and not in a simulator room.

[0022] Unlike prior AR implementations focused on navigational aid and system display within confined bridge spaces, the present invention supports the realistic and immersive training of personnel through the dynamic rendering of external virtual elements (including vessels, aircraft, incoming munitions, explosions, coastal terrain, and other). These virtual elements are viewed through augmented reality glasses as if they exist in the natural seascape beyond the bridge windows, without any visual interference from or overlay onto the bridge’s internal structure.

[0023] The present invention is not intended to enhance the display or interpretation of navigation data. Rather, its core objective is to enable true-to-life training environments by projecting operationally relevant virtual content into the real-world visual field of the user. Through this approach, it allows for continuous and realistic simulation of complex maritime scenarios, ensuring that personnel remain mission-ready, even when real-world assets and live exercises are not feasible or cost-effective.

[0024] The present invention offers a substantial advancement over the existing capabilities of augmented and virtual reality technologies, particularly those used in static, classroom-based simulators. By enabling the direct application of AR-based training simulators on the bridges of operational ships (whether underway at sea or stationary in port) the present invention introduces a new paradigm in maritime and naval training effectiveness. The advantages of this method of the present invention include:

[0025] - Creation of a Realistic Training Environment: The present invention allows for the construction of a highly immersive and operationally relevant training environment directly onboard of any vessel. Unlike traditional land-based simulators, this approach integrates actual shipboard systems and surroundings, preserving the authenticity of spatial orientation, sensory input, and environmental context.

[0026] - Significant Resource Savings: The need for physical training assets (such as aircraft, ships, or specialized vessels) is eliminated. Instead, virtual equivalents can be generated on demand and in tailored configurations, dramatically reducing training costs and logistical burdens while expanding the range and complexity of training scenarios.

[0027] - Training in Realistic Combat Conditions: Operators experience the full spectrum of visual and auditory stimuli encountered in real combat situations, including realistic depictions of explosions and the simulated impact of ordnance on or near the ship. This sensory immersion enhances stress conditioning, threat recognition, and decision-making under pressure. - Provision of a ‘Mission-Ready’ Training Standard: Unlike conventional simulators that merely prepare crews to begin live training at sea, the method according to the present invention allows for immediate operational readiness. Trainees become proficient in executing tactical tasks and command procedures under realistic conditions, increasing the effectiveness and safety of future real-world deployments.

[0028] - Versatility Across Dynamic Environments: While specifically tailored for maritime and naval bridge operations, the underlying method may be adapted for other high-intensity training contexts, such as surgical procedures or other mission-critical environments where real-time responsiveness and environmental immersion are critical.

[0029] - Operability in Any Setting: The present method is equally effective while the vessel is underway, anchored, or moored alongside a pier. In stationary cases, simulated vessel motion is introduced to preserve the continuity of the operational training environment, ensuring the same level of realism and utility regardless of physical location. Moreover, the method can also be applied in land-based simulator rooms, where the AR display output can be integrated with the simulator engine. In this configuration, the system can dynamically adjust the behavior and positioning of simulated artifacts in response to the movements and orientation of the user wearing the AR headset, preserving a fully interactive and immersive experience.

[0030] These benefits collectively position the method according to the present invention as a transformative solution in the field of maritime training, addressing long-standing limitations of prior AR technologies and simulators by delivering a dynamic, scalable, and resource-efficient platform for real-world preparedness.

[0031] SUMMARY OF THE INVENTION

[0032] The present invention provides a method called Two-Level Anchoring / Alignment (TLA), which enables the stable and realistic projection of augmented reality (AR) virtual content within a dynamically changing maritime environment. The TLA method consists of two primary anchoring mechanisms: the internal anchoring, which governs the positioning of virtual content relative to the interior structure of the ship's -bridge, and the external anchoring, which maintains the alignment of virtual objects with the external environment, regardless of the vessel’s motion or the user’s own head and body movement.

[0033] The method according to the present invention ensures that virtual content (such as simulated ships, aircraft, munitions, coastlines, or other) remains visually stable and spatially accurate within the user’s real- world field of view, even under the complex conditions of vessel movement (pitch, roll, yaw, speed, and heading changes) and dynamic user interaction (head orientation, line of sight). This system enables immersive, life-like training experiences without the need for physical assets, and at a fraction of the cost of live operational exercises.

[0034] The virtual content used for the training process via the AR glasses is called External Virtual Objects I Content (EVO-EVC). The term EVO refers to virtual 3D structures like ships, airplanes, missiles, UAVs, US Vs, small boats, humans and other various objects which are positioned within the real world, via the TLA method, creating a real -virtual fused training environment. The term EVC refers to virtual 3D graphics which are positioned upon the real world and create a virtual training environment.

[0035] The TLA method is implemented using the two types of AR glasses:

[0036] The Optical See-Through Augmented Reality headsets are used for the implementation of training scenarios with External Virtual Objects and the Video See-Through AR glasses are used for the implementation of training scenarios with External Virtual Content. Both types of AR Glasses are equipped with the following components:

[0037] - Holographic Processing Unit (HPU): Manages real-time sensor fusion, spatial mapping, and tracking data to maintain precise environmental awareness and content stability.

[0038] - Eye Tracking System: Monitors the user's gaze for interactive functionality and foveated rendering, enhancing performance and accuracy.

[0039] - Microphones and Voice Command Interface: A 3-5 channel microphone array supporting handsfree operation through spoken commands.

[0040] - Spatial Audio System: Open-ear stereo speakers that simulate directional sound, preserving awareness of real-world ambient noise while enhancing AR immersion.

[0041] - Camera and Sensor Suite: Includes a high-resolution RGB front-facing camera (minimum 8 MP), depth sensors, grayscale tracking cameras, and ambient light sensors for spatial detection and environment analysis.

[0042] - Spatial Mapping System: Generates a real-time 3D mesh of the user's surroundings, supporting the physical interaction of virtual objects with real-world geometry and enabling persistent anchoring.

[0043] - Inertial Measurement Unit (IMU): A core component consisting of a gyroscope, accelerometer, and magnetometer, which accurately detects head orientation and movement across pitch, roll, and yaw axes. The IMU data is essential for both horizon alignment and continuous AR content stabilization in relation to the user and vessel movement.

[0044] Through this integrated system, the present invention allows for real-time compensation of both platform and user dynamics, ensuring the consistent and realistic alignment of AR objects and content across various operational conditions, whether the ship is underway, anchored, alongside or in a training room. The result is a highly reliable, cost-efficient, and mission-representative training platform capable of elevating operational preparedness.

[0045] The preferred embodiments of the present invention are defined in claims 1 TO 15.

[0046] Other objects and advantages of the present invention will become apparent to those skilled in the art in view of the following detailed description.

[0047] BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Fig. 1 shows a general block diagram and the main parts of an in-classroom navigational simulator of the present invention.

[0049] Fig. 2 shows a general block diagram and the main parts of the TLA Simulator, for open sea use, based on the block diagram of the in-classroom navigational simulator of the present invention.

[0050] Fig. 3 shows the general block diagram and the main parts of the TLA Simulator, for use alongside, based on the block diagram of the in-classroom navigational simulator and the TLA Simulator of the present invention.

[0051] Fig. 4 shows a depiction of the Bridge Interior Rejection Area and the External AR Area (as seen from within the ship's bridge interior), the Visual Horizon (represented by a dashed line), and the Internal and External Virtual Anchoring lines (represented by vertical black lines). The External Virtual Content is represented as a ship inside the window frame which is part of the External AR Area of the present invention. Fig. 5 shows the partial absorption of the External Virtual Content (presented by a ship), as it overlays the Bridge Interior Rejection Area of the present invention.

[0052] Fig. 6 shows a depiction where the guided missiles remain stable in relation to the ship's true bearing despite its movement from a course of 090° to 045° of the present invention.

[0053] Fig. 7 shows the initialization process and the logic for deriving the anchoring angle of the external virtual content, represented as angle 3, and the logic for determining the sign of angle OA of the present invention.

[0054] Fig. 8 shows a depiction where the External Virtual Content (represented by a ship), remains stable in relation to the ship's true heading and initialization heading despite the movement of the ship 45° to starboard of the present invention.

[0055] Fig. 9 shows the anchoring of targets in relation to the horizon line of the present invention.

[0056] Fig. 10 shows the logic of the stabilization of the External Virtual Content towards the Horizon Line of the present invention.

[0057] Fig. 11 shows the steps of the TLA method implementation of the present invention.

[0058] DETAILED DESCRIPTION OF THE INVENTION

[0059] The bridge is the compartment of the ship where all navigation data is available, such as Speed through water STW (via agilog); Course (via gyrocompass); Pitch and roll (via gyrocompass); GPS position; GPS time; Speed over ground SOG (via GPS); Actual depth (via echo sounder).

[0060] The GPS time, speed over ground, ship's heading, pitch and roll, and horizon recognition are the key elements that feed into and form the basis of the Two-Level Anchoring (TLA) method.

[0061] A standard maritime navigation simulator system can be represented as a block diagram (Fig. 1) comprising six main interconnected components (based on the architectural principles described in IMO Model Course 1.22 and DNV Standard ST-0033). At the core is the Ship Simulator Engine, which synchronizes all modules and processes real-time simulation logic. It receives control inputs from the Control Panel, where the user adjusts heading, speed, and thrusters (if fitted), and sends this data to the Ship Dynamics Model, which simulates the vessel’s physical response (e.g., acceleration, turning, pitch, roll). The updated ship state is passed on to the Visual System, which renders the 3D environment including sea conditions, other vessels, and ports, based on environmental inputs (e.g., weather, time of day). Simultaneously, the Navigation System receives ship data to update the Electronic Chart Display and Information System (ECDIS), radar view, and Automatic Identification System (AIS) overlays. All simulation data flows through to the Instructor Console, which acts as a live monitoring and control interface, allowing an instructor to insert scenario events (e.g., equipment failures, new traffic), adjust conditions, and assess user performance. Each block processes and generates data: controls from the user, sensor feedback from the ship model, environmental conditions, and visual / navigational updates, working together in a real-time closed-loop system.

[0062] The TLA simulator can convert a ship’s bridge into a training simulator for use even when the ship is underway in open sea. It utilizes the basic components, characteristics and functions of the standard maritime navigation simulator but with the following key structural differences (Fig. 2):

[0063] The Control Panel and Ship Dynamic Model are not utilized, as the ship’s actual heading, speed, and dynamic behavior in the real world replace the simulated movement and reactions. The ship’s heading (provided by the gyro), speed through water (from the Agilog system), speed over ground (from the GPS), as well as pitch, roll, and three-dimensional acceleration (also from the gyro) are transmitted to and processed by the TLA Simulation Engine.

[0064] This engine synchronizes all TLA simulator modules and handles real-time simulation logic, seamlessly integrating the real-world data into the virtual AR environment. These data are exchanged as digital values, through serial communication (RS232) or the local network (Ethernet or WiFi) using TCP sockets for real-time communication. The readings of the actual ship’s heading, speed through the water (STW) and over the ground (SOG), and the pitch and roll are the inputs for the implementation of the External Anchoring / Parallelization process in the Horizontal and Vertical plane and the External Anchoring / Stabilization of External Virtual Content, as described in the following paragraphs for the TLA method. The AR Visual System receives the output of the TLA process, and anchors / aligns the virtual content to the real world, rendering the external virtual content through the ship’s glass windows as it is described in the following paragraphs. The AR glasses used for the AR training are components of the AR Visual System using Wi-Fi connection. The Wi-Fi connection between the AR Visual System and the AR glasses enables real-time, two-way communication necessary for an interactive augmented reality experience. Through this wireless link, the AR Visual System sends 3D icons, models, textures, and scene data (external virtual content) to the AR glasses, while the AR glasses continuously transmit sensor information such as head position, orientation, and hand gestures back to the AR Visual System. This allows the system to dynamically update the scene based on user’s movement and interactions. The communication between the AR glasses and the AR Visual System typically uses TCP / IP for reliable data and UDP for low-latency real-time updates, ensuring smooth and synchronized AR performance. All the data for the virtual scenario flows through the Instructor Console, which acts as a live monitoring and control interface — allowing an instructor to insert scenario events (e.g., ships, airplanes, missiles, ports and landscape, etc) and assess the user performance. This can be a laptop or a tablet. The Instructor Console receives the position and time of the ship via GPS (through the AR Simulation Engine) in order to continuously adjust the AR scenario regarding the relative distance between the external virtual content and the ship / user (adjustment of ships’ proximity with the external virtual content). The external virtual content data that are generated by the Instructor Console are also transmitted as Navigational data to the relevant ship’s system in the proper form (NMEA 0183 or 2000), via the TLA Simulation Engine, to present the external virtual objects and content as relevant tracks and data on the ECDIS I Warship ECDIS (WECDIS) and the Navigational Radar. This enables the external virtual content to be presented not only visually but also to be overlayed as simulation tracks and data / info on the real-world Navigational Systems of the bridge, improving the realism of the AR training.

[0065] In addition to open sea environment, the TLA simulator can be also applied to static environment, alongside or at anchor (see Fig. 3). As the ship is static, the Control Panel and the Ship Dynamic Model will be fitted in this specific architecture.

[0066] Control Panel receives the control outputs from the steering system (steering wheel reading) and from the Machinery Control and Condition Monitoring System (MCCMS, main engines’ level which gives the speed through the water and thruster level). In fact, the user adjusts heading, speed, and thrusters (if fitted), utilizing the real assets of the bridge. This data is sent via the TLA Simulator Engine to the Ship Dynamics Model, which simulates the vessel’s physical response (e.g., acceleration, turning, pitch, roll). If the horizon line is not present the system uses the snapping to reality model for the anchorage alignment of the virtual content (Bridge Interior Rejection Area and the External AR Area) but still the external virtual object / content is visible on AR glasses when it does not coincide / overlap with the Bridge Interior Rejection Area, in order to maintain the sufficient level of realism as it happens in the TLA method.

[0067] These setups are not binding for manufacturers; they are included here for a better understanding of the TLA method’s implementation. The TLA method can be implemented with different construction layouts.

[0068] A, The TLA Method

[0069] The TLA (Two Layer Anchorage / Alignment) method is processed in the TLA Simulation Engine. It consists of two layers of anchorage / alignment: the internal and the external. The process is illustrated as a block diagram in Fig. 11. Analysis of the implementation steps as follows:

[0070] 1. Internal Anchoring / Parallelization

[0071] The bridge of a ship is a structure / space that allows its crew to have visual contact with the external environment through the glass windows located around it, as well as through the bridge wings, depending on the ship’s design.

[0072] The user of the augmented reality (AR) glasses must detect and visually track the external virtual object / content only when they are projected through the glass windows. Otherwise, realism is significantly reduced, as the external virtual object / content would appear even in front of the bridge's structural elements (superstructures, radar devices, electronic aids, etc.).

[0073] To achieve this, a 3D mapping is executed using a Photogrammetry process to create a 3D digital model of the bridge. It works like reverse-engineering a 3D shape from multiple photos. Depending on the photogrammetry tools the user takes many overlapping photos of the bridge room from all sides. A photogrammetry algorithm finds common points in these photos (called "keypoints"). It calculates the position in 3D space of each point by comparing how it appears in different images (triangulation). It reconstructs a 3D mesh and applies textures from the photos for a realistic look. Afterwards we margin the areas that consist of the bridge glass windows, creating two sections of the 3D model as shown in figure 4: The 3D virtual bridge room without the glass windows (Bridge Interior Rejection Area) and the glass windows (External AR Area). The real elements used to anchor / parallelize the Bridge Interior Rejection Area and the External AR Area with the actual bridge, using the snapping to reality process, are fixed characteristic devices, such as radar consoles, electronic aids, and visible structural elements, such as the frames of the bridge window glasses, following the following three basic steps, based on the snapping to reality process mentioned above:

[0074] -Detection of the Real-World Features. The AR system begins by scanning and interpreting the real-world environment using a combination of sensing and computer vision techniques. This allows it to detect key reference points like window frames.

[0075] - Anchoring a Virtual Line (Virtual Anchoring lines). Once a reliable physical feature is detected, the AR system creates a virtual line or axis that is mathematically aligned with that feature (A vertical line aligned with the edge of a window frame, A horizontal line along the surface of a desk). This virtual line becomes a stable reference for placing virtual content (the Bridge Interior Rejection Area and the External AR Area and the External Virtual Content).

[0076] - Snapping Virtual Objects to the Line. Virtual content (the Bridge Interior Rejection Area and the External AR Area and the External Virtual Content) is then snapped to the anchored virtual line.

[0077] For the method of the present invention there are two types of Virtual Anchoring Lines:

[0078] - The Internal Virtual Anchoring Line for the anchorage of the Bridge Interior Rejection Area and the External AR Area. This line remains stable regardless of the actual movement of the ship in the dynamic environment of the sea; and

[0079] - The External Virtual Anchoring Line for the anchorage of the External Virtual Objects / Content. This line is being adjusted in relation to the three axis defined by the ship’s heading, pitch and roll movement, to maintain the external virtual objects / content always realistically aligned with the dynamic environment of the sea, regardless the real movement of the ship within it.

[0080] When the external virtual object / content overlaps the Bridge Interior Rejection Area, it is automatically rejected from appearing on the AR glasses but it’s still running on the background (the system is running but the specific virtual content is not presented). The external virtual object / content reappears to the user’s AR glasses once it no longer overlaps with the Bridge Interior Rejection Area and is included inside the External AR Area.

[0081] The Bridge Interior Rejection Area and the External AR Area do not appear in the AR glasses' view. The actual layout of the bridge remains visible. The only virtual content visible to the AR glasses user is the external virtual object / content. This method provides a sufficient level of realism as the user who is inside the bridge realizes that the external virtual objects are indeed located outside of the bridge (see Fig. 5).

[0082] Another parameter for the display of the external virtual object / content is the projection of the horizon (see Fig. 4) within the field of view of the user (human eye field of view: 200° horizontal

[0083] - 130° vertical) and the binocular devices' field of view (binocular field of view: 7° horizontal - 7° vertical). The identification of the Horizon can be implemented with OpenCV (Open- Source Computer Vision Library) which is an open-source software library that provides tools for realtime computer vision, image processing, and machine learning, using the camera of AR glasses. The AR camera of the glasses uses OpenCV to detect the horizon line in three steps: (1) It detects edges in the image using the Canny Edge Detection algorithm. (2) Hough Transform applies to identify horizontal lines within the image. (3) It selects the straightest and most horizontal line as the horizon line.

[0084] Below is the logic of the Two-Level Anchoring / Parallelization (TLA) method for the display of the external virtual objects / content in the user’s field of view or binocular field of view.

[0085] The external virtual object / content is visible to the AR glasses user through the application of the Two-Level Anchoring (TLA) method, according to this invention:

[0086] - If it does not coincide / overlap with the Bridge Interior Rejection Area; and

[0087] - And at the same time, the visible horizon is within the human eye’s field of view or the binocular field of view, depending on the user’s selection.

[0088] Maintaining the horizon within the field of view (either of the human eye or the binocular device) adds value to the training process. This ensures that trainees remain focused on detecting targets in the maritime and aerial environment and stay alert, rather than being distracted by secondary tasks inside the bridge (e.g., focusing on navigational aids, radar, GPS, etc.).

[0089] 2. External Anchoring / Parallelization

[0090] External anchoring / parallelization ensures the realistic depiction of the external virtual object / content, regardless of the ship’s movement relative to true north (see Fig. 6 and 7). 2.1 External Anchoring / Parallelization in the Horizontal Plane

[0091] Before starting the training scenario, the system goes through an initialization phase, using the Inertial Measurement Unit (gyroscope) of the AR glasses.

[0092] During this phase, the gyroscope is aligned with the ship’s initial gyrocompass reading for all pairs of AR glasses used.

[0093] A gyroscope measures angular velocity in 3 axes (usually x, y, z) and the actual heading of the ships. To track orientation and detect changes relative to a starting direction, an initial alignment (initialization) is being implemented. The subsequent gyroscope readings are used to compute changes from that starting point. The are two basic steps for this process:

[0094] - Initial Alignment (Calibration). The AR glasses system is placed in a reference position and the gyroscope’s actual readings are recorded; and

[0095] - Storage of the Initial Orientation. The AR system stores a rotation matrix or quaternion representing the initial orientation. From this point on, all gyroscope data can be integrated over time to track rotation relative to this starting / initial point.

[0096] For this specific method, the AR glasses user must keep their head steady, looking forward along the ship’s centerline or parallel to it. The ship must maintain a stable course for the system initialization time (3-5 sec).

[0097] The ship’s heading, as indicated by the gyrocompass, is transmitted to the TLA Simulator Engine using serial NMEA 0183 or 2000 protocol. The TLA Simulation Engine receives the actual gyro values from the AR glasses IMU via the AR Visual System unit. Alternatively, if the ship’s gyro heading is not present, the system utilizes the GPS heading, under the prerequisite that the ship maintains a minimum speed of 6 knots.

[0098] Once the initialization phase is complete, the system defines the initial heading as the initialization heading. All calculations for anchoring the external virtual object / content relative to the true north are referenced to the initialization heading.

[0099] After the system initialization phase, every ship’s turn and every head movement by the user’s AR glasses creates three angles relative to the initialization heading (horizontal plane) at each dt time interval (see Fig. 7), wherein:

[0100] - 6 is the angle between the initialization heading and the ship’s current actual heading. In fact, this angle determines the value for the digital displacement of the External Virtual Line to the opposite direction of the ship’s turn and subsequently relevant displacement of the external virtual object / content;

[0101] - 0 is the angle between the initialization heading and the AR glasses user's current line of sight;

[0102] - T is the angle between the initialization heading and the virtual object / content’ s line of sight, as defined in the operational scenario for each dt time interval.

[0103] The anchoring angle (OA) of the external virtual object / content corresponds to the angle of the external virtual object / content relative to the initialization heading at each dt time interval: OA = T ± 8

[0104] The sign of 8 angle follows the logic presented in Fig. 7. The 8 angle value is the input defining the movement of the external virtual object / content in the horizontal plane, in order to be presented realistically to the user. Figure 8 presents the utilization of 8 angle. In fact, the OA angle is the value of the actual position of the external virtual objects / content taking into account the initial position set by the training scenario (T) and the angle 8 created by the movement of the ship on the horizontal plane.

[0105] The 0 angle is used in training scenario analysis and, when considered relatively to OA, shows the AR user's response after each external virtual object / content appearance (e.g. correct tracking of targets based on threat warnings).

[0106] 2.2 External Anchoring / Parallelization in the Vertical Plane

[0107] The reference point for anchoring / parallelizing the external virtual object / content in the vertical plane relative to the Earth’s surface is the horizon line, as shown in Fig. 9.

[0108] The recognition of the horizon by the AR glasses follows the process described above.

[0109] Anchoring targets to the horizon line ensures a realistic depiction of both elevation and distance perception of the external virtual object / content relative to the ship using the AR simulator. In addition, the use of AR glasses’ IMU gyroscope stabilizes the virtual content regardless of the movement of the user’s head.

[0110] Beyond the anchoring function, the methods used for their visualization are identical to those of a maritime simulator inside a training room (distance, inclination, relative motion, etc.), taking into account the ship's bridge height above sea level and the Earth's curvature.

[0111] As for an in-classroom virtual reality simulator, the virtual horizon is aligned with the eye level of the user. The horizon in the simulator appears at the same height as it would in a real ship. This creates a powerful feeling of realism, especially when turning or rolling. As for this AR implication, the camera system of the AR glasses is positioned on the same level or (+-3cm) within the level of human’s eyes. That ensures that the recognized horizon line by the camera system of each AR glass is on the same level as the human eye level, and subsequently the external virtual content which is anchored with the horizon captured by the AR glasses’ camera, appears naturally and is realistically positioned for the AR training scenario (precise horizon alignment).

[0112] As for the virtual camera's FOV (usually 60°-90°), it is matched to the physical layout and curvature of the display system. This prevents distortion and ensures that virtual objects appear on the correct scale and direction relative to the user’s viewpoint. In addition, advanced simulators use projection warping software to distort the 3D rendering in just the right way so it fits seamlessly across curved or angled displays. This maintains consistent perspective and spatial alignment across multiple screens. These two features do not need to be used in this AR implication. The horizon line acts as a reference framework that keeps everything aligned and realistic. Furthermore, spatial anchors (such as ARKit and AR Foundation) and IMU / gyroscope data help stabilize virtual content, ensuring realistic projection of external digital elements.

[0113] 3. External Stabilization of External Virtual Content

[0114] The ship is a platform that moves and tilts (due to waves, immersive maneuvering, close to ship explosion etc), which affects the orientation of the AR system. Pitch and roll are two types of rotational movements a ship experiences due to the motion on the sea:

[0115] Pitch is the up-and-down tilting motion of a ship's bow and stem (front and back). It occurs along the ship's lateral (side-to-side) axis. Pitching is usually caused by waves hitting the ship head-on or from the stem. Roll is the side-to-side rocking motion of the ship, where the port (left) and starboard (right) sides move up and down in turn. It happens along the ship’s longitudinal (front-to-back) axis. Rolling is typically caused by waves striking the ship from the side.

[0116] If we use the snapping to reality method, anchoring the external virtual content on the bridge structure, the virtual content stays fixed relative to the ship — meaning if the ship pitches or rolls, the AR content moves with the user and appears stationary within the ship's frame of reference. That’s because the glasses use inertial sensors (IMU) and visual tracking to anchor content to the bridge room. This does not provide a sufficient level of realism, especially during harsh weather conditions, as the external virtual content will not stay fixed relative to the external environment.

[0117] To keep the external virtual objects / content visually stable and parallel to the horizontal line, the ship’s gyroscope continuously measures pitch (tilt forward / backward) and roll (tilt left / right). This pitch and roll information is sent to the AR system. The AR system uses this data to counter-rotate the External Virtual Anchorage Line and subsequently the external virtual content — basically applying the inverse of the platform's tilt. The output of this data, typically transmitted in NMEA 0183 or NMEA 2000 format via serial communication, is received by the TLA Simulator Engine, where it is parsed and converted into usable digital values. By accessing this real-time motion data, the AR system can dynamically adjust the positioning, orientation, and stability of the external virtual content. This ensures that external virtual objects remain visually “anchored” in their intended positions, even as the ship moves due to wave motion or maneuvers. The system compensates for the ship’s physical tilting and swaying by synchronizing the virtual scene with the gyro's continuous input, effectively stabilizing the augmented reality overlay and maintaining alignment with the external maritime environment (see Fig. 10).

[0118] B. HyperRealistic Visibility Determination Using Ray-Based Rendering and Spatial Pose Estimation in TLA Systems

[0119] The present invention further advances the realism of the TLA (Two-Layer Anchoring / Alignment) augmented reality (AR) simulation by introducing the concept of HyperRealism.

[0120] HyperRealism refers to an enhanced simulation fidelity that leverages precise internal-external anchoring and alignment to deliver an extraordinarily lifelike and accurate training experience. Specifically, this method ensures a coherent and exact representation of the user's perspective within a dynamically moving maritime environment by seamlessly integrating both the internal and external anchoring layers defined by the TLA method.

[0121] The HyperRealism methodology achieves the following:

[0122] - Precise User Positioning within Ship's Bridge:

[0123] The augmented reality system utilizes a predefined Internal Virtual Anchoring Line (IV AL), established via photogrammetry and computer vision-based feature detection as described in the TLA method. By anchoring AR glasses to the IV AL, the system continuously computes and updates the user's precise position inside the bridge compartment, specifically relative to fixed reference points, including bridge window frames and consoles.

[0124] - Window Identification for External Viewing:

[0125] Using computer vision algorithms (e.g., semantic segmentation or edge-detection techniques), the system continuously identifies transparent viewing areas (windows) within the bridge structure. These areas constitute the valid External AR Areas through which virtual content must be visible to maintain realism.

[0126] - Spatial Orientation through Real-Time Ship Dynamics:

[0127] The orientation and positioning of the user relative to the ship’s real-time dynamic conditions (pitch, roll, and yaw) are continuously updated using data from the ship’s inertial navigation systems (INS) or gyrocompass sensors. Specifically:

[0128] - Yaw (Heading): Adjustments to the user's orientation relative to true north are continually updated, using gyro or GPS heading data as described in the TLA method; and

[0129] - Pitch and Roll: Real-time ship pitch and roll values are acquired through onboard gyroscopic sensors (IMU) and integrated into the AR simulation. These values dynamically adjust the user's virtual perspective, aligning it accurately with the external maritime environment, thereby simulating the realistic effects of ship motion on visibility.

[0130] - Determining Visibility of External Virtual Objects using Ray Casting:

[0131] - The invention employs a ray casting (ray tracing) method to calculate real-time visibility of external objects such as missiles, aircraft, or vessels within the augmented reality simulation. Ray casting involves mathematically projecting virtual lines (rays) from the user's eyes or AR glasses camera outward into the environment;

[0132] - For each external object positioned in real-world coordinates, the AR system calculates the object's position relative to the user, taking into account:

[0133] 1. User’s precise internal bridge position (anchored to the IV AL).

[0134] 2. Real-time ship movements (pitch, roll, yaw).

[0135] 3. User’s head orientation and field of view, tracked via the AR glasses IMU; and

[0136] - Once the relative position is computed, rays are cast from the user's viewpoint toward each external object's real-world coordinates. Intersection checks are performed against the identified bridge window areas (External AR Areas). If a ray intersects a valid window area without obstruction (defined by the Bridge Interior Rejection Area), the external object is rendered visible through the AR glasses. If the intersection is blocked by internal bridge structures, the object is rendered invisible, maintaining realism.

[0137] - Real-Time Ray Casting Calculations:

[0138] Real-time computational performance for ray casting is maintained by employing optimized spatial indexing structures such as Binary Space Partitioning (BSP) trees, Octrees, or uniform grids. These structures facilitate rapid spatial queries and minimize computational load, enabling real-time updates at frame rates (30-60 Hz or greater) suitable for realistic and smooth AR experiences.

[0139] - Real-Time Horizon Detection:

[0140] The horizon line serves as an additional anchoring and orientation reference, dynamically identified using computer vision methods such as OpenCV-based edge detection (Canny Edge Detection) combined with Hough Transform methods as described in the TLA method. The detected horizon line ensures vertical stability of virtual content relative to the real-world environment.

[0141] Through this integration of precise internal positioning, external orientation tracking, window identification, and sophisticated ray casting techniques, the invention significantly enhances the realism of maritime augmented reality training scenarios. The combined result is an unparalleled training environment wherein external threats and operational scenarios appear seamlessly embedded within the user's real-world view, significantly elevating the effectiveness and realism of training exercises.

[0142] 1. HyperRealism: Method steps

[0143] The HyperRealism methodology describes a systematic approach for rendering external virtual objects in a Two-Level Anchoring (TLA) augmented reality framework in a physically accurate and perceptually realistic way. The method ensures that virtual elements such as threats, targets, or markers outside a vehicle (e.g., ship) are only rendered when they are truly visible through designated viewing areas, taking into account user viewpoint, structural obstructions, and vehicle motion. Overview of the Method

[0144] The method comprises five primary phases:

[0145] 1. User Pose Acquisition

[0146] 2. Bridge Window Mapping

[0147] 3. World Position Transformation

[0148] 4. Ray-Based Visibility Determination

[0149] 5. Selective Rendering

[0150] Each step is executed in real time and tightly integrated with both the internal spatial structure (e.g. ship's bridge layout) and the external simulation world.

[0151] Step 1 : User Pose Acquisition

[0152] The AR system continuously captures the user’s real-time physical position and head orientation. This is done using built-in sensors in the AR headset, such as inertial measurement units (IMUs) and visual-inertial tracking.

[0153] • The headset tracks how the user moves (forward / backward, side to side, up / down).

[0154] • It also tracks how the user is looking (turning head, tilting up / down, leaning left / right).

[0155] • Together, these define the user’s pose, a combination of position and gaze direction.

[0156] This pose is recorded in reference to the anchored interior of the environment, such as a digital twin of the ship’s bridge.

[0157] Step 2: Bridge Window Mapping

[0158] During an initial setup or calibration phase, the structural elements of the environment are mapped. Of particular interest are the transparent view areas, such as windows or glass panels.

[0159] • These windows are defined as visual portals through which external objects may appear.

[0160] • The system segments and stores the position, orientation, and geometry of each window surface.

[0161] • This data defines what is known as view-permissible zones, areas where external visibility is logically and optically valid.

[0162] Step 3: World Position Transformation

[0163] The position and orientation of the ship or moving platform (e.g. vehicle, aircraft, etc.) are constantly changing due to motion, turning, or sea conditions.

[0164] • The system receives real-time data from navigational sensors (e.g. GPS, gyros) to determine the ship’s heading, pitch, and roll.

[0165] • This information is combined with the user’s pose inside the ship to determine the absolute viewpoint of the user in world space.

[0166] • This transformation allows external objects to be accurately located in relation to the user’ s eye position and gaze direction.

[0167] Step 4: Ray-Based Visibility Determination

[0168] For each external virtual object of interest (e.g. missile, drone, waypoint):

[0169] • The system calculates the direction from the user’s eyes to the object.

[0170] • A virtual line of sight (ray) is drawn to check if the object falls within the user’s current field of view. • The system performs collision checks: o Does the line pass through a valid window? o Is the object blocked by any part of the ship’s interior (walls, consoles, ceilings)?

[0171] • Only if the line is unobstructed and passes through a mapped window, the object is flagged as visible.

[0172] This process runs for each relevant object, typically 30 to 60 times per second, to match real-time rendering rates.

[0173] Step 5: Selective Rendering

[0174] Objects flagged as visible are passed to the rendering engine for display in the AR headset.

[0175] • The system draws only those virtual objects that are realistically visible from the user’s location.

[0176] • Objects outside the field of view, behind structural obstructions, or outside the window zones are hidden or ignored.

[0177] • Optional rendering enhancements include: o Dynamic detail levels (based on distance), o Eye tracking for foveated rendering, o Environmental lighting for photorealism.

[0178] This results in a HyperRealistic experience where users only see external elements if they truly would be visible through physical line-of-sight.

[0179] 2. Environmental attenuation layer: Time and Weather-Responsive visibility control

[0180] To augment the HyperRealistic rendering system and increase operational realism, the TLA framework includes an Environmental Attenuation Layer. This layer simulates the real-world effects of lighting conditions (e.g., time of day) and atmospheric interference (e.g., fog, rain, or snow) on the user’s ability to perceive external virtual content through view-permissible surfaces. This enhancement ensures that rendered content conforms to actual visibility expectations based on both natural lighting and weather conditions, thus producing more immersive, context-accurate, and training- valid AR experiences.

[0181] 2.1. Time of Day Modulation

[0182] The TLA engine includes a configurable Time-of-Day Module, which accepts simulation parameters or real-time data corresponding to solar cycles (e.g., dawn, noon, dusk, night).

[0183] • This module calculates ambient light levels based on the simulated or actual time.

[0184] • It adjusts the user’s effective field of view or viewing distance through attenuation curves, mimicking human eye performance in low light.

[0185] • At night, external virtual objects are only visible if additional virtual light sources (e.g., spotlights, flares) are simulated or present.

[0186] • Bright daytime conditions may allow for greater visibility range, whereas twilight conditions may gradually fade out distant objects.

[0187] This modulation affects the final rendering decision: virtual content may be present but not rendered unless sufficient visibility is confirmed based on the current light model.

[0188] 2.2. Weather-Responsive Visibility Simulation

[0189] The system includes a Weather Simulation Module that can be configured manually or through real-time API integration (e.g., live weather feeds or pre-scripted scenarios). This module introduces visual degradation and spatial occlusion effects typical of weather events, including: • Fog: Reduces visibility distance uniformly; simulates light scattering.

[0190] • Rain: Introduces refraction and motion blur effects; adds environmental audio for realism.

[0191] • Snow or Dust: Introduces particulate occlusion and affects visibility differently by direction or elevation.

[0192] These conditions are encoded as visibility attenuation fields that apply to the view-frustum computation logic and to ray -based rendering evaluations.

[0193] • If an object falls outside the visibility threshold imposed by the current weather layer, it is excluded from rendering even if otherwise visible through structural windows.

[0194] • The engine supports both global attenuation (e.g., uniform fog) and directional attenuation (e.g., rain blocking forward but not lateral view).

[0195] 2.3. Integrated Workflow

[0196] The Environmental Attenuation Layer works in tandem with the visibility pipeline as follows:

[0197] • After calculating the user’s global viewing pose,

[0198] • And confirming that a line of sight passes through a valid window without obstruction,

[0199] • The system queries the Environmental Conditions Module to determine if the object is theoretically visible given current light and weather.

[0200] Only if visibility is confirmed under these compounded conditions is the object passed to the rendering engine.

[0201] This approach replicates human perceptual limitations and supports more nuanced simulation training e.g., night-time threat detection, foggy port entry, or storm-response drills.

[0202] C. Post-Training Analysis Tools

[0203] The present invention further comprises a comprehensive Post-Training Analysis (PTA) module designed to enhance the overall effectiveness of maritime and naval operational training by facilitating detailed reviews and analytical assessments of each training session. This component systematically captures, records, and evaluates various types of data generated throughout augmented reality (AR) training scenarios, enabling both real-time and retrospective analyses.

[0204] 1. Data Collection and Types

[0205] The PTA module will continuously collect and record a diverse array of data during each AR training session, including but not limited to:

[0206] • Scenario Event Logs: Time-stamped records of every scenario event, including the appearance, movement, and interaction of virtual elements such as enemy vessels, aircraft, missiles, drones, and environmental factors.

[0207] • User Interaction Data: Detailed information on trainee interactions, including gaze tracking, head and body movement patterns, voice commands issued, hand gestures, and manual controls operated during the scenario.

[0208] • Environmental Data: Continuous logging of real-time environmental conditions and simulated conditions, including weather scenarios, visibility constraints (fog, rain, night), and sea state dynamics (pitch, roll, heading).

[0209] • Performance Metrics: Precise metrics on trainee response times, accuracy in threat identification, effectiveness of decisions made, communication clarity, and adherence to predefined tactical or operational procedures. • Physiological and Ergonomic Metrics: Optional biometric data capturing trainee physiological responses such as stress indicators, heart rate variability, fatigue levels, and simulator-induced discomfort or disorientation.

[0210] 2. Interactive Playback and Review

[0211] The PTA module will support interactive scenario playback, providing the instructor and trainees with the ability to review the training session from multiple perspectives:

[0212] • First-person trainee perspective playback, recreating precisely what each trainee viewed during the session, synchronized with the recorded AR virtual content.

[0213] • Third-person perspective playback, offering an external or overhead view of the entire operational scenario, useful for examining broader tactical situations, team coordination, and situational awareness.

[0214] Playback controls include pause, rewind, fast-forward, and slow-motion options, allowing in- depth analysis of critical moments, errors, decision-making processes, and successes.

[0215] 3. Performance Evaluation and Scoring

[0216] The module will generate detailed performance reports and scores based on quantitative and qualitative criteria. The evaluation encompasses:

[0217] • Decision-making effectiveness under varying conditions of stress and scenario complexity.

[0218] • Reaction speed and accuracy regarding threats or changes within the simulated environment.

[0219] • Communication efficiency, evaluating clarity, speed, and accuracy of communication among trainees during complex operations.

[0220] Scores and performance metrics can be customized according to specific training objectives, operational doctrines, or instructor preferences.

[0221] 4. Advanced Data Analytics and Cross-Referencing

[0222] The PTA system may also include an advanced analytics engine capable of performing statistical and comparative analyses across multiple training sessions and scenarios. The analytics tools facilitate:

[0223] • Critical Action Identification: Automatic identification and ranking of trainee actions that significantly correlate with successful outcomes, enabling reinforcement of effective tactics and procedures.

[0224] • Scenario Success Correlation: Filtering and grouping successful and unsuccessful scenarios to identify common factors, decision points, or critical junctures that influence mission success or failure.

[0225] • Predictive Performance Modeling: Utilizing machine learning algorithms trained on historical training data to predict trainee performance in future scenarios, identifying potential skill gaps, and recommending personalized or targeted training exercises.

[0226] • Trend Analysis and Reporting: Visual and statistical trend reporting, identifying performance improvements, degradation, or stability over time, allowing instructors to monitor trainee progression and adapt training strategies accordingly.

[0227] 5. Exporting and Integration Capabilities

[0228] The PTA module will support seamless integration with external Learning Management Systems (LMS), performance management tools, or mission planning software. This interoperability allows: • Exporting Detailed Reports: Generated performance data and analytical reports can be exported in industry-standard formats (e.g., XML, CSV, JSON) for further integration into broader organizational training and operational databases.

[0229] • Scenario Database Building: Enabling the creation of an extensive database of AR scenarios and outcomes, contributing to institutional knowledge, adaptive training evolution, and improved scenario realism and effectiveness.

[0230] D. Cognitive Fatigue Simulation Module (CFSM)

[0231] To enhance training realism and operational effectiveness, the present invention comprises an innovative Cognitive Fatigue Simulation Module (CFSM). This module simulates the cognitive impacts of fatigue on trainees, reflecting realistic operational conditions involving prolonged duty cycles, limited rest periods, or sleep deprivation. By dynamically adjusting trainees' perceptual, cognitive, and motor response capabilities within augmented reality (AR) scenarios, the CFSM prepares personnel to perform effectively under realistic conditions of stress and fatigue.

[0232] 1. Functionality and Implementation

[0233] The CFSM will operate by applying adjustable fatigue parameters to trainees, modifying reaction times, decision-making processes, visual perception accuracy, and overall cognitive performance levels in real-time or based on predefined fatigue profiles.

[0234] • Fatigue Profiles:

[0235] The module will employ fatigue profiles derived from scientific models of sleep deprivation and cognitive impairment (such as the SAFTE model or similar), allowing instructors to specify realistic fatigue scenarios based on the simulated operational demands.

[0236] • Dynamic Response Adjustment:

[0237] CFSM will dynamically modify trainee interactions, reaction speed, precision in threat identification, situational awareness, and decision-making speed according to specified fatigue parameters. For example, as simulated fatigue increases, trainees might experience perceptual delays, slower cognitive processing, or reduced accuracy in detecting and tracking threats.

[0238] • Visual and Auditory Cue Alterations:

[0239] Fatigue simulation may manifest through subtle reductions in visual clarity, diminished auditory alertness, increased ambient noise distractions, or slowed response of the AR content, closely mimicking cognitive impacts of real fatigue.

[0240] 2. Integration with PTA Module

[0241] Data on cognitive fatigue levels, including simulated fatigue severity and trainee responses under fatigue conditions, will be collected and incorporated into the Post-Training Analysis (PTA) Module. These data will enrich analytical insights into trainee performance, enabling:

[0242] • Comparison of trainee effectiveness under varying fatigue conditions.

[0243] • Identification of critical fatigue thresholds impacting mission success.

[0244] • Personalization of fatigue management strategies for operational readiness.

[0245] 3. Adaptive Training and Mitigation Strategies

[0246] The CFSM will enable the development of fatigue management and mitigation training, guiding trainees on recognizing fatigue symptoms, effectively managing cognitive resources, and maintaining performance under stress. Trainers will be able to customize scenarios to gradually expose trainees to fatigue, allowing controlled exposure and adaptive skill-building.

[0247] The following examples illustrate preferred embodiments in accordance with the present invention without limiting the scope or spirit of the invention:

[0248] EXAMPLES

[0249] Example 1 : AR Training Session according to the present invention

[0250] During an AR training session onboard a vessel based on the TLA (Two-Level Anchoring / Alignment) method, the ship’s navigation team is tasked to execute collision-avoidance maneuvers against a virtual ship crossing from starboard to port. The scenario is delivered through AR glasses worn by the bridge team, with the virtual vessel appearing outside the real windows, moving across the sea in a realistic trajectory.

[0251] The trainee observes the virtual ship’s bow emerge on the starboard side horizon and steadily approach, simulating a crossing situation at sea. Meanwhile, the real ship is executing a controlled turn to port to avoid a collision.

[0252] The complexity of the scenario arises from the fact that both the real ship and the user’s head are in motion — yet the virtual ship must remain visually stable, properly aligned, and realistically anchored to the external seascape.

[0253] This realism is made possible through the TLA method’s external anchoring and stabilization mechanisms. First, the initial heading of the ship is recorded during system initialization, and all subsequent ship movements are monitored via gyrocompass. As the vessel turns, the system calculates the angular change (5) and compensates for it by adjusting the position of the virtual ship relative the user's line of sight. In real time, the AR glasses account for pitch and roll through continuous gyro input, ensuring that the virtual ship doesn’t sway or drift unrealistically during the turn.

[0254] Simultaneously, the parallelization in the horizontal plane maintains the virtual ship’s trajectory aligned with true geographic directions, while stabilization in the vertical plane keeps the virtual ship level with the real sea horizon, regardless of deck pitch and roll. This allows the navigation team to assess the relative motion, judge closest point of approach (CPA), and practice real collision-avoidance maneuvers in a visually coherent and lifelike operational setting.

[0255] The implementation of the Two-Level Anchoring / Alignment (TLA) method introduces a transformative capability that allows a standard navigational simulator to be modified and installed directly onto the bridge of a real vessel, enabling high-fidelity AR-based training and situational simulation under two distinct operational conditions: (1) while the ship is underway in open sea and (2) while the ship is moored at the pier. These two installation modes leverage the ship’s existing navigation and sensor systems to create a seamless fusion between the real-world environment and simulated training scenarios, making it possible to conduct immersive, cost- effective, and operationally relevant training sessions on the same working place they sail in real conditions.

[0256] For open-sea operations, the TLA simulator is reconfigured to function without a conventional dynamic simulation model since the ship’s actual heading, speed, pitch, roll, and course over ground serve as real-time inputs. The installation process begins by integrating the TLA Simulation Engine with the vessel’s navigation data sources, typically via standard maritime protocols such as NMEA 0183 or NMEA 2000 over serial (RS232) or Ethemet / TCP communication. These data streams include readings from the gyrocompass (heading, pitch and roll), Agilog or speed log (speed through water), GPS (position and speed over ground. Once connected, the TLA engine processes these inputs in real-time and communicates with AR- capable Optical See-Through or Video See-Through glasses worn by the trainee. These glasses are equipped with cameras, IMUs, and spatial mapping tools that allow virtual content — such as ships, buoys, or weather hazards — to be anchored on the real ocean environment with stable alignment to the real horizon, preserving spatial accuracy despite continuous vessel movement. Internal anchoring ensures that the AR system disregards areas like bridge walls, radar consoles, and superstructures (known as the Bridge Interior Rejection Area), rendering virtual content only through the ship’s glass panels (the External AR Area).

[0257] This setup allows deck officers and bridge teams to train in real-time threat detection, maneuvering decisions, and emergency procedures as though the scenario were occurring around them, without interfering with the ship’s actual operations.

[0258] When the ship is moored at the pier or anchored, the TLA simulator reverts to a hybrid mode in which the dynamic ship motion must be simulated since the vessel itself is stationary. In this configuration, additional control inputs are captured from the ship’s steering systems (rudder position, thrusters) and engine telegraphs, which are read by the Control Panel component of the TLA simulator. These inputs are processed by a software-based Ship Dynamics Model, which replicates realistic vessel movement, including acceleration, turning behavior, and pitch / roll responses. This simulated movement is then treated as if it were actual motion and used by the TLA Simulation Engine to render external virtual content as if the ship was underway. To maintain continuity and realism, the AR glasses still perform internal alignment using photogrammetry of the bridge interior, allowing the virtual content to remain consistent with what would be visible during a real voyage. Notably, if the actual horizon is not visible from the pier due to port infrastructure, cranes, or land, the system compensates using OpenCV-based image analysis and simulated horizon projection. This ensures training continuity even in visually obstructed environments.

[0259] Overall, this dual-mode adaptation of the TLA-enabled simulator allows any vessel to serve as flexible, mobile training platforms, at sea or at berth, providing crews with continuous access to hyper-realistic, immersive training environments tailored to real-world maritime conditions and bridge layouts, all without reliance on shore-based simulators or costly live assets.

[0260] While the present invention has been described with respect to the particular embodiments, it will be apparent to those skilled in the art that various changes and modifications may be made in the invention without departing from the spirit and scope thereof, as defined in the appended claims.

Claims

CLAIMS1. A method for anchoring and / or aligning of Augmented Reality (AR) of virtual object and / or content with a dynamic open-sea environment, for the construction of Maritime and Naval operational training on board of a ship’s bridge, wherein said method comprises two layers of anchoring and alignment steps, namely a) Internal anchoring and / or alignment of the virtual object and / or content. b) External anchoring and / or alignment of the virtual object and / or content.

2. The method according to claim 1, wherein the virtual object and / or content is external and said internal anchoring and / or alignment of the external virtual object and / or content comprises the determination of the Bridge Interior Rejection Area within the ship’s bridge and the External Augmented Reality (AR) Area which are anchored and / or aligned to the Internal Virtual Lines located internally of the ship’s bridge.

3. The method according to claim 1 or 2, wherein said internal anchoring and / or alignment of the external virtual object and / or content is based on a determination of External Virtual Lines also located internally of the ship’s bridge.

4. The method according to claim 1, wherein the external virtual object and / or content are not presented to augmented reality glasses of a user when they overlap with areas defined by real installed elements within the ship's bridge, which represent the Bridge Interior Rejection Area.

5. The method according to claim 1 or 2, wherein for the internal anchoring and / or alignment of the virtual object and / or content, a visible horizon is used as a condition for the display of the external virtual object and / or content.

6. The method according to claim 1 or 5, wherein for the external anchoring and / or alignment of the external virtual object and / or content, the visible horizon is used on the Vertical Plane.

7. The method according to claim 6, wherein for the external anchoring and / or alignment of the external virtual object and / or content, the ship’s heading coming from ship’s gyro is used, as input data for the initialization of the gyroscope of the augmented reality glasses, which is essential for calculating an anchoring angle 8 value corresponding to the relevant correction or displacement of the external virtual line and / or lines and subsequently of the external virtual object and / or content relatively to the initial heading, on the Horizontal Plane.

8. The method according to claim 7, wherein for the external anchoring and / or alignment of the external virtual object and / or content, the ship’s pitch and roll values coming from ship’s gyro are used, as input data for the relevant correction or displacement of the external virtual line or lines and subsequently of the external virtual object and / or content relatively to the horizon line.

9. The method according to any of the preceding claims, wherein said virtual object and / or content is animated and not necessarily static, and wherein said animation is dynamically synchronized with the internal and / or external anchoring and alignment framework, such that the animated virtual object and / or content maintains spatial and directional coherence relative to: (a) internal anchoring references including Internal Virtual Lines and Bridge Interior Rejection Areas;(b) external anchoring references including the visible horizon, ship’s heading, pitch, roll, and yaw; and(c) user viewpoint as determined through real-time pose tracking and motion compensation, whereby said animated virtual object and / or content may include temporally changing behaviors, movements, or interactions, such as simulated maritime traffic, weather elements, navigational cues, or operational scenarios, while remaining accurately positioned and visually consistent within the augmented reality training environment aboard a dynamically moving ship platform.

10. The method according to claim 1, wherein said method further comprises a HyperRealism mode of operation, wherein said HyperRealism mode is characterized by the simultaneous utilization of the internal anchoring lines for precise user positional determination within the ship's bridge and the external anchoring lines dynamically adjusted based on real-time pitch, roll, and yaw data to create a highly realistic augmented reality viewing experience.

11. The method according to claim 10, wherein the method for determining the visibility of the external virtual object and / or content in an augmented reality system using the two-level anchoring framework, comprises the following steps: a) acquiring the real-time position and orientation of a user within an internal environment using an augmented reality device; b) identifying transparent or view-permissible surfaces within a digital model of the internal environment; c) acquiring the real-time motion state of a moving platform including pitch, roll, and yaw; d) calculating the user’s absolute viewpoint in world coordinates by combining the user’s pose with the platform’s motion state; e) determining, for each external virtual object, whether a simulated line of sight from the user’s absolute viewpoint intersects a view-permissible surface without obstruction by internal geometry; and f) rendering the external virtual object only if visibility is confirmed based on the intersection.

12. The method according to claim 11, wherein said method further comprises adjusting the user's effective viewing capability based on a simulated and / or real-world environmental parameters, comprising time of day and weather conditions, such that: a) during low-light time periods including dusk or night, the visibility of external virtual object and / or content is attenuated unless compensated by simulated artificial light sources; b) under simulated weather conditions including fog, rain, snow, or dust, visibility is modified using corresponding attenuation profiles that define visibility distance and field of view constraints; c) fog condition applies a depth-based attenuation curve along the user's view vector to reduce the maximum renderable distance for virtual objects; d) rain or snow conditions introduce directional or volumetric occlusion effects that prevent rendering of external virtual object and / or content beyond a configurable distance threshold; and wherein time of day and weather parameters are integrated into a ray casting visibility determination process, such that a virtual object is rendered only when it is not obstructed by physical or virtual structures, lies within view-permissible zones, and remains perceivable under current environmental conditions.

13. The method according to any of claims 1 or 9, wherein said method further comprises a PostTraining Analysis (PTA) module, characterized by systematic collection, recording, and analysis of data generated during augmented reality training scenarios, the method comprising the following steps:a) continuous logging of scenario event data, trainee interaction data, environmental parameters, and trainee physiological responses; b) interactive scenario playback capability, including first-person and third-person perspectives, enabling detailed post-session review; c) performance evaluation and scoring mechanisms based on predefined quantitative and qualitative criteria, assessing trainee effectiveness, accuracy, and decision-making.

14. The method according to any of claims 1, 10 or 13, wherein said method further comprises a Cognitive Fatigue Simulation Module (CFSM), configured to dynamically adjust trainee cognitive performance parameters within augmented reality training scenarios based on simulated cognitive fatigue resulting from prolonged operational activity or sleep deprivation, wherein said CFSM being characterized by: a) modulation of reaction times, perceptual accuracy, decision-making speed, and situational awareness in real time to reflect fatigue-induced cognitive impairment; b) utilization of predefined Fatigue Profiles derived from scientifically validated sleep deprivation and fatigue models to enable realistic and evidence-based simulation of cognitive fatigue effects; c) manifestation of fatigue impacts through multi-sensory alterations, including reduced visual clarity, delayed auditory processing, slowed responsiveness of virtual elements, and increased cognitive distractors within the simulation environment; d) dynamic adjustment of cognitive fatigue intensity based on trainee performance metrics, enabling adaptive fatigue management and progressive training under varying operational stress conditions; e) real-time generation and recording of cognitive fatigue data during training, scenarios, with subsequent integration into the Post-Training Analysis (PTA) module for performance assessment, identification of fatigue thresholds, and formulation of individualized fatigue mitigation protocols; and f) generation of analytical insights into trainee cognitive resilience, including predictive indicators of fatigue susceptibility to support personalized training interventions and the planning of proactive fatigue countermeasures.

15. An Augmented Reality Training Simulator (TLA Simulator) comprising structural amendments, additions and connections among the relevant components of a standard inclassroom navigational simulator for the construction of Augmented Reality Training Simulator on any ship’s bridge for use while at open sea or moored at the pier.

Images