Marine digital twin holographic sand table interaction method, device, medium and equipment

Abstract

The present application relates to the field of marine engineering simulation and holographic display technology, and discloses a marine digital twin holographic sand table interaction method, device, medium and equipment, the method comprises the following steps: generating a logical time according to the timestamp of the state data of the autonomous underwater vehicle; extracting the state information corresponding to the logical time in the state history information, time aligning the state information with the detection information based on the logical time, and generating a digital twin snapshot; based on the spatial calibration relationship, the geographic space information in the digital twin snapshot is mapped to the physical display space of the holographic sand table to generate a mapped digital twin snapshot; and synchronously driving multiple display terminals to generate a visual image within the same display period. The above method does not need to switch between two-dimensional interfaces and manually establish spatial correspondence, and can intuitively and accurately observe the three-dimensional spatial relationship between the position, attitude, track and detection results of the autonomous underwater vehicle in the holographic sand table, thereby improving the efficiency and accuracy of underwater situation awareness.

Description

Technical Field

[0001] This invention relates to the field of marine engineering simulation and holographic display technology, and in particular to an interactive method, device, medium and equipment for marine digital twin holographic sand table. Background Technology

[0002] Autonomous Underwater Vehicles (AUVs) can perform tasks such as topographic mapping, target detection, pipeline inspection, and environmental surveys in underwater environments that are difficult for personnel to access directly. Due to the limited electromagnetic propagation, low visibility, and distinct three-dimensional features of the operating space underwater, shore-based personnel typically rely on discrete state parameters, track maps, and sonar images transmitted back by the AUV to determine its position, attitude, and surrounding environment. With the development of digital twin and holographic display technologies, the industry is attempting to present the AUV operation process in a three-dimensional visualization format to enhance the intuitiveness of underwater situational awareness. Among these, holographic sand tables, as a display medium that can provide three-dimensional observation, are also being explored for immersive displays of marine scenarios.

[0003] However, existing 3D visualization and holographic sand table display solutions for AUV operations still have the following shortcomings: First, the sampling frequencies and transmission delays of AUV status data and sonar detection data are different. If each display module refreshes independently, it will lead to mismatch between AUV attitude and sonar image at the same time. Network jitter will also cause model motion jumps. Second, there is a lack of a stable coordinate mapping mechanism between the 3D virtual scene and the physical space of the holographic sand table. When the geographical range, display device, or observation perspective changes, the proportion and position of terrain, track, and detection results are prone to drift, making it impossible to ensure that multiple display terminals present a twin state under a unified coordinate reference at the same time. Third, the holographic sand table lacks a processing link that integrates and drives the fusion of AUV real-time status, track, and sonar detection results, making it impossible to achieve real-time closed-loop interaction from data reception to holographic presentation.

[0004] Therefore, a method for achieving unified spatiotemporal fusion and real-time interactive display of the AUV underwater operation status in a three-dimensional holographic sand table is urgently needed. Summary of the Invention

[0005] In view of this, this application provides a marine digital twin holographic sand table interaction method, device, medium and equipment, the main purpose of which is to solve the technical problem that the existing technology cannot fuse heterogeneous data according to a unified spatiotemporal reference and synchronously drive the holographic sand table to interact and display with multiple display terminals in real time.

[0006] According to a first aspect of the present invention, a method for interactive marine digital twin holographic sand table is provided, comprising: Acquire the status data of the autonomous underwater vehicle and generate a logical time based on the timestamp of the status data; Extract the state information corresponding to the logical moment from the state history information of the autonomous underwater vehicle, obtain the detection information associated with the state information, and time-align the state information and the detection information based on the logical moment to generate a digital twin snapshot; Based on a preset spatial calibration relationship, the geospatial information in the digital twin snapshot is mapped to the physical display space of the holographic sand table to generate a mapped digital twin snapshot. Based on the mapped digital twin snapshot, multiple display terminals are synchronously driven to generate visualizations corresponding to the logical moment within the same display cycle.

[0007] Optionally, extracting the state information corresponding to the logical moment from the state history information of the autonomous underwater vehicle includes: when there is no corresponding state frame for the logical moment in the state history information, selecting an adjacent state frame for the logical moment; performing position interpolation on the position parameters in the adjacent state frame to generate interpolated position information corresponding to the logical moment; performing attitude interpolation on the attitude parameters in the adjacent state frame to generate interpolated attitude information corresponding to the logical moment; and combining the interpolated position information and the interpolated attitude information into the state information corresponding to the logical moment.

[0008] Optionally, obtaining the detection information associated with the state information includes: sending a detection trigger command to the autonomous underwater vehicle and recording the trigger time, determining the sensor pose of the autonomous underwater vehicle at the trigger time; determining the target detection space based on the sensor pose, transmitting a detection signal to the target detection space, receiving the echo signal and generating a detection result; associating and storing the detection result with the sensor pose, and using the associated and stored detection result as the detection information associated with the state information.

[0009] Optionally, the method for establishing the preset spatial calibration relationship includes: selecting at least three non-collinear physical control points in the physical display space of the holographic sandbox, and determining virtual control points corresponding to the physical control points in the engine world scene; calculating the transformation matrix between the coordinate system of the engine world scene and the coordinate system of the physical display space based on the physical control points and the corresponding virtual control points; calculating the reprojection error of the transformation matrix based on the physical control points and the corresponding virtual control points; and using the transformation matrix as the preset spatial calibration relationship when the reprojection error is less than a preset threshold.

[0010] Optionally, the step of mapping the geospatial information in the digital twin snapshot to the physical display space of the holographic sand table based on a preset spatial calibration relationship includes: acquiring longitude data, latitude data, and depth data from the geospatial information; using a preset geographic reference origin as a benchmark, converting the longitude data and latitude data into horizontal positions in the coordinate system of the engine world scene, and converting the depth data into vertical positions in the coordinate system of the engine world scene, generating engine world coordinates based on the horizontal and vertical positions; and mapping the engine world coordinates to the physical display space of the holographic sand table based on the preset spatial calibration relationship.

[0011] Optionally, the step of synchronously driving multiple display terminals to generate a visualization corresponding to the logical moment within the same display cycle based on the mapped digital twin snapshot includes: extracting the fan-shaped boundary and the hit point cloud from the detection information in the mapped digital twin snapshot, and obtaining the position data of the fan-shaped boundary and the hit point cloud in the physical display space of the holographic sand table; obtaining the pose data of the autonomous underwater vehicle in the physical display space from the mapped digital twin snapshot; spatially aligning the position data of the fan-shaped boundary and the hit point cloud with the pose data of the autonomous underwater vehicle; and generating the visualization corresponding to the logical moment in the physical display space based on the spatially aligned position data.

[0012] Optionally, after acquiring the state data of the autonomous underwater vehicle and generating a logical time based on the timestamp of the state data, the method further includes: entering a playback mode in response to the state data reading operation; receiving a playback control command in the playback mode and determining a target playback time based on the playback control command; using the target playback time as the logical time, extracting historical state information corresponding to the logical time from the state history information of the autonomous underwater vehicle, and acquiring historical detection information associated with the historical state information; and aligning the historical state information with the historical detection information based on the logical time to generate a historical digital twin snapshot.

[0013] According to a second aspect of the present invention, a marine digital twin holographic sand table interactive device is provided, comprising: The clock unification module is used to acquire the status data of the autonomous underwater vehicle and generate a logical time based on the timestamp of the status data. The snapshot generation module is used to extract the state information corresponding to the logical moment from the state history information of the autonomous underwater vehicle, obtain the detection information associated with the state information, and time-align the state information with the detection information based on the logical moment to generate a digital twin snapshot. The spatial mapping module is used to map the geospatial information in the digital twin snapshot to the physical display space of the holographic sand table based on a preset spatial calibration relationship, thereby generating a mapped digital twin snapshot. The synchronous display module is used to synchronously drive multiple display terminals to generate a visualization corresponding to the logical moment within the same display cycle, based on the mapped digital twin snapshot.

[0014] According to a third aspect of the present invention, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the above-described interactive method for marine digital twin holographic sand table.

[0015] According to a fourth aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described marine digital twin holographic sand table interaction method.

[0016] This invention provides a method, apparatus, medium, and device for interactive marine digital twin holographic sand table. First, a unified logical time is generated based on the timestamps of the state data, incorporating all data into the same time base. Then, state information corresponding to the logical time is extracted from historical state information, and the detection information and state information are time-aligned based on the logical time to generate a digital twin snapshot. This ensures precise matching between the attitude of the autonomous underwater vehicle and the sonar image at the same moment, eliminating the phenomenon of model motion jumps caused by network jitter. Second, based on a preset spatial calibration relationship, the geospatial information in the digital twin snapshot is mapped to the physical display space of the holographic sand table, establishing a calculable alignment between the marine geographic coordinates and the sand table physical coordinates. This approach avoids proportional drift in terrain, trajectory, and detection results when the geographical scope, display device, or observation perspective changes, ensuring that multiple displays simultaneously present a twin state under a unified coordinate reference. Finally, by acquiring state data and generating logical moments, followed by time alignment to create snapshots, and then spatial mapping to synchronously drive the displays, the autonomous underwater vehicle's state data, detection information, and holographic sand table display are driven uniformly within the same cycle. This achieves real-time closed-loop interaction from data reception to holographic presentation, ensuring that the control terminal screen and the holographic sand table screen are output based on the same logical moment and coordinate reference, and refresh synchronously within the same display cycle, guaranteeing complete consistency of the digital twin state observed by multiple displays. This method allows shore-based personnel to intuitively and accurately observe the three-dimensional spatial relationships between the autonomous underwater vehicle's position, attitude, trajectory, and detection results in the holographic sand table without switching between multiple two-dimensional interfaces and manually establishing spatial correspondences, thus improving the efficiency and accuracy of underwater situational awareness.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 A flowchart illustrating an interactive method for a marine digital twin holographic sand table provided by an embodiment of the present invention is shown. Figure 2 A flowchart illustrating another interactive method for marine digital twin holographic sand table provided by an embodiment of the present invention is shown; Figure 3 This illustration shows a flowchart of the generation of a unified timeline and a digital twin snapshot in another interactive method for marine digital twin holographic sand table provided by an embodiment of the present invention; Figure 4 This paper illustrates a flowchart of the fusion process between sonar detection results and the holographic sand table in another interactive method for marine digital twin holographic sand table provided by an embodiment of the present invention. Figure 5 This illustration shows a schematic diagram of the real-time, recording, and playback state switching in another interactive method for marine digital twin holographic sand table provided by an embodiment of the present invention; Figure 6 This invention provides a schematic diagram of the structure of another marine digital twin holographic sand table interactive system. Figure 7 This diagram illustrates the structure of an interactive marine digital twin holographic sand table device according to an embodiment of the present invention. Figure 8 This invention provides a schematic diagram of the structure of another marine digital twin holographic sand table interactive device. Figure 9 A schematic diagram of the device structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0019] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0020] This application provides an interactive method for a marine digital twin holographic sand table, such as... Figure 1 As shown, the method includes the following steps: 101. Obtain the status data of the autonomous underwater vehicle and generate a logical time based on the timestamp of the status data.

[0021] Among them, the state data of the autonomous underwater vehicle refers to the set of parameters reflecting the current motion state and spatial attitude of the autonomous underwater vehicle (AUV), including at least the longitude, latitude, depth, heading angle, pitch angle, roll angle, forward velocity, lateral velocity, and vertical velocity of the AUV; the timestamp is a time mark associated with each frame of state data, used to identify the time when the frame of state data was generated, and its source can be the system clock on the AUV or an external timing system; the logical moment refers to the unified time reference value used to drive the update of the digital twin scene. In real-time mode, the logical moment is determined based on the timestamp of the state data combined with the system monotonic clock and display delay. In playback mode, the logical moment is determined based on the time position indicated by the playback progress. By acquiring the state data of the AUV and generating the logical moment, a unified time reference is established for all subsequent processing steps.

[0022] Specifically, the system first receives status data frames from external data sources via the data access and protocol processing module. These external data sources include the AUV simulation platform or the actual data forwarding equipment. The system performs frame header recognition, length verification, and field decoding on the data frames to ensure the integrity and validity of the data entering the system. Next, the system extracts the timestamp from the decoded status data frame and generates a unified logical time based on the current operating mode. In real-time mode, the system uses the current monotonic clock minus a preset display delay as the target display time. The display delay is determined by a base value and the statistical value of the most recent sampling interval. When a new status data frame arrives, the system uses this frame's timestamp. The system updates the logical time based on the baseline. During short-term network data interruptions, the system calculates the logical time based on the statistical values ​​of the most recent sampling period to maintain continuous system operation. In one optional implementation, AUV status data is transmitted using fixed-length UDP frames, which contain fields such as carrier identifier, six-degree-of-freedom attitude, three-axis velocity, latitude and longitude, and depth. After the system completes frame parsing, it obtains the timestamp and the corresponding status parameters. In playback mode, the system's logical time is determined by the playback progress, rather than relying on the timestamp of real-time data. Through the above methods, this step completes the conversion from raw data input to a unified time base, providing a reusable time anchor point for subsequent steps.

[0023] In this embodiment, by acquiring AUV status data and generating a unified logical moment based on timestamps, multi-source data from different sensors are incorporated into the same time base, laying the foundation for time alignment of subsequent status information and detection information. This fundamentally avoids data mismatch problems caused by different sampling frequencies and inconsistent transmission delays of various sensors. Through unified management of logical moments, the ordinary display end and the holographic sand table display end share the same time anchor point, ensuring that the digital twin snapshots generated subsequently have uniqueness and consistency in the time dimension. Reusable logical moments can be generated in both real-time and playback modes, enabling real-time monitoring and historical review to share the same time-driven mechanism, ensuring the consistency of display semantics in different working modes. Through adaptive adjustment of display delay and time estimation based on sampling statistics during network outages, the impact of network jitter on display continuity is effectively eliminated, making the movement of the AUV model in the holographic sand table smoother and avoiding model movement jump problems caused by unstable data packet arrival times.

[0024] 102. Extract the state information corresponding to the logical time from the state history information of the autonomous underwater vehicle, obtain the detection information associated with the state information, and align the state information and detection information in time based on the logical time to generate a digital twin snapshot.

[0025] Among them, the status history information refers to the collection of historical status data of the AUV stored in chronological order. Each frame of status data contains at least parameters such as AUV identifier, position, attitude, velocity, thruster speed, and rudder angle, and is associated with a corresponding logical time, which is used to support querying and tracing the AUV status at any historical time. The detection information refers to the detection data collected by the sonar detection component on the AUV at a specific time, including sonar images, echo intensity data, target detection results, and the corresponding Ping sequence number and trigger time. Time alignment refers to the process of associating the status information and the detection information with the same logical time, ensuring that the sensor pose corresponding to the detection information is consistent with the status information. The poses of AUVs are consistent and correspond in time; a digital twin snapshot is an immutable data structure containing information such as logical time, data validity, position and attitude of each AUV in the AUV set, velocity, trajectory index, sonar frame index, and scene version. It is used as a single data source to drive the generation of images on multiple display ends within a display cycle. During the validity period of the snapshot, all the data contained therein remains unchanged, thus ensuring that the data used by each display end within the same display cycle is completely consistent. By completing the time alignment of state information and detection information in the snapshot generation step, spatiotemporally consistent input data is provided for subsequent spatial mapping and synchronous display.

[0026] Specifically, using the generated logical time as the query index, the system locates and extracts the AUV state information corresponding to this time from the state history buffer sorted by logical time. If the logical time precisely matches the logical time of a certain historical state frame, this frame is directly extracted as the corresponding state information. If the logical time is between two adjacent state frames, the adjacent frames are selected and the state information for this time is generated by interpolation. Specifically, linear or spline interpolation is performed on the position parameters, and spherical linear interpolation is performed on the attitude parameters after converting them to quaternions. The interpolated position information and attitude information are then combined into the complete state information corresponding to the logical time. If a valid state frame for the logical time does not exist in the state history due to network interruption or other reasons, the most recent valid state is retained and marked as outdated, and no unbounded extrapolation is performed. At the same time, the system acquires the detection information associated with the extracted state information. In an optional implementation, the sonar detection module... When a detection is triggered, the trigger time and the corresponding sensor pose are recorded, and the generated detection results are associated with and stored with the sensor pose. When the system needs to obtain detection information associated with the state information, it matches the corresponding detection time based on the logical time, associates the detection information belonging to the same logical time with the state information, and then aligns the obtained state information and detection information based on the logical time, so that the sensor pose corresponding to the sonar detection result and the AUV pose in the state information are accurately matched in the time dimension. Finally, the system assembles the time-aligned state information, detection information, along with data such as trajectory index, sonar frame index, and current scene version number, into an immutable digital twin snapshot, and assigns a unique snapshot identifier and display generation to the snapshot to identify the snapshot's position in the time series. Once the snapshot is generated, it cannot be modified and remains unchanged within the current display cycle. Each display terminal uses the snapshot as the sole data source for image generation.

[0027] In this embodiment, state information is extracted from the state history using a unified logical time as an index, and the state information and detection information are time-aligned using the same logical time. This enables the sonar detection results to accurately match the AUV's true pose at that moment, fundamentally solving the problem of sonar image and AUV attitude mismatch caused by different generation frequencies and transmission delays between state data and sonar images in existing technologies. This ensures the accuracy of the positional superposition of detection information in the subsequent holographic sand table. By using position interpolation and attitude spherical linear interpolation techniques to generate state information corresponding to the logical time, smooth and continuous state data can be obtained even if there are no precisely corresponding sampling points between adjacent state frames. This effectively avoids AUV model motion jumps caused by irregular network sampling, improving the smoothness and visual continuity of the digital twin's motion in the holographic sand table. When network data is interrupted, the system maintains the most recently valid state and marks it as outdated instead of performing unbounded extrapolation, avoiding erroneous virtual poses in the absence of data and ensuring the system's security and reliability. By assembling time-aligned state information with detection information into an immutable digital twin snapshot, the ordinary monitoring terminal and the holographic sand table terminal share a unique data source within the same display cycle, ensuring that the AUV attitude, track, and sonar detection results observed by multiple display terminals are completely consistent in the time dimension. This eliminates display differences caused by different display terminals using data from different times, providing shore-based personnel with accurate and unified underwater situational awareness information. Finally, by assigning unique identifiers and display generations to snapshots, a reliable basis for judgment is provided for subsequent asynchronous rendering result management and multi-version scene switching, ensuring data consistency of the system in high-concurrency interactive scenarios.

[0028] 103. Based on the preset spatial calibration relationship, the geospatial information in the digital twin snapshot is mapped to the physical display space of the holographic sand table to generate a mapped digital twin snapshot.

[0029] The preset spatial calibration relationship refers to the coordinate transformation relationship between the coordinate system of the geospatial information and the coordinate system of the physical display space of the holographic sand table, which is established in advance through a calibration process. This relationship is stored in the form of a homogeneous transformation matrix, including translation, rotation, scaling, and necessary axial inversion components, used to convert any geospatial coordinates into coordinates in the physical display space of the sand table. Geospatial information refers to the AUV location information carried in the digital twin snapshot, including at least longitude, latitude, and depth data, and may further include spatial points with geographic coordinates such as track points, terrain feature points, and detection target points. The physical display space of the holographic sand table refers to the effective display space of the holographic projection sand table. The physical three-dimensional space corresponding to the display area has a defined physical size, display resolution, and viewing direction. It is the target space of the geospatial information mapped into the physical world after coordinate transformation. The mapped digital twin snapshot refers to the new snapshot generated after converting all the geospatial information in the original digital twin snapshot to the physical display space of the holographic sand table. All spatial points in the snapshot are expressed based on the physical coordinates of the sand table and can be directly used to drive the holographic display device to generate images. By completing the transformation of all spatial coordinates in advance in the snapshot, the sand table display end does not need to perform real-time coordinate conversion during the rendering process, which reduces the computational load of the display end and simplifies the processing logic of the display end.

[0030] Specifically, the system first acquires geospatial information from a digital twin snapshot, storing the information in the form of geographic coordinates. Then, using a preset geographic reference origin, the system converts longitude and latitude data into horizontal positions in the engine world coordinate system and depth data into vertical positions, generating engine world coordinates. After completing the conversion from geospatial information to engine world coordinates, the system further maps the engine world coordinates to the physical display space of the holographic sand table according to a preset spatial calibration relationship—the transformation matrix from the engine world coordinate system to the physical display space coordinate system of the holographic sand table. For position information, the system uses a homogeneous transformation matrix T_HU = T (translation) x R (rotation) x S (scale) to perform a linear mapping from the engine world coordinates to the local coordinates of the sand table. First, the attitude information is obtained through a control point calibration process. For attitude information, the system first converts the heading angle, pitch angle, and roll angle into rotation quaternions, and then combines the geographic tangent plane direction with the sand table axis transformation to obtain the sand table display attitude, so as to keep the AUV's orientation in the sand table consistent with the actual navigation direction. In an optional implementation, when the holographic sand table is a planar display structure, the system uses at least three non-collinear control points to calculate the planar mapping relationship. When the holographic sand table is a stereoscopic display structure with a real three-dimensional volume, the system uses at least four non-coplanar control points to calculate the three-dimensional transformation relationship. After completing the conversion of all geospatial information to the physical display space of the sand table, the system encapsulates all the converted spatial data into a mapped digital twin snapshot. All spatial information in the snapshot is expressed in the physical coordinates of the sand table and can be directly used to drive the display end to generate the screen.

[0031] In this embodiment, by establishing geographic coordinates and then converting them into physical coordinates of the sand table using engine world coordinates, a hierarchical transformation link is formed. All coordinate mappings are pre-completed at the snapshot level, ensuring a clear and calculable correspondence between the geographic coordinates of the sea area and the physical coordinates of the sand table. This fundamentally solves the problems of disproportionate terrain, track, and detection results caused by inconsistent coordinate systems in existing technologies, as well as positional drift, guaranteeing the precise spatial correspondence of all displayed elements in the holographic sand table. By employing control point calibration and reprojection error verification mechanisms, the accuracy of the transformation matrix is ​​ensured to meet display requirements. By saving the calibration version and marking calibration failures when device parameters or scene ranges change, errors caused by changes in display resolution, sand table size, or scene origin are effectively avoided. The system eliminates display deviations, ensuring consistency and reproducibility across different configuration environments. By generating pre-mapped digital twin snapshots with fully transformed coordinates, both the ordinary monitoring terminal and the holographic sand table terminal generate their respective viewpoints based on the same snapshot data. The AUV positions, tracks, and detection results at both ends remain spatially consistent, eliminating visual discrepancies between multiple display terminals and providing shore-based personnel with a unified and accurate underwater situational awareness view. Finally, by uniformly completing spatial mapping in the snapshots instead of real-time coordinate conversion on the display terminals, the computational overhead of the display terminals is reduced, simplifying the rendering logic of multiple display terminals. This allows the system to achieve synchronous output with lower latency when driving multiple display terminals, improving the real-time response performance of the holographic sand table interaction.

[0032] 104. Based on the mapped digital twin snapshot, synchronously drive multiple display terminals to generate visualizations corresponding to logical moments within the same display cycle.

[0033] Among them, the mapped digital twin snapshot refers to the data set generated after converting all geospatial information in the original digital twin snapshot to the physical display space of the holographic sand table through a preset spatial calibration relationship. All spatial points in the snapshot are expressed in the physical coordinates of the sand table and can be directly used to drive the display end to generate images. The same display cycle refers to the fixed time interval corresponding to one complete iteration of the system's main loop. Within this time interval, the system completes the entire processing flow from reading the snapshot to driving all display ends to generate images. All display ends generate images based on the same mapped digital twin snapshot within the same display cycle, ensuring that the images output by each end are precisely synchronized in time. The display end refers to the output device connected to the system and used to present the visualized images, including at least The control terminal display device and the holographic sand table display device are used. The control terminal display device can be a regular monitor to present traditional two-dimensional or three-dimensional perspective images. The holographic sand table display device includes a monitor or video output device, a holographic imaging structure and a structural carrier, used to present stereoscopic images in the physical display space of the holographic sand table. The visualization screen refers to the visual output that can be observed by the human eye, generated based on the spatial data in the mapped digital twin snapshot. It includes at least semantic layers such as the AUV digital twin model, flight path, sonar detection sector, terrain layer and status sign. By driving all display terminals to refresh synchronously with a unified snapshot in each display cycle, the control terminal screen and the holographic sand table screen have the same time and spatial reference.

[0034] Specifically, using mapped digital twin snapshots as input data, within their validity period, this serves as the sole data source driving the generation of visualizations on all display terminals. Each display terminal does not directly read network reception caches or playback files. The system first determines the start time of the current display cycle and uses the snapshot data at that time as the basis to drive the rendering of images on both the ordinary monitoring terminal viewport and the holographic sand table viewport. For the ordinary monitoring terminal, the system generates a traditional display image containing a 3D scene, AUV model, track lines, status signs, and sonar images based on the AUV pose data, track data, and detection information from the mapped digital twin snapshot. For the holographic sand table terminal, the system synthesizes the terrain layer, AUV layer, track layer, sonar layer, and interactively selected status layer according to a preset hierarchical order based on the spatial data from the mapped digital twin snapshot that has been pre-converted to the physical coordinates of the sand table. The holographic display structure generates corresponding single-view, dual-view, or multi-view images, presenting stereoscopic images in the physical display space of the holographic sand table. During the compositing process, the system employs visible range clipping and hierarchical detail strategies to control the rendering load of each layer. Objects exceeding the effective display area of ​​the sand table are clipped, and the detail level of objects far from the observation position is reduced. In one optional implementation, the terrain layer uses low-frequency colors and contour lines to represent the overall shape, the AUV layer uses high-contrast outlines for highlighting, the trajectory layer decays color intensity over time, and the sonar layer displays fan-shaped boundaries, hit point clouds, or target detection areas. Each layer can be turned on and off independently, but all layers are generated based on the same mapped digital twin snapshot. All rendering work on the display end must be completed within the current display cycle and output uniformly at the boundary moments of the display cycle to ensure precise temporal synchronization of all images. When a logical moment or scene version jump occurs, the system increments the display generation and discards asynchronous rendering tasks of the old generation that have not yet been completed within the current display cycle to prevent old frames from overwriting new frames.

[0035] In this embodiment, by using the mapped digital twin snapshot as the sole data source for each display terminal within the same display cycle, the ordinary monitoring terminal screen and the holographic sand table screen are generated based on the exact same logical time, the exact same AUV status, and the same detection information. This ensures that the digital twin status presented by multiple display terminals at the same time is completely consistent, allowing shore-based personnel to observe a unified situation at the same time and in the same scene on different display terminals. By presenting the three-dimensional spatial relationship of AUV position, attitude, track, and sonar detection results in three-dimensional physical space in the holographic sand table, shore-based personnel can intuitively observe the underwater operation situation without switching between multiple two-dimensional interfaces and manually establishing spatial correspondences, reducing the cognitive burden on operators and improving the efficiency and accuracy of underwater situational awareness. Within the cycle, unified driving of all display terminals is completed, ensuring precise time synchronization between the control terminal screen and the holographic sand table screen. This eliminates visual delays caused by rendering time differences between different display terminals, guaranteeing a real-time feedback experience for interactive operations. Through layer-by-layer compositing, visible range clipping, and hierarchical detail strategies, the rendering load is effectively controlled while ensuring rich content in the holographic sand table screen. This avoids performance degradation caused by driving multiple display terminals simultaneously, ensuring smooth operation of the system in complex marine scenes. Through a display generation management mechanism, the generation is automatically incremented and asynchronous rendering tasks of the old generation are discarded when a logical moment or scene version changes. This effectively prevents screen corruption caused by old frames overwriting new frames, ensuring the correctness and continuity of the displayed screen during snapshot switching, and improving the stability and reliability of the system.

[0036] This invention provides an interactive method for marine digital twin holographic sand table. First, a unified logical time is generated based on the timestamps of the state data, incorporating all data into the same time base. Then, state information corresponding to the logical time is extracted from historical state information, and the detection information and state information are time-aligned based on the logical time to generate a digital twin snapshot. This ensures precise matching between the attitude of the autonomous underwater vehicle and the sonar image at the same moment, eliminating the phenomenon of model motion jumps caused by network jitter. Second, based on a preset spatial calibration relationship, the geospatial information in the digital twin snapshot is mapped to the physical display space of the holographic sand table, establishing a calculable correspondence between the geographic coordinates of the sea area and the physical coordinates of the sand table. This method avoids proportional drift in terrain, track, and detection results when the geographical range, display device, or observation perspective changes, ensuring that multiple displays simultaneously present a twin state under a unified coordinate reference. Finally, by acquiring state data and generating logical moments, then performing time alignment to generate snapshots, and finally synchronously driving the displays through spatial mapping, the autonomous underwater vehicle's state data, detection information, and holographic sand table display are uniformly driven within the same cycle. This achieves real-time closed-loop interaction from data reception to holographic presentation, ensuring that the control terminal screen and the holographic sand table screen are output based on the same logical moment and the same coordinate reference, and are synchronously refreshed within the same display cycle, guaranteeing complete consistency of the digital twin state observed by multiple displays. This method allows shore-based personnel to intuitively and accurately observe the three-dimensional spatial relationships between the autonomous underwater vehicle's position, attitude, track, and detection results in the holographic sand table without switching between multiple two-dimensional interfaces and manually establishing spatial correspondences, thus improving the efficiency and accuracy of underwater situational awareness.

[0037] This application provides another method for interactive marine digital twin holographic sand table, such as... Figure 2 As shown, it specifically includes: 201. Establish scene and device configuration.

[0038] In this embodiment, before executing the holographic sand table interaction method, the system first performs an initialization configuration step, reading the preset coordinates of the marine geographic reference origin, the seabed topography resource file of the operating sea area, the AUV 3D model file, the configuration parameters of the sonar equipment such as the horizontal beam angle, vertical beam angle, detection range, and number of beams, the IP address and port number of the network communication endpoint, and the physical parameters of the holographic display device such as the effective display area size, resolution, and display structure type. After the above resources are loaded, the system generates a unique configuration identifier and an initial scene version number for this session, which are used to identify this session in the system operation log and task file, and serve as the basis for version tracking when the scene content changes in the future, ensuring that the display object, collision geometry, and sonar echo object use the same scene state.

[0039] 202. Perform holographic sand table spatial calibration.

[0040] Specifically, the method for establishing spatial calibration relationships includes selecting at least three non-collinear physical control points in the physical display space of the holographic sandbox, and determining virtual control points corresponding to the physical control points in the engine world scene; calculating the transformation matrix between the coordinate system of the engine world scene and the coordinate system of the physical display space based on the physical control points and the corresponding virtual control points; calculating the reprojection error of the transformation matrix based on the physical control points and the corresponding virtual control points; and using the transformation matrix as the preset spatial calibration relationship when the reprojection error is less than a preset threshold.

[0041] In this embodiment, a mapping relationship is established between the engine world coordinates and the holographic sandbox physical coordinates to ensure that all spatial points in the subsequent virtual scenes can accurately correspond to the physical space of the sandbox. Specifically, before calibration, the operator needs to determine at least three non-collinear virtual control points in the engine world scene and mark the physical control points corresponding to each virtual control point in the physical display space of the holographic sandbox. Each control point should be distributed as dispersedly as possible at the edge and center of the effective display area of ​​the sandbox to cover the entire mapping range. The system reads the spatial coordinates of each physical control point and its corresponding virtual control point, and calculates the transformation matrix between the engine world scene coordinate system and the physical display space coordinate system based on the two sets of corresponding points. The matrix is ​​a homogeneous transformation matrix containing translation, rotation, scaling, and necessary axial reversal components. If the number of control points exceeds three, the system uses the least squares fitting method to calculate the optimal transformation matrix. After the transformation matrix is ​​calculated, the system calculates the reprojection error based on each physical control point and its corresponding virtual control point, that is, uses the calculated transformation matrix to calculate the reprojection error. After the transformation matrix projects the virtual control points onto the sand table space, it compares the actual positions with the physical control points and calculates the deviation between the two to verify the accuracy of the transformation matrix. The smaller the reprojection error, the higher the calibration accuracy. When the reprojection error is less than a preset threshold (1 mm), the system saves the transformation matrix as a preset spatial calibration relationship, along with the calibration version and the specific reprojection error value. Subsequent holographic sand table image generation is based on coordinate transformation performed according to the calibration relationship. When the reprojection error is greater than or equal to the preset threshold, the system prevents the transformation matrix from being saved as a calibration relationship and prompts for recalibration, thus avoiding entering the formal display state when the calibration accuracy is insufficient. Through the above calibration and verification mechanism, a calculable and accurate correspondence is established between the marine geographic coordinates, engine world coordinates, and sand table physical coordinates. This fundamentally eliminates the problems of scale misalignment and position drift caused by inconsistencies in coordinate systems in traditional solutions, ensuring the spatial accuracy and display consistency of each element in the holographic sand table image.

[0042] 203. Receive and verify AUV data.

[0043] In this embodiment, the system continuously monitors the AUV status data port through a network communication interface. When a UDP or TCP data frame is received, the system first performs frame header recognition to confirm the AUV status data frame; then, it performs length verification to check whether the data frame length is consistent with the fixed length specified in the protocol; finally, it performs checksum verification to verify whether any errors occurred during data transmission. After all three verifications pass, the system decodes the data frame, extracting status parameters such as AUV identifier, longitude, latitude, depth, heading angle, pitch angle, roll angle, forward speed, lateral speed, vertical speed, thruster speed, and rudder angle. The system also performs unit normalization on each field and marks the received time with a timestamp for the successfully verified data frame. The data is then stored in the state history information corresponding to the AUV identifier, serving as the data source for subsequent snapshot generation. If any step of frame header identification, length verification, or checksum verification fails, the data frame is considered invalid, and the system discards it directly, preventing corrupted or forged data from contaminating the state history. In playback mode, the system does not rely on real-time state frames received from the network but enters playback mode in response to state data read operations, reading historical state data from the storage medium as the input source for state history information. Through verification and filtering mechanisms, it is ensured that every frame of data entering the state history information is complete and valid, providing a reliable data foundation for the generation of subsequent logical moments and the construction of digital twin snapshots.

[0044] 204. Generate target display time.

[0045] In this embodiment, the system generates a target display time based on the sampling period of the most recent valid state and the current operating mode. The target display time is the unified time anchor point for all subsequent processing steps, and different generation strategies are adopted under different operating modes. In real-time mode, the system uses the current monotonic clock minus the adaptive display delay as the target display time. Specifically, the adaptive display delay is determined by the base delay value and the statistical value of the most recent sampling interval. The base delay value is used to compensate for the fixed transmission time from data generation to data arrival, while the statistical value of the most recent sampling interval is dynamically calculated based on the historical interval of data packet arrival to reflect the current network transmission status. When the network data packet arrival interval is stable, the delay value converges to the optimal display time; when the network jitter... The system automatically adjusts the latency value to smooth out the uncertainty of data arrival, ensuring visual continuity of AUV movement within the sand table. In playback mode, the system does not use real-time clocks or network data, but instead employs a playback logic clock. The logic clock is determined by the playback progress. Upon receiving playback control commands, the system updates the logic clock value accordingly based on the command type. When paused, the logic clock stops incrementing; when adjusting the speed, the logic clock increment rate changes accordingly; and when the progress jumps, the logic clock jumps directly to the specified position, thus supporting flexible historical task review operations. Through these two different mode adaptation time generation strategies, the system ensures that it can generate stable and accurate target display times in both real-time monitoring and historical playback, providing a unified time reference for subsequent snapshot generation.

[0046] 205. Form a unified digital twin snapshot.

[0047] Specifically, when there is no corresponding state frame in the state history information at a logical moment, an adjacent state frame of the logical moment is selected; position interpolation is performed on the position parameters in the adjacent state frame to generate interpolated position information corresponding to the logical moment; attitude interpolation is performed on the attitude parameters in the adjacent state frame to generate interpolated attitude information corresponding to the logical moment; and the interpolated position information and interpolated attitude information are combined into the state information corresponding to the logical moment.

[0048] In the embodiments of this application, such as Figure 3As shown, using the target display time as an index, the state information corresponding to the target display time is extracted from the state history information sorted by logical time. If there is a state frame in the state history that exactly matches the target display time, it is directly extracted. If there is no exactly matching frame, the interpolation process is triggered. The interpolation process includes selecting the adjacent state frames surrounding the target display time, performing linear or spline interpolation on the position parameters in the adjacent state frames to generate the interpolated position information corresponding to the target display time; at the same time, the attitude parameters in the adjacent state frames are first converted into rotation quaternions, and then spherical linear interpolation is performed to generate the interpolated attitude information corresponding to the target display time; finally, the interpolated position information and the interpolated attitude information are combined into complete state information.

[0049] By combining position interpolation with attitude spherical linear interpolation, the attitude jump phenomenon caused by direct linear interpolation of Euler angles is effectively avoided, so that the motion attitude of the AUV in the holographic sand table remains smooth and continuous. Even when the data sampling interval is uneven, a smooth visual effect can still be obtained. When the valid state frame of the target time is missing in the state history due to network interruption or other reasons, the system retains the most recent valid state and marks the data as old, without performing unbounded extrapolation, thereby avoiding the generation of incorrect virtual poses in the absence of data.

[0050] After the state information is extracted, the system executes the detection information acquisition and binding process, sends a detection trigger command to the autonomous underwater vehicle (AUV) and records the trigger time, and determines the sensor pose of the AUV at the trigger time; based on the sensor pose, it determines the target detection space, transmits a detection signal to the target detection space, receives the echo signal and generates the detection result; it associates and stores the detection result with the sensor pose, and uses the associated and stored detection result as the detection information associated with the state information; by binding the detection result with the sensor pose, it ensures that the sonar detection data corresponds precisely to the AUV's true pose at the time of triggering the detection, thus avoiding the problem of mismatch between sonar image and AUV attitude from the source.

[0051] After completing the extraction of state information and acquisition of detection information, the system performs time alignment between the state information and the detection information based on the logical time. That is, using the logical time as a reference, it ensures that the pose of the AUV in the state information is completely consistent with the sensor pose corresponding to the detection information in the time dimension. Subsequently, the system assembles the time-aligned state information and detection information, along with data such as trajectory index, sonar frame index, current scene version number, data age, and validity flag, into a complete digital twin snapshot, and assigns a unique snapshot identifier and display generation to the snapshot. Once the snapshot is generated, it cannot be modified and remains unchanged within the current display cycle. Each display terminal uses this snapshot as the sole data source for image generation, fundamentally eliminating display differences caused by different display terminals using data from different times. Through the time alignment and snapshot encapsulation mechanism, the AUV attitude and sonar image are accurately matched at the same time, providing spatiotemporally consistent input data for subsequent spatial mapping and synchronous display.

[0052] 206. Update the 3D scene and AUV digital twin.

[0053] Specifically, longitude, latitude, and depth data from geospatial information are acquired; using a preset geographic reference origin as a benchmark, the longitude and latitude data are converted into horizontal positions in the coordinate system of the engine world scene, and the depth data is converted into vertical positions in the coordinate system of the engine world scene; engine world coordinates are generated based on the horizontal and vertical positions; and the engine world coordinates are mapped to the physical display space of the holographic sandbox based on a preset spatial calibration relationship.

[0054] In this embodiment, the digital twin snapshot generated in the preceding steps is used as input. First, the longitude, latitude, and depth data of the AUV are extracted from it. Using a preset marine geographic reference origin as a benchmark, the longitude and latitude data are converted into horizontal positions in the engine world scene coordinate system, i.e., the X and Z axes, and the depth data is converted into vertical positions in the engine world scene coordinate system, i.e., the Y axis, generating the precise three-dimensional coordinates of the AUV in the engine world scene. Then, the AUV digital twin model is moved from its current position to the coordinate system, and the model is rotated according to the attitude information in the snapshot to make its orientation consistent with the actual flight direction. At the same time, the AUV's trackline is updated in the engine world scene based on historical trajectory points. Then, the system uses the scene version in the digital twin snapshot... This module loads the corresponding scene content, including loading the seabed topography grid of the operational area from the terrain resource library, as well as loading obstacles or mission target objects placed by the editing module during runtime. This ensures that the displayed objects, virtual collision objects, and detectable geometry used by subsequent sonar detection are completely consistent, avoiding the problem of changing only the visual model without changing the detection scene. In addition, the system also generates auxiliary layers such as status text labels, depth curves, height-off-bottom curves, and minimaps, which are output to the control terminal display device along with the 3D scene. Through the above scene updates and auxiliary information overlay, shore-based personnel can obtain a complete AUV operational situational awareness on the ordinary display terminal, while the holographic sand table terminal will generate corresponding stereoscopic display images in subsequent steps based on the same snapshot.

[0055] 207. Perform sonar detection and result binding.

[0056] Specifically, a detection trigger command is sent to the autonomous underwater vehicle and the trigger time is recorded to determine the sensor pose of the autonomous underwater vehicle at the trigger time; the target detection space is determined based on the sensor pose, a detection signal is transmitted to the target detection space, the echo signal is received and the detection result is generated; the detection result is associated with the sensor pose and stored, and the associated and stored detection result is used as the detection information associated with the state information.

[0057] In the embodiments of this application, such as Figure 4As shown, the system sends a detection trigger command to the sonar detection component of the autonomous underwater vehicle (AUV) and records the trigger time, determining the sensor pose of the AUV at the trigger time. The pose includes the three-dimensional spatial position and three-axis orientation of the sonar device in the engine world scene, which serves as the reference for all subsequent detection signal transmission and echo calculation. Subsequently, the system determines the target detection space based on the horizontal and vertical beam angle parameters of the sonar, combined with the current sensor pose. This target space is a conical area covered by the sensor position as the origin, the sensor orientation as the central axis, and the horizontal and vertical beam angles as the boundaries. The system then transmits detection signals into the target detection space and uses the hardware ray tracing capability of the graphics processor to perform parallel scene intersection calculations on the two-dimensional beam array. This involves emitting rays along each beam direction from the sensor position, detecting whether the rays intersect with scene geometry such as seabed topography and obstacles, and returning the hit distance and surface normal. All beam intersection calculations are executed in parallel by the graphics processor, significantly improving the efficiency of the detection simulation.

[0058] After completing the intersection calculation, the system calculates the echo intensity based on the hit distance and surface normal of each beam, combined with the sound source parameters, propagation distance, seawater absorption, incident angle, and object acoustic reflection parameters. This intensity is then quantized into pixel values ​​to generate grayscale or pseudo-color sonar images. Simultaneously, it can further extract target candidate regions from background noise. Since the graphics processor's intersection calculation and echo image generation processes are executed asynchronously, the system records the display generation of the snapshot to which the detection task belongs when it is submitted. After asynchronous readback, the system verifies whether the snapshot generation bound to the detection result is consistent with the current display generation. Only when they are consistent is the detection result published as valid detection information. If the generations are inconsistent, it indicates that an error has occurred during the detection execution. If a logical jump or scene version change occurs, the detection results are discarded directly. Through a generational verification mechanism, the problem of old frames overwriting new frames due to asynchronous execution timing is effectively avoided, ensuring strict consistency between the detection results and the current display scene. Finally, the system associates and stores the detection results with the sensor pose as detection information associated with the state information. That is, the sonar image and detection results correspond precisely to the actual pose of the AUV when the detection was triggered. By binding the detection results with the sensor pose and verifying the generational validity after readback, the problem of mismatch between the sonar image and the AUV pose is avoided from the source, providing an accurate data foundation for the subsequent overlay and display of the sonar sector, hit point cloud, and target detection area in the holographic sand table.

[0059] 208. Generate a holographic sand table image.

[0060] Specifically, the fan-shaped boundary and hit point cloud from the probe information are extracted in the mapped digital twin snapshot, and the position data of the fan-shaped boundary and hit point cloud in the physical display space of the holographic sand table are obtained; the pose data of the autonomous underwater vehicle in the physical display space is obtained in the mapped digital twin snapshot; the position data of the fan-shaped boundary and hit point cloud are spatially aligned with the pose data of the autonomous underwater vehicle; and in the physical display space, the visualization screen corresponding to the logical moment is generated based on the spatially aligned position data.

[0061] In the embodiments of this application, such as Figure 4 As shown, the system uses a mapped digital twin snapshot as its data source. All spatial information in the snapshot, including AUV location, track points, terrain data, sonar sector boundaries, and hit point clouds, has been converted to the coordinate reference of the holographic sand table physical display space through the preset spatial calibration relationship established in step 202. Therefore, the system does not need to perform real-time coordinate conversion during the rendering process and can directly drive the generation of the sand table image. Specifically, the system extracts the sector boundaries and hit point clouds from the detection information in the mapped digital twin snapshot and obtains the position data of these sonar features in the sand table physical display space. Simultaneously, the system acquires the pose data of the autonomous underwater vehicle (AUV) in the physical display space of the sand table from the mapped digital twin snapshot; it spatially aligns the position data of the sector boundary and the hit point cloud with the pose data of the AUV, ensuring that the sonar sector appears at the correct azimuth and pitch angles in front of the AUV, so that the sonar detection coverage area observed in the holographic sand table precisely corresponds to the actual orientation of the AUV, thus intuitively presenting the spatial relationship of which direction the AUV is detecting and what it has detected; subsequently, the system integrates the terrain layer, AUV layer, trajectory layer, sonar layer, and intersection layer... The selected state layers are composited in a preset hierarchical order. Each layer can be turned on and off independently, but all are generated based on the same mapped digital twin snapshot, ensuring precise alignment between layers in time and space. During the compositing process, the system generates corresponding single-view, dual-view, or multi-view images based on the specific structure of the holographic display device, allowing observers to see a three-dimensional marine scene and AUV operational status in the physical space of the sand table without wearing auxiliary equipment. At the same time, the system adopts a visible range clipping strategy, clipping objects that exceed the effective display area of ​​the sand table and not rendering them, and a hierarchical detail strategy, reducing mesh precision and texture resolution for objects far from the observation position to control the rendering load and avoid performance degradation caused by driving multiple display ends simultaneously, ensuring smooth operation of the system in complex marine scenes. Through the above layer compositing and rendering load control, shore-based personnel can observe the three-dimensional spatial relationships between AUV position, attitude, track, and sonar detection results in the holographic sand table from an intuitive God's-eye view, without switching between multiple two-dimensional interfaces and manually establishing spatial correspondences, significantly improving the efficiency and accuracy of underwater situational awareness.

[0062] 209. Process the interaction and write back the scene state.

[0063] In this embodiment, when a user performs an operation in the physical space of the sand table using a spatial positioning input device, the system first converts the pose data of the input device in the tracking coordinate system into an interactive ray in the holographic sand table physical display space. This ray is a ray originating from the device's position and oriented towards the device, possessing a clear three-dimensional position and direction within the sand table space. Subsequently, the system performs intersection detection between the interactive ray and the bounding volumes or collision geometry of objects in the sand table space to determine the target object currently being pointed at or selected by the user. When the user performs operations such as clicking, dragging, rotating, or zooming, the system executes corresponding object operations based on the operation type, including AUV selection and viewpoint locking, and display of trajectory and sonar layers. The system includes functions such as displaying switches, playback progress control, and placing / moving / deleting scene objects. Modifications resulting from these interactive operations are encapsulated as interactive commands and submitted to the unified state management module. The module updates the scene state based on the command type, increments the scene version number, and generates a new display generation. The generation of the new display generation triggers updates to subsequent processing links, from regenerating the digital twin snapshot in step 205 to refreshing the holographic sandbox screen in step 208, forming a complete interactive loop. Through the complete link of interactive ray transformation, object picking and operation, command submission, and state write-back, every operation performed by the user in the holographic sandbox can be reflected on the display screen in real time, achieving a WYSIWYG interactive experience.

[0064] 210. Real-time display or task replay.

[0065] Specifically, in response to the read operation of state data, the system enters playback mode; in playback mode, it receives playback control commands and determines the target playback time based on the playback control commands; using the target playback time as the logical time, it extracts the historical state information corresponding to the logical time from the state history information of the autonomous underwater vehicle, and obtains the historical detection information associated with the historical state information; based on the logical time, it aligns the historical state information with the historical detection information in time to generate a historical digital twin snapshot.

[0066] In the embodiments of this application, such as Figure 5As shown, in real-time mode, the system continuously executes steps 203 to 209, that is, using the real-time received network data as input, it continuously generates digital twin snapshots and drives the holographic sandbox and control terminal to display synchronously. Simultaneously, in recording mode, it writes the verified AUV status frames and necessary scene events into the task file in chronological order and establishes a time index for subsequent playback. In playback mode, the system shuts down or isolates real-time network input, enters playback mode in response to status data reading operations, and uses the stored task file as the sole status source. Specifically, the system receives user data through holographic interaction or operation... The playback control command issued by the control terminal determines the target playback time based on the command. The target playback time is used as the logical time. The historical state information and historical detection information corresponding to the logical time are extracted from the task file. Based on the logical time, the historical state information and historical detection information are time-aligned to generate a historical digital twin snapshot. The historical digital twin snapshot then enters the processing link from step 206 to step 208 to update the 3D scene, perform sonar detection and result binding, and generate a holographic sand table screen. This enables real-time monitoring and historical review to share the same snapshot generation, spatial mapping, and synchronization drive link.

[0067] When the system switches between real-time and playback modes, since different modes correspond to different data sources and display generations, the system increments the display generation at the time of switching and clears asynchronous rendering tasks and temporary display caches that have not yet been completed within the current display cycle and belong to the old generation. This ensures that the snapshots and images generated in the new mode are not affected by the data left over from the old mode, and guarantees that the content displayed on the holographic sandbox and the control terminal is completely consistent. Through the above mechanism of sharing the link between real-time and playback, the historical task review can fully reuse the holographic presentation method of real-time monitoring, so that operators can obtain the same immersive observation experience as real-time monitoring during the playback process, which significantly improves the utilization efficiency of historical data and the intuitiveness of review analysis.

[0068] 211. Implement multi-AUV collaborative organization.

[0069] In this embodiment, when multiple AUV identifiers exist in a session, the system independently maintains its own state history information, trajectory data, and sonar detection queue for each AUV identifier. Each AUV can have a different data reporting cycle. The system does not require all data packets to arrive simultaneously. Instead, it extracts the state information of each AUV from its state history at a common target display time by interpolation. Then, it aligns and combines the state information of all AUVs at the target display time to generate a ensemble digital twin snapshot containing multiple AUVs. Each AUV in the ensemble snapshot carries its own trajectory index and sonar frame index. Subsequent spatial mapping and synchronization driving steps process all AUVs in the ensemble snapshot uniformly, so that the holographic sandbox presents the multi-AUV situation at the same logical moment in the same display frame, rather than a mixture of the most recently arrived states of each data packet. Based on the task group identifier, role identifier, expected relative position, and task partition of each AUV, the system generates semantic layers such as formation lines, task partition boundaries, and safety envelopes in the holographic sandbox. Formation lines are used to visually display the relative positional relationships between AUVs belonging to the same task group, and task partitions are used to identify the detection areas that each AUV is responsible for. The safety envelope is used to characterize the safe distance range around each AUV. Based on this, the system estimates the relative distance, relative azimuth, depth difference, and predicted nearest approach distance between each AUV according to its current position and speed. When the predicted nearest approach distance is lower than a preset safety threshold, a conflict warning is generated in the holographic sand table, which alerts the operator by changing the outline color of the corresponding AUV, highlighting the safety envelope, or displaying warning lines. When multiple AUVs are equipped with sonar detection equipment, the system uses a sonar scheduler to perform time-division scheduling management of the detection requests of multiple AUVs. The scheduler is for each... The detection request records the AUV identifier, Ping sequence number, target time slot, and scene version. Detection requests are submitted using polling, priority, or time-division triggering by task area to limit the number of concurrent detections on the same graphics processor and avoid computational resource contention and performance degradation caused by multiple AUVs performing GPU ray tracing simultaneously. Through this multi-AUV collaborative organization mechanism, operators can simultaneously observe the formation status of multiple AUVs, their respective sonar detection coverage, and their mutual safety relationships in a holographic sandbox, improving situational awareness and command and decision-making efficiency in multi-AUV collaborative operation scenarios.

[0070] Based on this, the present invention provides a marine digital twin holographic sand table system, such as... Figure 6As shown, it includes a data access and protocol processing module, a time and status management module, a geospatial conversion module, a marine 3D scene module, an AUV digital twin module, a multi-AUV collaborative management module, a sonar virtual detection module, a mission recording and playback module, a runtime scene editing module, a holographic sand table calibration and rendering module, an immersive interaction module, and a display synchronization control module. Each module is decoupled through a unified digital twin snapshot. The digital twin snapshot includes at least the logical time, data validity, AUV set, position, attitude, velocity, trajectory index, mission group, sonar frame index, and scene version of each AUV. The data access and protocol processing module receives AUV status data, control feedback data, and sonar data via UDP and / or TCP. It performs frame header identification, length verification, checksum verification, field decoding, and unit normalization on the data frames. Each AUV has a unique AUV identifier within the session. AUV status includes at least the identifier, longitude, latitude, depth, heading angle, pitch angle, roll angle, forward speed, lateral speed, vertical speed, thruster speed, and rudder angle. The module's output data serves as the input to the time and status management module. The time and status management module stores an ordered history of AUV statuses according to their identifiers. The module records the monotonic moment and logical simulation moment of reception, determines the display delay based on the most recent sampling period, interpolates position, attitude, and continuous physical quantities between adjacent valid states, maintains the most recent valid state and sets a data stale flag when subsequent samples are missing, and does not perform unbounded extrapolation. The module generates a unified digital twin snapshot indexed by the target display time, serving as the sole data source for each display terminal within the same display period. The geospatial conversion module is used to convert the longitude, latitude, and depth of the geospatial information in the digital twin snapshot into a three-dimensional position in the engine's world scene coordinate system, using the selected sea area's geographic reference origin as a benchmark. Based on the preset spatial calibration relationship, the engine world coordinates are mapped to the physical display space of the holographic sand table. Position transformation and attitude transformation are handled separately, and the semantics of ocean coordinates with positive depth are uniformly transformed in the display coordinate system. The ocean 3D scene module is used to load environmental objects such as seabed topography, water bodies, lighting, ice bodies or obstacles, and generate scene content that can be shared by ordinary viewport and holographic sand table viewport according to the current sea area range and display level. The AUV digital twin module is used to update the position, rotation and motion performance of the AUV model according to the digital twin snapshot, and at the same time generate semantic layers such as track, status sign and detection range.The sonar virtual detection module uses the pose of the AUV or sonar components as the sensor pose, generates a two-dimensional ray array according to the horizontal and vertical beam angles, determines the target detection space based on the sensor pose, obtains the hit distance and surface normal through ray tracing using a graphics processor, and generates grayscale or pseudo-color sonar images, detection sectors, hit point sets, and target detection results by combining propagation loss, incident angle, acoustic material, noise, and detection algorithms. The sonar detection results are associated with and stored as the detection information associated with the sensor pose that triggered the detection. The task recording and playback module is used to write the verified AUV status frames into the task file in real time in chronological order and establish a time index. In playback mode, it responds to the reading operation of status data. Upon entering playback mode, the system reads status frames from the task file, using the target playback time determined by the playback control command as the logical time. Historical status and detection information are extracted, and a historical digital twin snapshot is generated. Pause, speed adjustment, skipping, and looping all alter the playback logic clock. The display synchronization control module ensures that the normal viewport, holographic sand table, track, and sonar layers are refreshed at the same target time. The runtime scene editing module receives scene editing commands during simulation, updates scene objects according to these commands, and increments the scene version number. The scene version is written to the digital twin snapshot. The sonar virtual detection module reads detectable geometry from the same scene version, ensuring consistency between the sand table display objects, virtual collision objects, and sonar echo objects. (The last sentence appears to be incomplete and possibly refers to a holographic view.) The sand table calibration and rendering module is used to save the effective display area, physical size, projection frustum, display device parameters, and at least one set of control point correspondences of the sand table. By selecting at least three non-collinear physical control points in the physical display space of the holographic sand table and determining the corresponding virtual control points in the engine world scene, the transformation matrix between the engine world scene coordinate system and the physical display space coordinate system is calculated. Then, the reprojection error of the transformation matrix is ​​calculated based on each physical control point and its corresponding virtual control point. When the reprojection error is less than a preset threshold, the transformation matrix is ​​saved as a preset spatial calibration relationship. Single-view, dual-view, or multi-view images are generated according to the holographic display structure, and the terrain, AUV, trajectory, sonar coverage area, and interactive selection state are layered and synthesized. The immersive interaction module receives pose and key events from controllers, head-mounted displays, infrared positioning devices, touchscreens, mice, or other input devices. It transforms the interaction ray from device coordinates to sandbox coordinates and engine world coordinates, and completes object picking, view switching, layer switching, playback control, obstacle editing, and parameter adjustment. The interaction results are written back to the unified scene state and synchronized to each display terminal in the next display submission cycle. The display synchronization control module reads an immutable digital twin snapshot once in each display cycle and uses the snapshot to generate the control terminal screen and the holographic sandbox screen respectively. When the logical moment, running mode, scene version, or playback jump changes, the display generation is incremented and the asynchronous sonar or rendering results of the old generation are discarded to prevent old frames from overwriting the new state.The multi-AUV collaborative management module is used to establish mutually isolated status histories, trajectories, sonar queues, and control channels according to AUV identifiers. It interpolates data for each AUV at the same target display time and assembles them into an AUV ensemble snapshot. The module calculates relative distances, relative azimuths, depth differences, and predicts the nearest approach distance based on task group relationships, generating formation alignments, task partitions, and conflict warning layers.

[0071] Furthermore, the system hardware provided in this application includes at least a computing and control unit, a holographic display unit, an interaction and positioning unit, and a network communication unit. The computing and control unit includes a graphics processor with hardware ray tracing capabilities, a central processing unit, memory, and storage devices, used to perform scene rendering, coordinate transformation, state interpolation, sonar ray tracing, and task file reading and writing. The holographic display unit includes a display or video output device, a holographic imaging structure, and a structural carrier, connected to the computing and control unit, used to receive the display images generated by the computing and control unit and present holographic stereoscopic images in the physical space of the sand table. The holographic imaging structure can adopt LED / LCD splicing display, oblique... The display structure can be a semi-reflective, semi-transparent imaging device, a naked-eye stereoscopic display, a light field display, or other structures capable of creating a sand table-like stereoscopic observation effect. The specific display structure does not constitute a limitation on the data processing and coordinate calibration methods of this invention. The interaction and positioning unit includes at least one spatial positioning input device and its matching signal receiving device, which is connected to the computing and control unit. It is used to obtain the user's position and orientation in the tracking coordinate system and transmit them to the computing and control unit. The input device can be a VR headset, a positioning handle, an infrared marker, a touch component, or a combination thereof. The network communication unit is connected to the computing and control unit and is used to connect to an external AUV simulation platform, an actual data forwarding device, or a sonar display terminal.

[0072] Furthermore, as Figure 1 In terms of specific implementation of the method, this application provides a marine digital twin holographic sand table interactive device, such as... Figure 7 As shown, the device includes: a time unification module 301, a snapshot generation module 302, a spatial mapping module 303, and a synchronous display module 304.

[0073] The clock unification module 301 is used to acquire the status data of the autonomous underwater vehicle and generate a logical time based on the timestamp of the status data. The snapshot generation module 302 is used to extract the state information corresponding to the logical time from the state history information of the autonomous underwater vehicle, obtain the detection information associated with the state information, and align the state information and the detection information in time based on the logical time to generate a digital twin snapshot. The spatial mapping module 303 is used to map the geospatial information in the digital twin snapshot to the physical display space of the holographic sand table based on a preset spatial calibration relationship, thereby generating a mapped digital twin snapshot. The synchronous display module 304 is used to synchronously drive multiple display terminals to generate visual images corresponding to logical moments within the same display cycle based on the mapped digital twin snapshot.

[0074] In specific application scenarios, the snapshot generation module 302 is specifically used to select an adjacent state frame of the logical time when there is no corresponding state frame in the state history information at the logical time; perform position interpolation operation on the position parameters in the adjacent state frame to generate interpolated position information corresponding to the logical time; perform attitude interpolation operation on the attitude parameters in the adjacent state frame to generate interpolated attitude information corresponding to the logical time; and combine the interpolated position information and the interpolated attitude information into the state information corresponding to the logical time.

[0075] In specific application scenarios, the snapshot generation module 302 is specifically used to send a detection trigger command to the autonomous underwater vehicle and record the trigger time, determine the sensor pose of the autonomous underwater vehicle at the trigger time; determine the target detection space based on the sensor pose, transmit a detection signal to the target detection space, receive the echo signal and generate the detection result; associate and store the detection result with the sensor pose, and use the associated and stored detection result as the detection information associated with the state information.

[0076] In specific application scenarios, the method for establishing the preset spatial calibration relationship in the spatial mapping module 303 includes: selecting at least three non-collinear physical control points in the physical display space of the holographic sandbox, and determining the virtual control points corresponding to the physical control points in the engine world scene; calculating the transformation matrix between the coordinate system of the engine world scene and the coordinate system of the physical display space based on the physical control points and the corresponding virtual control points; calculating the reprojection error of the transformation matrix based on the physical control points and the corresponding virtual control points; and using the transformation matrix as the preset spatial calibration relationship when the reprojection error is less than a preset threshold.

[0077] In specific application scenarios, the spatial mapping module 303 is specifically used to acquire longitude, latitude, and depth data from geospatial information; using a preset geographic reference origin as a benchmark, it converts the longitude and latitude data into horizontal positions in the coordinate system of the engine world scene, and converts the depth data into vertical positions in the coordinate system of the engine world scene, generating engine world coordinates based on the horizontal and vertical positions; and based on preset spatial calibration relationships, it maps the engine world coordinates to the physical display space of the holographic sandbox.

[0078] In specific application scenarios, the synchronous display module 304 is specifically used to extract the sector boundary and hit point cloud from the probe information in the mapped digital twin snapshot, and obtain the position data of the sector boundary and hit point cloud in the physical display space of the holographic sand table; obtain the pose data of the autonomous underwater vehicle in the physical display space in the mapped digital twin snapshot; spatially align the position data of the sector boundary and hit point cloud with the pose data of the autonomous underwater vehicle; and generate the visualization screen corresponding to the logical moment in the physical display space based on the spatially aligned position data.

[0079] In specific application scenarios, such as Figure 8 As shown, the above-mentioned device also includes a playback multiplexing module 305, which is specifically used to enter playback mode in response to the state data reading operation; in playback mode, it receives playback control instructions and determines the target playback time according to the playback control instructions; it uses the target playback time as the logical time, extracts the historical state information corresponding to the logical time from the state history information of the autonomous underwater vehicle, and obtains the historical detection information associated with the historical state information; it aligns the historical state information and the historical detection information in time based on the logical time to generate a historical digital twin snapshot.

[0080] It should be noted that other corresponding descriptions of the functional units involved in the marine digital twin holographic sand table interactive device provided in this embodiment can be found in [reference]. Figure 1 and Figure 2 The corresponding description in [the document] will not be repeated here.

[0081] Based on the above, Figure 1 and Figure 2 Accordingly, this embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described interactive method for marine digital twin holographic sand table.

[0082] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product to be identified can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), including several instructions to enable a computer device (such as a personal computer, server, or network device) to execute the marine digital twin holographic sand table interaction method of various implementation scenarios of this application.

[0083] Based on the above, Figure 1 and Figure 2 The method shown, and Figure 7 and Figure 8 The illustrated embodiment of the marine digital twin holographic sand table interactive device, in order to achieve the above objectives, such as... Figure 9As shown, this embodiment also provides a physical device for interactive marine digital twin holographic sand table. This device includes a communication bus, a processor, a memory, and a communication interface. It may also include input / output interfaces and a display device. The various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the marine digital twin holographic sand table interactive method described in the above embodiment.

[0084] Optionally, the physical device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0085] Those skilled in the art will understand that the structure of the interactive physical device for marine digital twin holographic sand table provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0086] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs to be identified. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms, or it can be implemented by hardware. By applying the technical solution of this application, firstly, a unified logical time is generated based on the timestamp of the state data, and all data are incorporated into the same time base. Then, by extracting the state information corresponding to the logical time from the state history information, and aligning the detection information and state information based on the logical time, a digital twin snapshot is generated, so that the attitude of the autonomous underwater vehicle and the sonar image are accurately matched at the same time, eliminating the phenomenon of model motion jump caused by network jitter. Secondly, based on the preset spatial calibration relationship, the geospatial information in the digital twin snapshot is mapped to the physical display space of the holographic sand table, so that the geographic coordinates of the sea area and the physical coordinates of the sand table are established in a calculable correspondence, avoiding geographic... The scaling of terrain, track, and detection results shifts with changes in range, display device, or observation angle, ensuring that multiple displays simultaneously present a twin state under a unified coordinate reference. Finally, by acquiring state data and generating logical moments, and then performing time alignment to generate snapshots, spatial mapping ultimately drives the displays synchronously. This unifies the driving of autonomous underwater vehicle (AUV) state data, detection information, and the holographic sand table display within the same cycle, achieving real-time closed-loop interaction from data reception to holographic presentation. This ensures that the control terminal screen and the holographic sand table screen are output based on the same logical moment and coordinate reference, and are synchronously refreshed within the same display cycle, guaranteeing complete consistency of the digital twin state observed by multiple displays. This method eliminates the need for shore-based personnel to switch between multiple two-dimensional interfaces and manually establish spatial correspondences. They can intuitively and accurately observe the three-dimensional spatial relationships between the AUV's position, attitude, track, and detection results in the holographic sand table, improving the efficiency and accuracy of underwater situational awareness.

[0088] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0089] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for interactive use of a marine digital twin holographic sand table, characterized in that, include: Acquire the status data of the autonomous underwater vehicle and generate a logical time based on the timestamp of the status data; Extract the state information corresponding to the logical moment from the state history information of the autonomous underwater vehicle, obtain the detection information associated with the state information, and time-align the state information and the detection information based on the logical moment to generate a digital twin snapshot; Based on a preset spatial calibration relationship, the geospatial information in the digital twin snapshot is mapped to the physical display space of the holographic sand table to generate a mapped digital twin snapshot. Based on the mapped digital twin snapshot, multiple display terminals are synchronously driven to generate visualizations corresponding to the logical moment within the same display cycle.

2. The marine digital twin holographic sand table interaction method as described in claim 1, characterized in that, Extracting the state information corresponding to the logical moment from the state history information of the autonomous underwater vehicle includes: When the logical moment does not have a corresponding state frame in the state history information, the adjacent state frame of the logical moment is selected. Perform position interpolation operations on the position parameters in the adjacent state frames to generate interpolated position information corresponding to the logical time. Perform attitude interpolation operations on the attitude parameters in the adjacent state frames to generate interpolated attitude information corresponding to the logical time. The interpolated position information and the interpolated attitude information are combined to form the state information corresponding to the logical time.

3. The marine digital twin holographic sand table interaction method as described in claim 1, characterized in that, The acquisition of the detection information associated with the status information includes: Send a detection trigger command to the autonomous underwater vehicle and record the trigger time to determine the sensor pose of the autonomous underwater vehicle at the trigger time; Based on the sensor pose, the target detection space is determined, a detection signal is transmitted into the target detection space, the echo signal is received, and a detection result is generated. The detection results are associated with and stored in relation to the sensor pose, and the associated and stored detection results are used as the detection information associated with the state information.

4. The marine digital twin holographic sand table interaction method as described in claim 1, characterized in that, The method for establishing the preset spatial calibration relationship includes: Select at least three non-collinear physical control points in the physical display space of the holographic sandbox, and determine the virtual control points corresponding to the physical control points in the engine world scene; Based on the physical control points and the corresponding virtual control points, calculate the transformation matrix between the coordinate system of the engine world scene and the coordinate system of the physical display space; The reprojection error of the transformation matrix is ​​calculated based on the physical control points and the corresponding virtual control points. When the reprojection error is less than a preset threshold, the transformation matrix is ​​used as a preset spatial calibration relationship.

5. The interactive method for marine digital twin holographic sand table as described in claim 1, characterized in that, The process of mapping the geospatial information in the digital twin snapshot to the physical display space of the holographic sand table based on a preset spatial calibration relationship includes: Obtain longitude, latitude, and depth data from the geospatial information; Using a preset geographic reference origin as a benchmark, the longitude and latitude data are converted into horizontal positions in the coordinate system of the engine world scene, and the depth data is converted into vertical positions in the coordinate system of the engine world scene. Engine world coordinates are generated based on the horizontal and vertical positions. Based on a preset spatial calibration relationship, the engine world coordinates are mapped to the physical display space of the holographic sand table.

6. The marine digital twin holographic sand table interaction method as described in claim 1, characterized in that, The step of synchronously driving multiple display terminals to generate a visualization corresponding to the logical moment within the same display cycle based on the mapped digital twin snapshot includes: Extract the sector boundary and hit point cloud from the detection information in the mapped digital twin snapshot, and obtain the position data of the sector boundary and the hit point cloud in the physical display space of the holographic sand table; The pose data of the autonomous underwater vehicle in the physical display space is obtained in the mapped digital twin snapshot; Spatially align the position data of the sector boundary and the hit point cloud with the pose data of the autonomous underwater vehicle. In the physical display space, a visualization corresponding to the logical moment is generated based on the spatially aligned position data.

7. The interactive method for marine digital twin holographic sand table as described in claim 1, characterized in that, After acquiring the state data of the autonomous underwater vehicle and generating a logical time based on the timestamp of the state data, the method further includes: In response to the read operation of the status data, enter playback mode; In the playback mode, a playback control command is received, and the target playback time is determined according to the playback control command; The target playback time is taken as the logical time. The historical state information corresponding to the logical time is extracted from the state history information of the autonomous underwater vehicle, and the historical detection information associated with the historical state information is obtained. Based on the logical time point, the historical state information and the historical detection information are time-aligned to generate a historical digital twin snapshot.

8. A marine digital twin holographic sand table interactive device, characterized in that, include: The clock unification module is used to acquire the status data of the autonomous underwater vehicle and generate a logical time based on the timestamp of the status data. The snapshot generation module is used to extract the state information corresponding to the logical moment from the state history information of the autonomous underwater vehicle, obtain the detection information associated with the state information, and time-align the state information with the detection information based on the logical moment to generate a digital twin snapshot. The spatial mapping module is used to map the geospatial information in the digital twin snapshot to the physical display space of the holographic sand table based on a preset spatial calibration relationship, thereby generating a mapped digital twin snapshot. The synchronous display module is used to synchronously drive multiple display terminals to generate a visualization corresponding to the logical moment within the same display cycle, based on the mapped digital twin snapshot.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images