A task logic clock based autonomous underwater vehicle multi-modal synchronized playback method and system

Abstract

The application provides a task logic clock-based underwater vehicle multi-modal synchronous playback method and system. The method provided by the application comprises: encapsulating multi-modal heterogeneous data as uniform event records; establishing a segmented mapping relationship of device time to task logic time for each data source, and uniformly mapping each data source event record to a task logic time axis, and re-segmenting the mapping relationship when the clock is reset or restarted; evaluating the time confidence of each event record, determining a global water line by comprehensively considering the arrival progress and time confidence of each data source, and determining a task logic time interval not greater than the global water line as a solidified interval; generating a playback checkpoint at a key task node in the solidified interval; in response to a playback request, loading the playback checkpoint, starting from an incremental event cursor, and advancing event playback along the task logic time axis at a playback speed specified in the playback request in the solidified interval, and loading sensor data objects based on a sensor data object list.

Description

Technical Field

[0001] This application relates to the fields of marine information processing, spatiotemporal data synchronization and three-dimensional playback technology, and in particular to a method and system for multimodal synchronous playback of underwater vehicles based on a mission logic clock. Background Technology

[0002] During sea trials, the submersible carries various sensors and payloads, generating multimodal heterogeneous data, including navigation data, attitude data, sonar data, point cloud data, video footage, control commands, and alarm logs. After the sea trials, this data needs to be played back synchronously to fully reproduce the submersible's underwater mission execution process, verify the collaborative working status of various systems, diagnose abnormal events, and evaluate mission effectiveness. Different modal data record the same mission process from different perspectives; only by aligning and playing them back at the same time reference can the complete mission overview be reconstructed.

[0003] Currently, multimodal data playback typically relies on the timestamps recorded by each sensor device for time alignment, arranging data from different data sources chronologically to construct a playback timeline. Existing methods have three main shortcomings: First, the data output formats of various sensors and payload devices on the submersible are heterogeneous, lacking a unified data carrier; furthermore, the clocks of each device are independent, with different start times and frequency drift. During long-term operation, clock resets or restarts may occur, resulting in timestamps from different data sources not being on the same time base, making accurate cross-source time alignment and unified processing difficult. Second, sea trial data is asynchronously transmitted via different links, with varying arrival progress and transmission delays for each data source. The underwater environment may also cause a decrease in the timing accuracy and transmission quality of some data. Existing methods lack quantitative assessment tools for data arrival integrity and time reliability, making it difficult to determine which time intervals of data are complete enough for safe playback. Third, the sea trial data is large in volume and spans a long period of time. Existing methods lack efficient playback positioning and on-demand loading mechanisms, making it difficult to quickly jump to the target time. Furthermore, the full loading of large-volume sensor data such as sonar and point clouds during playback will lead to severe response delays and bandwidth pressure.

[0004] Therefore, there is an urgent need for a method to solve the problems of inconsistent multi-source clocks, difficulty in determining data integrity boundaries, low playback positioning efficiency, and large object loading delays. Summary of the Invention

[0005] In view of this, this application provides a method and system for multimodal synchronous playback of underwater vehicles based on mission logic clock, in order to solve the problems of inconsistent multi-source clocks, difficulty in determining data integrity boundaries, low playback positioning efficiency, and large object loading delay.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides a method for multimodal synchronous playback of a submersible based on a mission logic clock, the method comprising:

[0008] Multimodal heterogeneous data generated from multiple data sources during the sea trial are encapsulated into a unified event record, which includes at least a data source identifier, a device timestamp, and a data type identifier.

[0009] For each data source, a segmented mapping relationship from device time to task logic time is established. Event records from different data sources are uniformly mapped to a task logic time axis independent of any device clock. When a clock reset or restart of the target data source is detected, the mapping relationship of the target data source is re-segmented, and adjacent segments use their own independent mapping parameters.

[0010] Assess the time confidence of each event record, combine the arrival progress and time confidence of each data source to determine the global water level, and determine the task logic time interval that is not greater than the global water level as the solidified interval;

[0011] At the preset key task nodes in the solidified area, a replay checkpoint is generated. The replay checkpoint includes a three-dimensional situational state snapshot at the corresponding time, an incremental event cursor that identifies the next event to be replayed, and a list of sensor data objects.

[0012] In response to a playback request, the playback checkpoint is loaded, and starting from the incremental event cursor, the event playback is advanced along the task logic timeline within the solidified interval at the playback speed specified in the playback request, and sensor data objects are loaded as needed based on the sensor data object list.

[0013] The second aspect of this application provides a multimodal synchronous playback system for underwater vehicles based on a mission logic clock. The system includes an encapsulation module, a segmentation module, a determination module, a generation module, and a loading module.

[0014] The encapsulation module is used to encapsulate multimodal heterogeneous data generated by multiple data sources during the sea trial into a unified event record. The event record includes at least a data source identifier, a device timestamp, and a data type identifier.

[0015] The segmentation module is used to establish a segmented mapping relationship from device time to task logic time for each data source, and to uniformly map the event records of different data sources to the task logic time axis independent of any device clock. When the clock of the target data source is detected to be reset or restarted, the mapping relationship of the target data source is re-segmented, and adjacent segments use their own independent mapping parameters.

[0016] The determining module is used to evaluate the time confidence of each event record, determine the global water level by combining the arrival progress and time confidence of each data source, and determine the task logic time interval that is not greater than the global water level as the solidified interval.

[0017] The generation module is used to generate replay checkpoints at preset key task nodes in the solidified interval. The replay checkpoints include a three-dimensional situational snapshot at the corresponding time, an incremental event cursor that identifies the next event to be replayed, and a list of sensor data objects.

[0018] The loading module is used to load the playback checkpoint in response to the playback request, starting from the incremental event cursor, advance the event playback along the task logic time axis within the solidified interval at the playback speed specified in the playback request, and load sensor data objects as needed based on the sensor data object list.

[0019] The multimodal synchronous playback method and system for underwater vehicles based on mission logic clocks provided in this application firstly addresses the issues of different start times for various sensor clocks, crystal oscillator frequency drift, and potential resets or restarts during long-term operation. It establishes a mission logic timeline independent of any single device clock and employs segmented mapping. Upon detecting a clock reset or restart, the system automatically re-segments the data, with adjacent segments independently estimating mapping parameters. This ensures that the mapping accuracy within each segment is unaffected by clock anomalies, precisely aligning event records from different data sources corresponding to the same real mission moment to the same position on the timeline, providing a unified and reliable time reference for all subsequent steps. Secondly, addressing the issue of asynchronous data transmission from different links and inconsistent arrival times from various data sources, the system assesses the time confidence of each event record and, based on this, determines a global waterline by considering the arrival times of each data source. This global waterline serves as the boundary for defining solidified intervals, enabling the system to automatically determine which time intervals contain complete and reliable data suitable for safe playback, achieving a balance between data integrity and playback timeliness. Secondly, the playback checkpoints employ a triplet structure: a 3D situational snapshot for rapid scene reconstruction, an incremental event cursor to identify the starting point for incremental advancement, and a sensor data object list for loading large volumes of data such as sonar and point clouds on demand. Compared to traditional methods that only save snapshots or event locations, this approach simultaneously addresses both playback positioning efficiency and large-volume data management requirements. Finally, the different steps work together seamlessly. Unified encapsulation eliminates the need for subsequent steps to concern themselves with the original data format. Time alignment provides a unified time coordinate, boundary judgment ensures the integrity of playback data, and pre-set checkpoints provide an efficient positioning and activation mechanism. On-demand playback ultimately outputs a synchronized 3D situational view. Attached Figure Description

[0020] Figure 1A flowchart of the underwater multimodal synchronous playback method based on mission logic clock provided in Embodiment 1 of this application;

[0021] Figure 2 This is a schematic diagram of the structure of the underwater multimodal synchronous playback system based on the mission logic clock provided in Embodiment 2 of this application. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0023] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0025] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0026] Figure 1 The flowchart illustrates the underwater vehicle multimodal synchronous playback method based on a mission logic clock, provided in Embodiment 1 of this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0027] S101. Encapsulate the multimodal heterogeneous data generated by multiple data sources during the sea trial into a unified event record.

[0028] The event log includes at least a data source identifier, a device timestamp, and a data type identifier.

[0029] Specifically, sea trials refer to the testing process of an underwater vehicle performing pre-set tasks in a real marine environment, encompassing stages such as departure, descent, depth-holding cruise, mission operations, surfacing, and recovery. Sea trials are the final testing phase to verify the collaborative working capabilities of the various systems of the underwater vehicle in a real underwater environment, during which all sensors and payload equipment continuously operate and generate data.

[0030] Multimodal heterogeneous data refers to the various types of data generated by different sensors and payloads during sea trials. These data differ in format, sampling frequency, data volume, and physical meaning. For example, navigation data records latitude, longitude, and depth at low frequencies; attitude data records pitch, roll, and heading angles at medium frequencies; sonar data stores underwater acoustic echo images in large volumes; point cloud data stores laser-scanned 3D points in large volumes; video images record optical images in continuous streams; while control commands and alarm records are discrete events.

[0031] Event logs are standardized data units obtained by uniformly encapsulating the aforementioned heterogeneous data. Each event log contains three basic fields: a data source identifier, used to distinguish which sensor or device it comes from; a device timestamp, recording the time the data was generated under the device's local clock; and a data type identifier, used to distinguish different categories such as navigation, attitude, sonar, and commands. After encapsulation, data from all data sources exists in a unified event log format. Subsequent steps such as segmentation mapping, confidence assessment, waterline determination, and playback progression all use event logs as the basic operation unit, no longer focusing on the heterogeneous format of the original data.

[0032] In practice, during sea trials, various sensors and payload devices continuously generate data, which is first collected through the data access layer. For each data source, the system assigns a unique data source identifier for subsequent data source differentiation. When a data item arrives, the system extracts the device's local timestamp from the original data message, i.e., the time the data was generated under the sensor's own clock. Simultaneously, it determines the data type identifier based on the data source; for example, data from the inertial navigation unit is labeled as navigation type, data from sonar as sonar type, and events from the control system as command type. Subsequently, the system combines the data source identifier, device timestamp, and data type identifier to construct a unified event record. For large-volume sensor data in the original data, such as sonar images or point cloud data, the event record only carries a reference to that data; the original data itself is stored independently and loaded on demand during playback using the reference in the event record. Data from all data sources is converted into event records one by one in this manner, forming a unified event record stream.

[0033] For example, in one embodiment, an event log can be represented as:

[0034] E =<C, S, P, M, O, Y, R, n, td, tr, tl, q, v, h, z> ;

[0035] Where E represents event log; C represents business domain; S represents stable scenario; P represents project; M represents task; O represents entity; Y represents data type; R represents data source; n represents source sequence number; td represents device time; tr represents reception time; tl represents task logic time; q represents time confidence level; v represents data version; h represents payload summary; and z represents tracking identifier.

[0036] S102. Establish a segmented mapping relationship from device time to task logic time for each data source, and map the event records of different data sources to the task logic time axis independent of any device clock. When the clock of the target data source is detected to be reset or restarted, the mapping relationship of the target data source is re-segmented, and adjacent segments use their own independent mapping parameters.

[0037] Specifically, device time refers to the time recorded by the internal clock of each sensor or load device. The device times of different devices are independent of each other, each with a different start time, and the crystal oscillator frequencies of the internal clocks of each device are different. After long-term operation, the device times of each device will accumulate and drift.

[0038] Task logical time refers to a logical time scale independent of any specific device clock, used to describe the progress of a sea trial mission. It does not rely on the clock of any single sensor or device, but rather provides a unified time measurement of the entire mission's progress from start to finish. Task logical time starts at the mission initiation time and increases linearly according to the actual sequence of mission execution; each real moment in the mission corresponds to a unique task logical time. The task logical time axis is a one-dimensional time axis scaled by task logical time, serving as a target coordinate system for the unified alignment of event records from all data sources after segmentation and mapping. On the task logical time axis, each position corresponds to a specific moment in the mission process. Event records from different data sources corresponding to the same real moment are mapped to the same position on the axis, thereby achieving synchronous orchestration of multi-source data.

[0039] Segmented mapping refers to the conversion rules for transforming device time from a data source to task logical time. This relationship is divided into multiple mapping segments by the device's clock reset or restart event. Within each mapping segment, the conversion from device time to task logical time uses an independent set of mapping parameters. When a device clock reset or restarts, the mapping relationship automatically terminates the current segment and starts a new segment, which uses re-estimated mapping parameters. Adjacent segments are independent and do not interfere with each other.

[0040] It should be noted that due to the different starting points of device times and frequency drift among various devices, as well as the possibility of clock resets or restarts during long-term operation, directly using device time for cross-source alignment will result in deviations or even jumps. The purpose of establishing segmented mapping relationships is to transform event records from their respective independent device time coordinate systems to the same task logical time coordinate system. This ensures that event records corresponding to the same real time in a task from different data sources have the same task logical time, thereby achieving accurate cross-source time alignment. The segmented design also ensures that the mapping relationship can automatically adapt when anomalies such as device clock resets or restarts occur, preventing the incorrect application of mapping parameters from before the anomaly to the time period after the anomaly.

[0041] In specific implementation, multiple alignment anchor points are selected, each corresponding to a reference event with a known task logic time and the device timestamp record of each data source when the reference event occurs. For each mapping segment, robust weighted least squares estimation is performed using the alignment anchor points contained in the mapping segment to determine the mapping parameters of the mapping segment. The weight of each alignment anchor point in the least squares estimation decreases as the time distance between the alignment anchor point and the midpoint of the mapping segment increases. Based on the mapping parameters, a linear mapping relationship from device time to task logic time is constructed within the mapping segment, and each data source aligns the event records to the task logic time axis through its respective linear mapping relationship.

[0042] Specifically, alignment anchors are calibration points in the course of a mission where the correspondence between the mission's logical moment and the timestamps of each device can be clearly determined. Each alignment anchor records the corresponding values ​​of the same real event in two time coordinate systems, including the mission's logical moment and the device timestamps recorded by each data source at the instant the event occurs. For example, when a submarine performs a standard maneuver in a mission, the mission's logical moment is known, and each sensor records its own device timestamp at the moment the maneuver occurs. These two sets of data together constitute an alignment anchor.

[0043] A reference event is a deterministic event in the task process that has a known logical time of occurrence and can be simultaneously perceived by multiple data sources. The selection of a reference event requires that its occurrence time be definite on the task's logical timeline and that a response signal for the event can be identified from records of multiple data sources.

[0044] A mapped segment refers to a continuous time period on the device timeline of a data source, divided by a clock reset or restart event. The time period between two clock resets or restarts constitutes a mapped segment. Different mapped segments of the same device use their own independent mapping parameters because the starting point and frequency of the device time may change after a clock reset, and the mapping parameters before the reset are no longer applicable to the time period after the reset.

[0045] Mapping parameters refer to a set of values ​​describing the linear conversion relationship between device time and task logic time within a certain mapping segment. Since the mapping from device time to task logic time uses a linear model, the mapping parameters include two quantities: slope and intercept. The slope and intercept are obtained through robust weighted least squares estimation of the alignment anchors contained within the mapping segment. During the estimation process, the weight of each alignment anchor decreases as its time distance from the midpoint of the segment increases. This ensures that anchors closer to the segment center contribute more to the parameter estimation, while anchors farther from the segment center contribute less, thus guaranteeing the highest accuracy of the mapping in the segment center region.

[0046] In practical implementation, firstly, multiple alignment anchor points are selected within the task process. When selecting these anchor points, a reference event is determined, the time of which is known on the task's logical timeline, denoted as the task's logical moment. Simultaneously, the timestamps recorded by each device at the instant the reference event occurs are retrieved from the records of each data source, denoted as the device timestamps of each data source at that anchor point. Thus, each alignment anchor point provides a known correspondence between each data source and its corresponding device timestamp and task logical moment. Secondly, for each data source, its clock status is monitored. When a clock reset or restart is detected, a segment boundary is defined at the moment of reset or restart. The device time interval between two adjacent clock resets or restarts constitutes a mapping segment, and each mapping segment has a continuous range of device time without clock transitions. Then, for each mapping segment, all alignment anchor points contained within that segment are collected. Using the midpoint of the segment interval as a reference position, the time distance between each alignment anchor point and the midpoint is calculated. Based on this, a weight is assigned to each anchor point: anchor points closer to the midpoint have a higher weight, and anchor points farther away have a lower weight. Using these weighted anchor data, robust weighted least squares estimation is performed to solve for a set of mapping parameters for the mapping segment, including the slope and intercept. Based on the obtained mapping parameters, a linear conversion relationship from device time to task logic time is constructed within the mapping segment. For any device timestamp within the segment, the device time is multiplied by the slope and then added to the intercept to obtain the corresponding task logic time. Finally, for each event record from each data source, its device timestamp is extracted, the mapping segment to which the device timestamp belongs is determined, and the mapping parameters corresponding to that segment are used to convert it into task logic time. After the event records from all data sources are converted according to their respective mapping relationships, they are uniformly aligned to the same task logic timeline.

[0047] It should be noted that the above-mentioned clock reset or restart detection is not limited to the reset signal actively reported by the device, but also includes the following abnormal situations that can be indirectly identified through data characteristics: the device timestamp of the current event record is less than the device timestamp of the previous event record from the same data source, i.e., time rollback has occurred; the device's startup count or running cycle count is inconsistent with the previously recorded value; the residual of multiple consecutive event records after conversion by the current mapping parameters exceeds a preset threshold; the clock frequency correction coefficient derived using anchor points experiences a jump exceeding a preset range. When any of the above situations occur, the current mapping segment is determined to end, and a new mapping segment is created.

[0048] In the above segmented mapping relationship, the mapping from device time td to task logic time tl can be represented by the linear model tl = a(s,k)·td + b(s,k) + d(i), where a(s,k) is the clock frequency correction coefficient of the k-th segment of the s-th data source, b(s,k) is the clock offset, and d(i) is a record-level compensation term used to correct network transmission delay or differences in sampling positions within a file. If the data source has a reliable unified time synchronization source, a(s,k) takes a value close to 1, and b(s,k) is directly determined by the time synchronization deviation; if there is no reliable time synchronization, both a(s,k) and b(s,k) are obtained by robust weighted least squares estimation through the alignment anchor points within the segment. The robust loss function used for estimation is used to suppress the influence of abnormal anchor points. When the number of alignment anchors within a segment is insufficient to simultaneously estimate a(s,k) and b(s,k), the frequency drift parameters are not forcibly fitted; instead, the clock offset b(s,k) is estimated using only the existing anchors, while simultaneously reducing the time confidence of event records within that segment. The set of alignment anchors is denoted as A={(tdj,trefj,wj)}, where trefj is the reference logic time, wj is the anchor weight, and tdj is the device timestamp. Robust weighted least squares estimation can be expressed as min Σ wj×ρ [a×tdj+b-trefj], where ρ is the robust loss function that suppresses the influence of abnormal anchors.

[0049] Furthermore, for sensor data that does not directly provide device timestamps for each record, its device time must be constructed before encapsulating event records. For sonar data that only provides the file start time and a fixed sampling frequency, the device time of the i-th sampling point is obtained by adding the cumulative sampling interval of the previous i samples to the file start time; if the data marks the location of lost samples, the corresponding lost sample compensation is added to the cumulative interval so that the device time of subsequent sampling points is not affected. For video data, a time index is established based on the display timestamps of keyframes and event records are encapsulated accordingly, without requiring each frame to enter the event queue.

[0050] The method provided in this embodiment fundamentally solves the core problem of inaccurate alignment of multi-source heterogeneous data due to inconsistent device clocks by establishing segmented mapping relationships from device time to task logic time for each data source and uniformly mapping all event records to a task logic time axis independent of any device clock. Since the device clocks of different sensors have different start times and frequency drift, if cross-source time alignment is directly performed using device timestamps, event records corresponding to the same real time in different data sources will be misaligned to different time positions, leading to a disconnect between navigation data and sonar images during playback, and disordered timing of attitude changes and command execution. By introducing a task logic time axis, all event records are uniformly converted to the same time coordinate system independent of any device clock. Event records from different data sources corresponding to the same real time in the task are mapped to the same position on the axis, achieving accurate cross-source synchronization. The segmented mapping design further ensures that when anomalies such as clock resets or restarts occur, the mapping relationship will not incorrectly apply parameters from before the anomaly to the time period after the anomaly. Each time an anomaly is detected, the current segment automatically ends and a new segment begins. Adjacent segments independently estimate their mapping parameters, ensuring that the mapping accuracy within each segment is unaffected by clock anomalies in adjacent segments. Robust weighted least squares estimation of aligned anchor points primarily uses anchor points near the midpoint of the segment interval, supplemented by anchor points far from the midpoint, ensuring the highest mapping accuracy in the segment's central region. Simultaneously, a robust loss function suppresses the interference of individual abnormal anchor points on the overall estimation. When anchor points are insufficient, a degradation strategy of estimating only the offset is employed to avoid forcibly fitting frequency drift and resulting in unreliable results due to insufficient information. Furthermore, for sensors such as sonar and video that do not directly provide per-record device timestamps, their device times are constructed before encapsulating event records, allowing these special data sources to also be incorporated into the unified segmented mapping framework. After this step, the event records from all data sources are uniformly arranged on the same task logic timeline, providing a unified and reliable time reference for subsequent time confidence assessment, global water level determination, playback checkpoint generation, and on-demand playback.

[0051] S103. Evaluate the time confidence of each event record, determine the global water level by combining the arrival progress and time confidence of each data source, and determine the task logic time interval that is not greater than the global water level as the solidified interval.

[0052] Specifically, time confidence is a quantitative evaluation value of the reliability of the time information of each event record. It comprehensively reflects multiple factors, including the reliability of the time synchronization of the data source corresponding to the event record, the size of the residual after segmented mapping of the event record, the stability of the data transmission link, whether there are jumps or missing events in the event record sequence, and whether the device that generated the data is in normal working condition. The higher the confidence, the more reliable the time positioning of the event record, and the more suitable it is for subsequent synchronization and orchestration; the lower the confidence, the more likely its time information is inaccurate, requiring careful handling.

[0053] The arrival progress refers to the maximum task logic time that each data source has continuously reached on the task logic timeline. Because data from different data sources is asynchronously transmitted through their own independent links, the arrival progress of each data source is different. Some data sources transmit data faster, and their arrival progress has advanced to a later task logic time; while other data sources have relatively lagging arrival progress due to link delays or data processing time.

[0054] The global watermark is a unified judgment of the point in time when data from all multi-source datasets is complete. The global watermark means that before the logical time of the task, data from all data sources is either in place or only slightly delayed within a reasonable range, and the data can be considered complete.

[0055] The solidified interval refers to the time interval on the task's logical timeline that is no greater than the global watermark. Within this interval, all event records from all data sources have arrived and completed confidence assessments, the data state is stable, and no new late data will change the content of this interval. Data within the solidified interval can be safely used for playback, and subsequent playback checkpoint generation and playback progression are all based on the solidified interval. The time interval after the global watermark, however, contains data from some data sources that has not yet arrived or may still contain late data; this is the unsolidified interval and is not included in the current playback scope.

[0056] In specific implementation, the evaluation of the time confidence of each event record includes: obtaining the timing reliability factor, the anchor fitting residual factor of the segmented mapping, the network stability factor of the data transmission link, the time continuity factor of the event record sequence, and the quality status factor of the data source device itself from the data source corresponding to each event record; assigning corresponding preset weights to each factor and performing weighted calculations to obtain the time confidence of each event record. The time confidence increases with the increase of the timing reliability factor, network stability factor, time continuity factor, and quality status factor, and decreases with the increase of the anchor fitting residual factor; comparing the time confidence with a preset first threshold and a second threshold, event records not lower than the first threshold directly participate in the synchronization orchestration, event records between the first threshold and the second threshold participate in the synchronization orchestration after interpolation compensation, and event records lower than the second threshold are marked as missing; wherein, the first threshold is higher than the second threshold.

[0057] Specifically, for each event record mapped to the task's logical timeline, five factors affecting its time reliability are first obtained. First, the timing reliability factor of the data source corresponding to the event record is obtained. If the data source has GPS or other unified timing capabilities, the timing reliability factor is set to a higher value, such as 1.0; if the data source only relies on internal crystal oscillator timing, the timing reliability factor is set to a lower value, such as 0.3. If the timing signal of the data source changes from available to unavailable during the task, the timing reliability factor is dynamically adjusted downwards. The rules for setting the timing reliability factor are determined when the data source is connected. Second, the anchor point fitting residual factor of the segmented mapping is obtained. The fitting residuals of each aligned anchor point in the parameter estimation of the mapping segment to which the event record belongs are extracted, and the root mean square value of the residuals is calculated. When the root mean square value of the residuals is less than the first residual threshold, the anchor point fitting residual factor takes a higher value, such as 1.0, indicating good segmented mapping accuracy. When the root mean square value of the residuals is greater than the second residual threshold, the anchor point fitting residual factor takes a lower value, such as 0.2, indicating that there may be a large deviation in the segmented mapping. When it is between the two, it is taken by linear interpolation. The value of the anchor point fitting residual factor decreases as the residual increases. Third, obtain the network stability factor of the data transmission link. Statistically analyze the network jitter amplitude and packet loss rate of the data source in the past preset time window. When both are lower than their respective stability thresholds, the network stability factor takes a higher value; when either indicator exceeds the corresponding instability threshold, the network stability factor takes a lower value; when it is between the two, it is taken proportionally. Fourth, obtain the time continuity factor of the event record sequence. Check the task logic time interval between the event record and its adjacent event records. If the intervals are uniform, without abnormal jumps and without missing records, the time continuity factor takes a higher value; if there are abrupt interval changes or missing records, the time continuity factor is reduced according to the missing proportion. Fifth, obtain the quality status factor of the data source device itself. The device's operating status is determined based on the self-test status code, operating temperature, and power supply voltage reported by the device. When the device self-test is normal, the operating temperature is within the rated range, and the power supply voltage is stable, the quality status factor is set to a higher value; when the device reports an abnormal alarm, the quality status factor is lowered according to the alarm level. Among the above five factors, the timing reliability factor and the anchor point fitting residual factor directly affect the accuracy of the time mapping of event records and are considered direct influencing factors; the network stability factor, time continuity factor, and quality status factor indirectly affect the reliability of event records and are considered indirect influencing factors. The weights of each factor are preset in the following way: On the validation dataset, using the actual time deviation of each event record as a reference, a grid search is performed on the five sets of weights, and the set of weights that maximizes the correlation between the confidence calculation result and the actual time deviation is selected as the preset weight. The basic principle for setting the weights is that the weight of the direct influencing factor is higher than the weight of the indirect influencing factor, and the sum of the five weights is 1. The factors are weighted according to their respective weights to obtain the time confidence of the event record.After weighted calculation, higher values ​​for the timing reliability factor, network stability factor, time continuity factor, and quality status factor result in greater time confidence; similarly, higher anchor point fitting residual factors also contribute to greater time confidence. Once the time confidence is obtained, it is compared with a preset first threshold and a second threshold. Event records with a time confidence not lower than the first threshold are considered highly reliable and are directly included in the synchronization orchestration. Event records with a time confidence between the first and second thresholds have generally usable time information but some uncertainty; interpolation compensation is performed using adjacent high-confidence event records, and the interpolated result replaces the original time information before being included in the synchronization orchestration. Event records with a time confidence lower than the second threshold have unreliable time information; these records are marked as missing and not included in the current synchronization orchestration, pending further processing when more reliable data arrives.

[0058] For example, time confidence can be expressed as:

[0059] q = q1^w1 × q2^w2 × q3^w3 × q4^w4 × q5^w5;

[0060] Where q is the time confidence level; q1 is the timing confidence factor; q2 is the anchor fitting residual factor; q3 is the network stability factor; q4 is the time continuity factor; q5 is the quality status factor; and w1, w2, w3, w4, and w5 are the corresponding weights.

[0061] q2 = exp[-RMSE 2 / τ 2 ];

[0062] Where q2 is the anchor point fitting residual factor; RMSE is the root mean square of the anchor point residuals within the current mapping window; and τ is the acceptable residual scale.

[0063] Furthermore, determining the global waterline by comprehensively considering the arrival progress and time confidence of each data source includes: obtaining the maximum task logic time of consecutively arriving event records from each data source on the task logic timeline; subtracting a preset allowable lateness tolerance from the maximum task logic time to obtain the local waterline of the data source; and taking the minimum value of the local waterline among all data sources as the global waterline; marking event records whose task logic time is later than the global waterline as late data; if the impact of the late data does not involve the published replay content, incorporating the late data into the current replay version; if the impact of the late data involves the published replay content, deriving a new version based on the current replay version and incorporating it, while simultaneously registering the arrival event of the late data in the bypass audit record.

[0064] Specifically, firstly, for each data source, the system maintains its arrival progress on the task logic timeline. It iterates through all event records for which time mapping has been completed for that data source, and considers event records marked as missing due to time confidence levels below the second threshold as data gaps. In the remaining event records, it finds the largest interval where the task logic time increases continuously without jumps, and takes the largest task logic time in that interval as the largest continuously arrived task logic time for that data source.

[0065] It should be noted that the arrival progress of each data source is predicated on the continuous arrival of event records on the task's logical timeline, and the determination of continuous arrival is also constrained by time confidence. Event records marked as missing due to time confidence below the second threshold, even though they have arrived at the storage level, are considered data gaps in the continuity determination and are not included in the range of continuously arriving event records. The arrival progress of this data source remains before the gap. Thus, time confidence participates in the determination of the global watershed by constraining the progress of each data source's arrival: when a data source continuously produces low-confidence records near a certain task's logical moment, the local watershed progress of that data source is slowed down, the global watershed is conservatively maintained, and the boundary of the solidified interval is correspondingly postponed, thereby ensuring that event records entering the solidified interval not only arrive completely but also have reliable time information.

[0066] The progress of data source arrival is determined by the continuity and absence of missing task logic moments in the event records. If an event record is missing at a certain task logic moment, the progress stops before that missing position. Secondly, for each data source, a preset allowable delay tolerance is subtracted from its longest continuously arriving task logic moment to obtain the local watermark for that data source. The allowable delay tolerance is a pre-set time length to allow for normal transmission delay differences between data sources. For example, if the allowable delay tolerance is set to 3 seconds and the longest continuously arriving task logic moment for a data source is 500 seconds, then the local watermark for that data source is 497 seconds. Then, the local watermarks of all data sources are iterated, and the minimum value is taken as the global watermark. Next, intervals on the task logic timeline that are not greater than the global watermark are defined as solidified intervals. All event records within solidified intervals are stable and can be used for playback. For event records whose task logic moments arrive later than the global watermark, the system marks them as late data. After marking late data, the system processes it differently based on the impact range of the late data. If the late data only involves unpublished replay content (i.e., its task logic time is later than the maximum task logic time currently being replayed or published to the client), the late data does not change any published replay results. The system directly incorporates the late data into the current replay version, and the replay version number remains unchanged. If the late data involves published replay content (i.e., its task logic time is earlier than or equal to the maximum task logic time published to the client), it indicates that the data used in the previous replay was incomplete. In this case, the system increments the current replay version number to obtain a new version number, creates a derived version, and incorporates the late data into this derived version. Simultaneously, the arrival event of this late data, including the data source identifier, task logic time, arrival time at the wall clock, and the affected replay version number, is written into a bypass audit record for later traceability and analysis. The replay service continues to provide services using the current version number, and analysts can use the audit record to know that a particular version has undergone late data correction.

[0067] For example, in one embodiment, the global water level can be represented as:

[0068] Wg = min[W(s)], where s belongs to the set of required data sources;

[0069] W(s) = Tmax(s) - L(s);

[0070] Where Wg is the global water level; W(s) is the local water level; Tmax(s) is the maximum task logic time; and L(s) is the allowable lateness tolerance.

[0071] S104. Generate a replay checkpoint at the preset key task node in the solidified area.

[0072] The replay checkpoint includes a snapshot of the three-dimensional situational state at the corresponding moment, an incremental event cursor that identifies the next event to be replayed, and a list of sensor data objects.

[0073] Specifically, replay checkpoints are pre-generated state archives at preset key task nodes within a fixed timeframe. They are used to quickly locate and restore to a specific task moment during replay, avoiding the need to reload all data from the beginning each time. A replay checkpoint is like a snapshot bookmark on the task's logical timeline, recording the complete replay state at that moment. When analysts jump to any point where a checkpoint has been generated, they only need to load that checkpoint to immediately begin replaying, without having to traverse from the beginning.

[0074] Optionally, the three-dimensional situational state snapshot records the three-dimensional spatial position, attitude, heading, and working status of each sensor of the underwater vehicle at the corresponding mission logic moment; the incremental event cursor records the position identifier of the first event to be replayed after the replay checkpoint in the event stream; the sensor data object list includes the temporal envelope range, spatial envelope range, content summary, and block storage index of each data source sensor data object, and sensor data object blocks with the same content summary are only retained once in the storage system.

[0075] Furthermore, the replay checkpoint contains three types of information. A 3D situational awareness snapshot records the complete state of the submersible in 3D space at the corresponding mission logic moment, including the submersible's spatial position, attitude angles, heading, velocity, and the operational status of each sensor and payload device at that moment. After loading this snapshot during replay, the client's 3D situational awareness view can directly switch to the display content at that moment. An incremental event cursor records the position identifier of the first event to be replayed after this checkpoint in the event stream. It is a pointer to a specific position in the event stream. When replay starts, it sequentially reads and pushes data starting from the event pointed to by this cursor, without repeating the processing of events already fixed before the cursor, thus achieving incremental progress. A sensor data object list records the index information of large-volume sensor data objects from various data sources, including the temporal envelope range, spatial envelope range, content summary, and block storage index of each object. During replay, these large objects are not loaded immediately, but are retrieved as needed according to the index in the list based on the needs of the replay window, avoiding the full loading of large-volume data such as sonar and point clouds at the start of replay.

[0076] In practice, the first step is to determine the preset key task nodes. During the sea trial, the mission is divided into several phases, each with a clear transition node, such as the end of departure and entry into diving, the end of diving and entry into depth-holding cruise, the start of mission operations, and the end of depth-holding cruise and entry into ascent. These phase transition nodes are the preset key task nodes. As sea trial data continues to arrive and the solidified intervals are continuously advanced, when the mission logic moment corresponding to a key task node falls into the solidified interval, a replay checkpoint is triggered at that node. Secondly, a three-dimensional situational snapshot is generated. At the mission logic moment corresponding to the key task node, all solidified event records before that moment are collected. Based on this, the current three-dimensional spatial position, pitch angle, roll angle, heading angle, velocity vector, and the working mode, activation status, and current parameter configuration of each sensor are calculated. This state information is packaged into a state snapshot, which serves as the situational baseline for that checkpoint. Then, an incremental event cursor is generated. The first event record immediately adjacent to the current mission logic moment in the event stream is located, and its position identifier in the event stream is recorded as the incremental event cursor. When subsequent playback begins, event playback reads sequentially starting from the cursor position; events already fixed and located before this cursor position are not pushed again. Next, a list of sensor data objects is generated. Information on all large-volume sensor data objects near the current moment is collected by traversing various data sources. For each sensor data object, its temporal envelope range (the start and end interval of the task logic time covered by the object) and spatial envelope range (the three-dimensional spatial region covered by the object) are recorded. A content summary of the object's data is calculated, which can be generated using a hash algorithm. The block storage index of the object in the storage system is recorded, indicating the location of each data block in the storage system. Sensor data object blocks with the same content summary are retained only once in the storage system, and multiple entries in the list can point to the same stored data. The aforementioned three-dimensional situational snapshot, incremental event cursor, and sensor data object list are combined into a playback checkpoint and stored in association with the corresponding task logic time.

[0077] For example, in one embodiment, a playback checkpoint can be represented as:

[0078] CP(k) =<tk, Vk, Sk, Ck, Mk, Hk> ;

[0079] Wherein, CP(k) is the replay checkpoint; tk is the checkpoint task logic time; Vk is the data version set; Sk is the 3D situation status snapshot; Ck is the incremental event fee cursor; Mk is the sensor data object list; and Hk is the checkpoint content summary.

[0080] S105. In response to the playback request, load the playback checkpoint, starting from the incremental event cursor, advance the event playback along the task logic time axis within the solidified interval at the playback speed specified in the playback request, and load sensor data objects as needed based on the sensor data object list.

[0081] Specifically, playback speed refers to the rate at which the task logic time progresses relative to the actual wall clock time during playback. When the playback speed is 1x, the task logic time and the wall clock time advance synchronously; that is, for every second of wall clock time that passes, the task logic time also advances by 1 second, simulating the rhythm of the actual task. Changes in playback speed do not alter the logical order between the fixed event records; they only affect the rhythm of the wall clock pushing events to the client.

[0082] Sensor data objects refer to large-volume raw sensor data generated during sea trials, including underwater acoustic echo image data acquired by sonar, 3D point cloud data from laser scanning, and video image data. Unlike lightweight event logs such as navigation attitude and control commands, a single data object can reach tens or even hundreds of megabytes in size. These large-volume sensor data are not directly embedded in the event logs themselves, but are stored independently in the storage system, with only a reference to the data retained in the event logs. During playback, the system loads these large objects only when they are relevant to the current playback window, based on the list of sensor data objects in the playback checkpoints, rather than preloading them all at the start of playback.

[0083] In specific implementation, starting from the position of the incremental event cursor, event records are read in the order of task logic time and advanced along the task logic time axis. During the advancement, differentiated synchronization window strategies are adopted for different types of event records to form a unified playback event stream. Specifically, for navigation data and attitude data, a preset duration is used as the synchronization window, and the data within the synchronization window is interpolated to generate continuous trajectories and attitude sequences. For command and alarm data, the playback event stream is directly inserted according to the original event record time. For sonar and point cloud data, the time of the target object segment is... The envelope is intersected with the current track window, and only target object fragments that overlap with the track window in time are loaded; when downsampling the target data, the target sampling points are retained; the target sampling points include at least event proximity points, direction change points, extreme points, and quality change points; the task logic timeline is advanced at the playback speed, and the current task logic time is determined based on the task logic time of the playback checkpoint, the playback speed, and the elapsed wall clock time; events in the playback event stream whose task logic time is not later than the current task logic time are pushed to the client to drive the 3D situation update.

[0084] Specifically, firstly, in response to a playback request, the system loads the playback checkpoint corresponding to the target task's logical time specified in the playback request. After loading the playback checkpoint, starting from the position pointed to by the incremental event cursor in the playback checkpoint, subsequent event records are read sequentially according to the task's logical time order, progressing forward along the task's logical time axis. During this process, the system uses differentiated synchronization window strategies to process different types of event records. For navigation and attitude data, a preset duration is used as the synchronization window. Multiple navigation or attitude event records falling within the current synchronization window are treated as a processing unit. Interpolation calculations are performed on the data within this unit to generate continuous trajectory and attitude sequences, ensuring a smooth transition of the submersible's position and attitude during playback, without jumps due to the original sampling interval. For command and alarm data, no downsampling or interpolation is performed; each event record is directly inserted into the playback event stream according to its original task logical time, ensuring that the precise timing of discrete events is completely preserved during playback. For sonar and point cloud data, the system maintains the current playback track window, which is a time period and spatial range near the submersible's current location. The system iterates through the temporal envelope range of each entry in the sensor data object list, intersecting the temporal envelope of each entry with the current track window. Loading is only performed on entries with temporal overlap; sonar or point cloud data without temporal overlap with the current track window are not loaded. During this processing, when downsampling is required, the system retains the following four types of target sampling points from the original sampling points: event proximity points (sampling points near the occurrence of discrete events, used to preserve data characteristics at the time of the event); direction change points (sampling points where the direction of motion changes significantly); extreme points (sampling points where the value reaches a local maximum or minimum); and quality change points (sampling points where the data quality indicators change significantly). Sampling points other than these four types are discarded according to downsampling rules to reduce the amount of data in the replay event stream. During playback, the system advances the task logical timeline at the playback speed specified in the playback request. The current task logic time is determined by three factors: the task logic time corresponding to the replay checkpoint, the replay speed, and the elapsed wall clock time since the replay started. Multiplying the elapsed wall clock time by the replay speed and adding the task logic time of the replay checkpoint yields the current task logic time. Finally, the system pushes events in the replay event stream whose task logic time is no later than the current task logic time to the client. Upon receiving an event, the client updates the 3D situational awareness display, including the submersible's position, attitude, sensor status, and the visualization of loaded sonar and point cloud data. As the wall clock time continues to elapse and the current task logic time advances, more events are pushed to the client, achieving continuous replay until the boundary of the fixed interval is reached or a stop playback command is received.

[0085] For example, in one embodiment, the current task logic moment can be represented as:

[0086] tl(now) = tl(start) + r × [tw(now) - tw(start)];

[0087] Where tl(now) is the current task logic time; tl(start) is the task logic time of the replay checkpoint; r is the replay speed; tw(now) is the current wall clock time; tw(start) is the starting wall clock time; and tw(now) - tw(start)] is the elapsed wall clock time.

[0088] Furthermore, based on the temporal and spatial envelopes of each entry in the sensor data object list, it is determined whether the current playback window needs to load the corresponding sensor data object. If loading is required, an access credential is generated based on the block index of the corresponding entry. The client obtains the data block through the access credential, and sensor data object blocks with the same content summary are directly reused in the cache.

[0089] Specifically, the system continuously maintains the current playback window during playback. The current playback window is determined by the current task logic time, playback speed, and the underwater vehicle's current spatial position, encompassing both a time range and a spatial range. For each entry in the sensor data object list, the system reads the time envelope and spatial envelope recorded for that entry. The time envelope describes the start and end time interval of the task logic covered by the sensor data object, and the spatial envelope describes the three-dimensional spatial region covered by the object. The system intersects the entry's time envelope with the time range of the current playback window, and simultaneously intersects the entry's spatial envelope with the spatial range of the current playback window. Only when both intersections are not empty is the system determined that the current playback window needs to load the sensor data object; if either intersection is empty, the entry is skipped, and loading is not performed. For sensor data object entries determined to need loading, the system determines the data block location of the object on each storage node based on the block storage index recorded in the entry. The block storage index records the storage node address and block identifier of each data block. The system generates an access credential for each data block that needs to be acquired. This credential includes the target storage node address, block identifier, authorization validity period, and allowed operation types. After generation, the access credential is sent to the client. Upon receiving the credential, the client initiates a data block acquisition request to the corresponding storage node. The storage node verifies the credential's validity and returns the requested data block. Once the client acquires the data block, it checks its local cache for a content digest. The content digest is obtained by calculating a hash value from the data block's content. If a data block with the same content digest already exists in the local cache, the cached copy is reused without further requesting from the storage node. If no match is found in the local cache, the newly acquired data block is written to the local cache for reuse in subsequent playback windows when encountering blocks with the same content digest. Through this process, sensor data object blocks are loaded only as needed within the actual spatiotemporal scope of the playback window, and blocks with the same content are transmitted only once between the client and the storage node, subsequently reused through caching.

[0090] Optionally, the method further includes: in response to a progress jump request, constructing a new three-dimensional situational state snapshot without changing the current display on the client, loading the replay checkpoint corresponding to the logical moment of the jump target task, verifying the content summary of the replay checkpoint against the summary of the client's current situation, and after passing the verification, switching the client's current situation to the situation corresponding to the replay checkpoint by atomic replacement, and incrementing the current generation number; in response to a disconnection reconnection request, obtaining the client's current generation number, verifying whether the replay version corresponding to the current generation number is still available, and if available, restoring the replay session based on the latest replay checkpoint of the replay version.

[0091] Specifically, when the system receives a progress jump request, it first determines the target task logical moment specified in the jump request and finds the replay checkpoint corresponding to that target task logical moment. During the jump preparation period, the client's current 3D situation view remains unchanged, and no intermediate state rendering switch is performed. The system builds a new 3D situation state snapshot in the background. The 3D situation state snapshot is read from the replay checkpoint corresponding to the target task logical moment and assembled into a complete situation state object in memory, including the submarine's spatial position, attitude angles, heading, velocity vector, and the working status of each sensor. After the snapshot is assembled, the system calculates a content summary of the 3D situation state snapshot of the replay checkpoint and simultaneously obtains a content summary of the 3D situation currently displayed on the client. The two are compared and verified to confirm whether the difference between them meets expectations, that is, the difference only stems from the difference in the task logical moment before and after the jump, and there is no data corruption or state inconsistency. After the verification is passed, the system performs an atomic replacement operation. The client's situation state is switched from the current state to the situation state corresponding to the target replay checkpoint in one go. There is no intermediate state during the switch, and the client screen jumps directly from the scene before the jump to the scene corresponding to the target task logical moment. Simultaneously, the system reads the current generation number and increments it by 1, using it as the new current generation number. The generation number identifies the current replay version the client is in, and it changes with each jump. After the atomic replacement is complete, the system continues the event replay from the incremental event cursor position of the target replay checkpoint, according to the replay speed specified in the replay request. When the connection between the client and the system is interrupted and then re-established, the client sends a reconnection request to the system, carrying the current generation number stored locally on the client. Upon receiving the reconnection request, the system retrieves the generation number and searches for the corresponding replay version in the maintained list of replay versions. If the replay version corresponding to that generation number still exists in the system and has not been reclaimed, then the replay version is considered usable. The system further searches for the latest replay checkpoint in that replay version, i.e., the last replay checkpoint generated before the disconnection and the one with the highest task logic time. The system sends the latest replay checkpoint to the client. After loading the replay checkpoint, the client restores the 3D situational awareness, incremental event cursor position, and sensor data object list before the disconnection. The replay continues from the task logic time corresponding to this checkpoint, and the replay speed uses the settings from before the disconnection. If the replay version corresponding to the generation number has been reclaimed or is no longer available, the system returns an unavailable notification to the client, and the client needs to re-initiate the replay request, starting the replay from the new replay checkpoint.

[0092] Optionally, the method further includes: maintaining a mode state machine, the mode state machine including at least a real-time monitoring state, a switching transition state, and a historical playback state; in response to a switching request from the real-time monitoring state to the historical playback state, disconnecting all subscription connections with the real-time data source, and not receiving new real-time subscription requests before the switching is completed; after the switching is completed, all event records in the current playback session are obtained from the pre-preserved playback event stream, and no real-time data injection is received.

[0093] Specifically, firstly, the system maintains a modal state machine, which defines three states: real-time monitoring state, transitional state, and historical playback state. In the real-time monitoring state, the system subscribes to real-time data streams from various data sources, continuously receiving newly arriving event records and pushing them to clients in real time. In the historical playback state, the system only reads event records from the pre-set playback event stream for playback. The transitional state is an intermediate state between the two states, used for cleanup and preparation operations required for state switching; no business operations are allowed in the transitional state. When the system receives a switch request from the real-time monitoring state to the historical playback state, it first changes the modal state machine from the real-time monitoring state to the transitional state. Upon entering the transitional state, the system performs the following disconnection operations: it iterates through all currently established real-time data source subscription connections, sends a cancel subscription command to each one, and closes the corresponding network connection after each data source confirms that the subscription has terminated. For subscription connections that have not yet completed cancellation confirmation, the system sets a timeout period; after the timeout, the connection is forcibly closed, and the subscription that did not terminate normally is recorded. During the period when all subscription connections are disconnected, the system rejects any new real-time subscription requests in the transitional state. When a new real-time subscription request arrives, the system immediately returns a rejection response, without establishing a new connection or adding the request to the waiting queue. After all real-time data source subscription connections are disconnected, the system clears any unprocessed event records in the real-time data receive buffer to ensure that the replay session after the switch is not affected by residual real-time data; it points the incremental event cursor to the user-specified historical replay start position and loads the replay checkpoint corresponding to that position; and it changes the mode state machine from the switching transition state to the historical replay state. After switching to the historical replay state, all event records in the current replay session are obtained from the pre-defined replay event stream. The system reads event records from the replay event stream in the task logic time sequence and pushes them to the client, without accepting any real-time data injection. If a real-time data source still attempts to push event records after the switch, the system directly discards the data and returns a rejection notification to the data source indicating that it is currently in the historical replay state.

[0094] The method provided in this embodiment, in its first aspect, eliminates the heterogeneity of data sources by uniformly encapsulating multimodal heterogeneous data—such as navigation data, attitude data, sonar data, point cloud data, video images, control commands, and alarm records—which vary in format, sampling frequency, and data volume, into event records containing data source identifiers, device timestamps, and data type identifiers. This allows all subsequent steps—segmentation mapping, confidence assessment, water level determination, and playback progression—to use the unified event record as the basic operational unit, eliminating the need for source-by-source adaptation of the original data format and significantly reducing the complexity of system implementation. Furthermore, for large-volume sensor data such as sonar images and point cloud data, the event record only carries a reference to the original data while the data itself is stored independently. This maintains the lightweight nature of the event record while ensuring that complete data can be loaded from the storage system as needed during the playback stage.

[0095] Secondly, by establishing a task logic timeline independent of any single device clock and employing segmented mapping for each data source, the system automatically re-segments and independently estimates parameters for adjacent segments when a clock reset or restart is detected. This fundamentally solves the cross-source alignment problem caused by inconsistent multi-source clocks. Since the start times of clocks from different sensors vary and their frequencies drift, directly aligning with device timestamps inevitably leads to misalignment of records corresponding to the same real time in different data sources. Introducing the task logic timeline uniformly converts all event records to the same position in the same time coordinate system, achieving precise cross-source synchronization. The segmented design ensures that mapping parameters before and after reset or restart do not interfere with each other; robust weighted least squares estimation primarily uses anchor points near the midpoint of the interval and secondarily uses those far from the anchor points, suppressing interference from abnormal anchor points; the degradation strategy of estimating only the offset when anchor points are insufficient avoids forcibly fitting frequency drift and introducing unreliable results when information is insufficient; special data sources such as sonar and video are also incorporated into the unified mapping framework by constructing device times, ensuring that all data sources are orchestrated under the same time base.

[0096] Thirdly, by comprehensively considering five dimensions—time synchronization reliability, anchor point fitting residuals, network stability, sequence continuity, and equipment quality status—time confidence is calculated. This quantifies the previously scattered and difficult-to-compare data quality information into a unified confidence index. A three-tiered differential processing is implemented using a first and second threshold: high-confidence data directly participates in orchestration, medium-confidence data participates after interpolation compensation, and low-confidence data is marked as missing. This maximizes the retention of usable data while preventing unreliable data from contaminating the playback results. Based on this, a global watermark is determined by considering the arrival progress of various data sources. This global watermark defines the solidified intervals, allowing the system to automatically determine which time intervals have complete and reliable data that can be safely replayed. The differential processing of late data further protects the stability of published playback content. Data that does not affect published content is directly included; if it does affect content, a new version is derived and a bypass audit is registered, balancing data integrity and playback continuity.

[0097] Fourthly, different synchronization window strategies are adopted for different data types, fully aligning with the characteristics and playback requirements of various data types. Navigation and attitude data are interpolated using preset duration windows to generate continuous trajectories and attitude sequences, avoiding screen jumps caused by the original sampling intervals. Command and alarm data are not downsampled or interpolated, but directly inserted at the original time, fully preserving the precise temporal sequence of discrete events. Sonar and point cloud data only load target segments whose time envelopes intersect with the current track window, avoiding the invalid loading of large volumes of data in irrelevant time periods. During downsampling, four types of target sampling points are retained: event proximity points, direction change points, extreme points, and mass change points, ensuring that the correspondence between events and trajectories is not destroyed while reducing the amount of data. Sensor data objects are loaded on demand through dual judgment of time and spatial envelopes. After content summary deduplication, identical blocks are directly reused in the cache, which reduces the loading latency at playback startup and saves bandwidth between the client and storage nodes.

[0098] Corresponding to the aforementioned embodiment of a multimodal synchronous playback method for underwater vehicles based on a mission logic clock, this application also provides an embodiment of a multimodal synchronous playback system for underwater vehicles based on a mission logic clock.

[0099] Figure 2 This is a schematic diagram of the structure of the underwater multimodal synchronous playback system based on the mission logic clock provided in Embodiment 2 of this application. Please refer to... Figure 2 The system provided in this embodiment includes an encapsulation module 210, a segmentation module 220, a determination module 230, a generation module 240, and a loading module 250.

[0100] The encapsulation module 210 is used to encapsulate multimodal heterogeneous data generated by multiple data sources during the sea trial into a unified event record. The event record includes at least a data source identifier, a device timestamp, and a data type identifier.

[0101] The segmentation module 220 is used to establish a segmented mapping relationship from device time to task logic time for each data source, and to uniformly map the event records of different data sources to the task logic time axis independent of any device clock. When the clock of the target data source is detected to be reset or restarted, the mapping relationship of the target data source is re-segmented, and adjacent segments use their own independent mapping parameters.

[0102] The determining module 230 is used to evaluate the time confidence of each event record, determine the global water level by combining the arrival progress and time confidence of each data source, and determine the task logic time interval that is not greater than the global water level as the solidified interval.

[0103] The generation module 240 is used to generate replay checkpoints at preset key task nodes in the solidified interval. The replay checkpoints include a three-dimensional situational snapshot at the corresponding time, an incremental event cursor that identifies the next event to be replayed, and a list of sensor data objects.

[0104] The loading module 250 is configured to, in response to a playback request, load the playback checkpoint, advance the event playback along the task logic time axis within the solidified interval starting from the incremental event cursor at the playback speed specified in the playback request, and load sensor data objects as needed based on the sensor data object list.

[0105] The system in this embodiment can be used to execute Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0106] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0107] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0108] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for multimodal synchronous playback of underwater vehicles based on a mission logic clock, characterized in that, The method includes: Multimodal heterogeneous data generated from multiple data sources during the sea trial are encapsulated into a unified event record, which includes at least a data source identifier, a device timestamp, and a data type identifier. For each data source, a segmented mapping relationship from device time to task logic time is established. Event records from different data sources are uniformly mapped to a task logic time axis independent of any device clock. When a clock reset or restart of the target data source is detected, the mapping relationship of the target data source is re-segmented, and adjacent segments use their own independent mapping parameters. Assess the time confidence of each event record, combine the arrival progress and time confidence of each data source to determine the global water level, and determine the task logic time interval that is not greater than the global water level as the solidified interval; At the preset key task nodes in the solidified area, a replay checkpoint is generated. The replay checkpoint includes a three-dimensional situational state snapshot at the corresponding time, an incremental event cursor that identifies the next event to be replayed, and a list of sensor data objects. In response to a playback request, the playback checkpoint is loaded, and starting from the incremental event cursor, the event playback is advanced along the task logic timeline within the solidified interval at the playback speed specified in the playback request, and sensor data objects are loaded as needed based on the sensor data object list.

2. The method according to claim 1, characterized in that, The assessment of the time confidence of each event record includes: The system obtains the timing reliability factor of the data source corresponding to each event record, the anchor fitting residual factor of the segmented mapping, the network stability factor of the data transmission link, the time continuity factor of the event record sequence, and the quality status factor of the data source device itself. After assigning corresponding preset weights to each factor, a weighted calculation is performed to obtain the time confidence of each event record. The time confidence increases with the increase of the timing reliability factor, network stability factor, time continuity factor, and quality status factor, and decreases with the increase of the anchor fitting residual factor. The time confidence level is compared with a preset first threshold and a second threshold. Event records that are not lower than the first threshold directly participate in the synchronization orchestration. Event records that are between the first threshold and the second threshold participate in the synchronization orchestration after interpolation compensation. Event records that are lower than the second threshold are marked as missing. The first threshold is higher than the second threshold.

3. The method according to claim 1, characterized in that, The determination of the global water level by integrating the arrival progress and time confidence of various data sources includes: Obtain the maximum task logic time of continuously arriving event records from each data source on the task logic timeline, subtract the preset allowable lateness tolerance from the maximum task logic time to obtain the local water level of the data source, and take the minimum value of the local water level among all data sources as the global water level. Events whose task logic time is later than the global water level are marked as late data. If the impact of the late data does not involve the published replay content, the late data is included in the current replay version. If the impact of the late data involves the published replay content, a new version is derived based on the current replay version and included, and the arrival event of the late data is registered in the bypass audit record.

4. The method according to claim 1, characterized in that, The segmented mapping relationship is divided into multiple mapping segments by the clock reset or restart events of each data source. The device time interval between two adjacent re-segmentations constitutes a mapping segment. The step of establishing a segmented mapping relationship from device time to task logic time for each data source includes: Multiple alignment anchor points are selected, each alignment anchor point corresponds to a reference event with a known task logic time and the device timestamp record of each data source when the reference event occurs; For each mapping segment, robust weighted least squares estimation is performed using the alignment anchors contained in the mapping segment to determine the mapping parameters of the mapping segment; wherein, the weight of each alignment anchor in the least squares estimation decreases as the time distance between the alignment anchor and the midpoint of the mapping segment increases; Based on the mapping parameters, a linear mapping relationship from device time to task logic time is constructed within the mapping segment. Each data source aligns the event records to the task logic time axis through its own linear mapping relationship.

5. The method according to claim 1, characterized in that, The three-dimensional situational snapshot records the three-dimensional spatial position, attitude, heading, and working status of each sensor of the underwater vehicle at the corresponding mission logic moment; the incremental event cursor records the position identifier of the first event to be replayed after the replay checkpoint in the event stream; the sensor data object list includes the temporal envelope range, spatial envelope range, content summary, and block storage index of each data source sensor data object, and sensor data objects with the same content summary are stored in only one copy in the storage system.

6. The method according to claim 1, characterized in that, The on-demand loading of sensor data objects based on the sensor data object list includes: Based on the temporal and spatial envelopes of each entry in the sensor data object list, it is determined whether the current playback window needs to load the corresponding sensor data object. If loading is required, an access credential is generated based on the block index of the corresponding entry. The client obtains the data block through the access credential, and sensor data object blocks with the same content summary are directly reused in the cache.

7. The method according to claim 1, characterized in that, In response to a playback request, the playback checkpoint is loaded, and starting from the incremental event cursor, the event playback is advanced along the task logic timeline within the solidified interval at the playback speed specified in the playback request, including: Starting from the position of the incremental event cursor, read the event records in the task logic time sequence and advance along the task logic time axis; During the process, differentiated synchronization window strategies are adopted for different types of event records to form a unified playback event stream. Specifically, for navigation data and attitude data, a preset duration is used as the synchronization window, and the data within the synchronization window is interpolated to generate continuous trajectories and attitude sequences. For command and alarm data, the playback event stream is directly inserted according to the original event record time. For sonar and point cloud data, the intersection of the time envelope of the target object segment and the current track window is calculated, and only the target object segments that overlap with the track window in time are loaded. When downsampling the target data, the target sampling points are retained; the target sampling points include at least event proximity points, direction change points, extreme points, and quality change points; The task logic timeline is advanced at the playback speed, and the current task logic time is determined based on the task logic time of the playback checkpoint, the playback speed, and the elapsed wall clock time. Events in the replay event stream whose task logic time is no later than the current task logic time are pushed to the client to drive the 3D situation update.

8. The method according to claim 1, characterized in that, The method further includes: In response to a progress jump request, a new three-dimensional situational state snapshot is constructed without changing the current display on the client. The replay checkpoint corresponding to the logical moment of the jump target task is loaded. The content summary of the replay checkpoint is verified against the summary of the current situation on the client. If the verification passes, the current situation on the client is switched to the situation corresponding to the replay checkpoint by atomic replacement, and the current generation number is incremented. In response to a disconnection and reconnection request, obtain the client's current generation number, verify whether the replay version corresponding to the current generation number is still available, and if available, restore the replay session based on the latest replay checkpoint of the replay version.

9. The method according to claim 1, characterized in that, The method further includes: A maintenance mode state machine, wherein the mode state machine includes at least a real-time monitoring state, a switching transition state, and a history playback state; In response to a switching request from the real-time monitoring state to the historical playback state, all subscription connections with the real-time data source are disconnected, and no new real-time subscription requests are received until the switching is complete. After the switch is completed, all event records in the current playback session are obtained from the pre-defined playback event stream and do not receive real-time data injection.

10. A multimodal synchronous playback system for underwater vehicles based on a mission logic clock, characterized in that, The system includes an encapsulation module, a segmentation module, a determination module, a generation module, and a loading module; The encapsulation module is used to encapsulate multimodal heterogeneous data generated by multiple data sources during the sea trial into a unified event record. The event record includes at least a data source identifier, a device timestamp, and a data type identifier. The segmentation module is used to establish a segmented mapping relationship from device time to task logic time for each data source, and to uniformly map the event records of different data sources to the task logic time axis independent of any device clock. When the clock of the target data source is detected to be reset or restarted, the mapping relationship of the target data source is re-segmented, and adjacent segments use their own independent mapping parameters. The determining module is used to evaluate the time confidence of each event record, determine the global water level by combining the arrival progress and time confidence of each data source, and determine the task logic time interval that is not greater than the global water level as the solidified interval. The generation module is used to generate replay checkpoints at preset key task nodes in the solidified interval. The replay checkpoints include a three-dimensional situational snapshot at the corresponding time, an incremental event cursor that identifies the next event to be replayed, and a list of sensor data objects. The loading module is used to load the playback checkpoint in response to the playback request, starting from the incremental event cursor, advance the event playback along the task logic time axis within the solidified interval at the playback speed specified in the playback request, and load sensor data objects as needed based on the sensor data object list.

Images