Synchronous playback method, device and equipment based on multi-modal data of marine target and medium

Abstract

The invention provides a synchronous playback method, device and equipment based on multi-modal data of a marine target and a medium, and relates to the technical field of marine target data processing, and the method comprises the steps: obtaining video data, collected by photoelectric equipment, of marine target motion; analyzing metadata of the video data to obtain video time information and a target list formed by at least one marine target included in the video data; according to the video recording time information, determining the starting time of track playback of the marine target in the target list as the starting time of video playing in the video recording time information; according to the video playing ending time in the video time information, obtaining a track data stream of the marine target with the playing duration from the starting time to the ending time; and when the video data in the playing duration is played, playing back the track data stream of the marine target in the playing duration according to the starting time of the track playback time. According to the scheme of the invention, accurate synchronization of the video being played and the trajectory data stream of the marine target is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of maritime target data processing, and in particular to a synchronous playback method, device, equipment and medium based on maritime target multimodal data. Background Art

[0002] In the field of maritime surveillance, optoelectronic devices are widely used for target tracking and situational awareness. Usually, when reviewing historical videos, it is necessary to simultaneously view the changes in the position and trajectory of maritime targets in the video.

[0003] Existing technical solutions use a separate architecture, processing optoelectronic video playback and target trajectory playback as independent modules. This presents the following issues: 1. Based on simple timestamp comparison, it cannot handle device clock deviation and network latency. 2. It ignores the three-dimensional posture change history of the optoelectronic device. 3. It cannot identify and display the target association during the optoelectronic device tracking process. 4. It requires users to manually coordinate multiple playback control interfaces. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a synchronous playback method, device, equipment and medium based on multimodal data of maritime targets, so as to accurately synchronize the video and the trajectory data stream of the maritime targets.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows: In a first aspect, an embodiment of the present invention provides a synchronous playback method based on multimodal data of maritime targets, comprising: Acquire video data of maritime target movement collected by optoelectronic equipment; parsing metadata of the video data to obtain video recording time information and a target list consisting of at least one marine target included in the video data; According to the video recording time information, determining the start time of the trajectory playback of the sea target in the target list as the start time of the video playback in the video recording time information; According to the end time of the video playback in the video time information, a trajectory data stream of the sea target with a playback duration from the start time to the end time is obtained; When the video data within the playback time is played, the trajectory data stream of the sea target within the playback time is played back according to the starting time of the trajectory playback time.

[0006] Optionally, the metadata of the video data is parsed to obtain video recording time information, including: The metadata of the video data is parsed to obtain the start time, end time, video file address and video detail object of the video, wherein the video detail object includes: a time series array of the photoelectric device posture data.

[0007] Optionally, the metadata of the video data is parsed to obtain a target list consisting of at least one marine target included in the video data, including: parsing metadata of the video data to obtain a target identifier of at least one marine target captured during the video recording process; A target list is obtained according to the target identifier of the at least one maritime target.

[0008] Optionally, determining, based on the video recording time information, the start time of the trajectory playback of the maritime target in the target list as the start time of the video playback in the video recording time information includes: According to the video recording time information, the trajectory playback data in the trajectory playback cache is cleared, and the start time of the trajectory playback cache is set to the start time of the video recording playback in the video recording time information.

[0009] Optionally, obtaining the trajectory data stream of the maritime target having a playback duration from the start time to the end time according to the end time of the video playback in the video time information includes: Determine the playback duration according to the end time of the video playback and the start time of the video playback in the video playback time information; Starting from the start time of the trajectory playback, the trajectory data stream of the sea target within the playback duration is obtained.

[0010] Optionally, when replaying the trajectory data stream of the sea target within the playback duration according to the start time of the trajectory playback time, the process further includes: generating a pointing line of the optoelectronic device pointing to the marine target based on the attitude data of the optoelectronic device; According to the starting time of the trajectory playback time, the direction line and the trajectory data stream of the sea target are synchronously displayed on the chart display interface.

[0011] Optionally, synchronously displaying the pointing line and the trajectory data stream of the sea target on a nautical chart display interface includes: Extract sea targets within the playback duration through the secondary cache; Loading the trajectory data stream of the maritime target through the first-level cache and displaying it on the chart display interface; The pointing line of the photoelectric device is displayed in the form of a vector diagram on the map content of the nautical chart display interface.

[0012] In a second aspect, an embodiment of the present invention further provides a synchronous playback device based on multimodal data of marine targets, comprising: An acquisition module is used to acquire video data of the movement of maritime targets collected by optoelectronic equipment; The processing module is configured to parse metadata of the video data to obtain video time information and a target list consisting of at least one maritime target included in the video data; determine, based on the video time information, a start time for trajectory playback of the maritime targets in the target list as the start time of the video playback in the video time information; obtain, based on the end time of the video playback in the video time information, a trajectory data stream of the maritime targets for a playback duration between the start time and the end time; and, when playing video data within the playback duration, replay the trajectory data stream of the maritime targets within the playback duration according to the start time of the trajectory playback time.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method described in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processor.

[0015] The above solution of the present invention includes at least the following beneficial effects: The above-mentioned solution of the present invention obtains video data of the movement of marine targets collected by optoelectronic equipment; parses the metadata of the video data to obtain video time information and a target list consisting of at least one marine target included in the video data; determines the start time of the trajectory playback of the marine targets in the target list as the start time of the video playback in the video time information based on the video time information; obtains the trajectory data stream of the marine targets within the playback time from the start time to the end time based on the end time of the video playback in the video time information; when playing the video data within the playback time, the trajectory data stream of the marine targets within the playback time is played back according to the start time of the trajectory playback time. This achieves accurate synchronization between the video being played and the trajectory data stream of the marine targets, overcoming the problem in the prior art of being based on simple timestamp comparison and being unable to handle device clock deviation and network delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 1 is a flow chart of an embodiment of a synchronous playback method based on multimodal data of marine targets of the present invention; Figure 2 This is a flow chart of an embodiment of an application scenario of the present invention; Figure 3 is a flow chart of an embodiment of another application scenario of the present invention; Figure 4This is a flow chart of an embodiment of another application scenario of the present invention; Figure 5 It is a schematic diagram of the directional lines and historical tracks displayed on the map after rendering; Figure 6 1 is a schematic structural diagram of an embodiment of a synchronous playback device based on multimodal data of marine targets according to the present invention; Figure 7 It is a schematic structural diagram of an electronic device embodiment of the present invention. DETAILED DESCRIPTION

[0017] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0018] like Figure 1 As shown, the present invention proposes a synchronous playback method based on multimodal data of marine targets, the method comprising: Step 1: Obtain video data of target movement at sea collected by optoelectronic equipment; Step 2: parsing metadata of the video data to obtain video time information and a target list consisting of at least one marine target included in the video data; Step 3, according to the video time information, determining the start time of the trajectory playback of the sea target in the target list as the start time of the video playback in the video time information; Step 4, according to the end time of the video playback in the video time information, obtaining the trajectory data stream of the sea target with the playback time from the start time to the end time; Step 5: When the video data within the playback duration is played, the trajectory data stream of the sea target within the playback duration is played back according to the start time of the trajectory playback time.

[0019] In this embodiment, the video data is obtained by using a certain photoelectric device to photograph multiple ships sailing on the sea (i.e., photographing at least one target at sea), where each ship has different identification information. During the shooting, the photoelectric device needs to record the shooting time, the posture of the photoelectric device, the identification information of the photographed ship, etc.

[0020] After acquiring the video data, it is parsed to obtain the recording time information and target list. The target time information is the start and end time of the recording (for example, the target start time is time a and the end time is time b). The target list is the identification information of different ships captured in the video and the list of trajectory return data.

[0021] If there is only one target in the target list, the start time c of the trajectory playback data for this target (i.e., a specific vessel) is obtained from the target list (the start time in the trajectory playback data is time c, which is less than or equal to time b and greater than or equal to time a). This time c is set as the playback start time of the video (i.e., the video starts at time c), and the video is played. This synchronizes the start time of the currently playing video with the start time of the trajectory playback data. Simultaneously, the end time b recorded in the video data is determined, and the trajectory data stream for the target from time c to time b is obtained. That is, the video plays from time c to time b, and the trajectory playback data also plays from time c to time b. This achieves time synchronization between the currently playing video and the trajectory data. If there are multiple targets in the target list, the start time of the trajectory playback data for each target is obtained separately (for example, if there are two targets, the trajectory playback data for the first target starts at time d, and the trajectory playback data for the second target starts at time e, where time d is earlier than time e, and both time d and time e are less than or equal to time b and greater than or equal to time a). The earliest of these trajectory playback data start times is set as the start time of the video (i.e., time d is set as the start time of the video). This means that the video plays from time d to time b, with the trajectory playback data for the first target playing from time d to time b, and the trajectory playback data for the second target playing from time e to time b. Furthermore, during video playback, the playback time period needs to be recorded. Since time c is the start time of the video playback, time c is added to the currently played time period to obtain the current playback time of the video. The trajectory data of the target at the current playback time is then retrieved, and the target trajectory is displayed in real time using the retrieved data.

[0022] For step 1, the optoelectronic device is a camera or photographic device that captures (or collects) images of a target at sea. For example, the optoelectronic device may be a fixed camera, a drone, or a satellite. The target at sea may be a moving ship.

[0023] In a certain application scenario, a camera device is fixed at a first location, and the target at sea is a cargo ship transporting cargo from a second location to a third location. The task of the camera device is to capture the cargo transportation video data of the cargo ship from the second location to the third location. During the shooting process, the camera device will adjust the shooting angle as the cargo ship moves.

[0024] In an optional embodiment of the present invention, parsing the metadata of the video data in step 2 to obtain the video recording time information includes: Step 21 , parsing the metadata of the video data to obtain the start time, end time, video file address and video detail object of the video, wherein the video detail object includes: a time series array of the photoelectric device posture data.

[0025] When photographing a target at sea, optoelectronic equipment must adjust its shooting posture to follow the target's movement, such as the device's altitude, horizontal angle, and the sector boundaries of its horizontal field of view. Therefore, a time series array of optoelectronic equipment attitude data represents the device's attitude data at consecutive moments. For example, the structure of the time series array of optoelectronic equipment attitude data is: [time, device altitude, horizontal angle, tilt angle, field of view angle].

[0026] In an optional embodiment of the present invention, in step 2, the metadata of the video data is parsed to obtain a target list consisting of at least one marine target included in the video data, including: Step 22: parsing metadata of the video data to obtain a target identifier of at least one marine target captured during the video recording process; Step 23: Obtain a target list according to the target identifier of the at least one maritime target.

[0027] In a specific application scenario, to monitor ships entering and leaving a port, the optoelectronic device's posture can be fixed (monitoring only that port). Therefore, multiple ships may enter and exit the port at a given time period (point in time), each with a target identifier. Consequently, multiple target identifiers of these ships may be present in the video data.

[0028] In another application scenario, an optoelectronic device may track and film a moving cargo ship. During this process, it may capture other ships, causing other targets to appear in the video data of the cargo ship. Furthermore, the video data may contain target identifiers of other ships in addition to the target identifier of the cargo ship.

[0029] In this embodiment, the target list is a list composed of multiple different target identifiers in the video data, and each identifier has a corresponding trajectory data stream.

[0030] As an example, the target identification may be a unique identifier and / or a MMSI maritime identification code of the tracking target.

[0031] As an example, the data structure of the target list can be [identifier 1, nth moment, trajectory playback data of identifier 1 at nth moment], [identifier 1, n+1th moment, trajectory playback data of identifier 1 at n+1th moment], [identifier 2, n+1th moment, trajectory playback data of identifier 2 at n+1th moment].

[0032] In an optional embodiment of the present invention, in step 3, determining, based on the video recording time information, the start time of the trajectory playback of the maritime target in the target list as the start time of the video playback in the video recording time information includes: Step 31: Clear the trajectory playback data in the trajectory playback cache according to the video recording time information, and set the start time of the trajectory playback cache to the start time of the video recording in the video recording time information.

[0033] In this embodiment, the trajectory playback data in the trajectory playback buffer is cleared, and the start time of the video playback is reset, thereby ensuring the temporal consistency between the trajectory playback data and the video being played.

[0034] In an optional embodiment of the present invention, step 4, obtaining a trajectory data stream of a maritime target having a playback duration from the start time to the end time according to the end time of the video playback in the video time information, includes: Step 41, determining the playback duration according to the end time and the start time of the video playback in the video playback time information; Step 42 : starting from the start time of the trajectory playback, obtaining the trajectory data stream of the sea target within the playback duration.

[0035] Using the above example, the start and end times of the video are from time a to time b, respectively. The start time is the start time of the trajectory playback data, that is, time c, and the playback duration is from time c to time b. The trajectory data stream of the maritime target from time c to time b is obtained.

[0036] In this embodiment, the playback time of the video and the start time of the trajectory data stream of the target at sea are set to the same time (ie, time c), thereby achieving time synchronization.

[0037] In an optional embodiment of the present invention, step 5, when replaying the trajectory data stream of the sea target within the playback time according to the start time of the trajectory playback time, further includes: Step 51, generating a pointing line of the optoelectronic device pointing to the marine target based on the attitude data of the optoelectronic device; Step 52 : Synchronously display the direction line and the trajectory data stream of the sea target on a nautical chart display interface according to the start time of the trajectory playback time.

[0038] The posture data of a photoelectric device (camera) includes the device's tilt angle, height, and horizontal angle. The tilt angle and height can be used to calculate the device's visible range. The horizontal angle can be used to calculate the direction of the pointing line and the device's monitoring coverage (sector boundary). Specifically, the pointing line is a vector arrow with the photoelectric device as its origin.

[0039] The trajectory data stream represents the coordinates of the target at consecutive moments, and its data structure can be referenced as [identifier 1, moment n, coordinate 1], [identifier 1, moment n+1, coordinate 2], and [identifier 2, moment n, coordinate 3]. The pointing line and the trajectory data stream of the maritime target are synchronously displayed on a nautical chart display interface. Specifically, the pointing line (a vector arrow) is calculated based on the attitude data from the optoelectronic device, and the pointing line and coordinates are displayed on the map.

[0040] The formula for calculating the pointing line (from the camera to the target) in the camera's local coordinate system is:

[0041] in, : The vector of the pointing line in the camera's local coordinate system; R: the camera's rotation matrix (calculated from the attitude angle); : World coordinates of each target (longitude, latitude, altitude); : The world coordinates of the camera (longitude, latitude, altitude); In the above step 51, generating a pointing line of the optoelectronic device pointing to the marine target based on the attitude data of the optoelectronic device includes: Convert the latitude and longitude coordinates of the camera and target into geocentric Cartesian coordinates; Calculate the pointing vector in the world coordinate system ; According to the camera attitude angle (azimuth , pitch angle , roll angle ) Construct the rotation matrix: ; Convert to the camera local coordinate system: ; The pointing line vector can finally be expressed as: [ ].

[0042] As an example, when a video is played, the playback time point of the video can be monitored in real time, and the posture data of the optoelectronic device and the trajectory data stream of the sea target at this playback time point can be obtained based on this playback time point, and the display of the pointing line and trajectory data on the map can be adjusted in real time.

[0043] As an example, when a video is played, the playback time period of the video can be monitored in real time, and the playback time point of the current video can be determined based on the time point when the video starts playing and the time period that has been played. Based on the playback time point of the current video, the posture data of the optoelectronic device and the trajectory data stream of the maritime target at the playback time point can be queried, and the display of the pointing line and trajectory data on the map can be adjusted.

[0044] In this embodiment, the start time of the video recording, the start time of the optoelectronic device's attitude data, and the start time of the maritime target's trajectory data stream (multimodal data) are set to the same time, achieving time synchronization. Furthermore, the optoelectronic attitude parameters (tilt angle, horizontal angle, field of view angle) are effectively associated with the target's motion trajectory data. This overcomes the problem of prior art pointing lines lacking historical attitude data. Dynamic reproduction of pointing lines based on historical attitude data is achieved.

[0045] In an optional embodiment of the present invention, step 52 of synchronously displaying the direction line and the trajectory data stream of the maritime target on a nautical chart display interface includes: Step 521, extracting sea targets within the playback duration through the secondary cache; Step 522: Load the trajectory data stream of the maritime target through the first-level cache and display it on the chart display interface; Step 523: Display the direction line of the optoelectronic device in the form of a vector diagram on the map content of the nautical chart display interface.

[0046] The purpose of the second-level cache is to determine all the data required for the long playback time of the video, while the first-level cache preloads a portion of the data based on the current playback time of the video. Using the above example, the second-level cache caches the attitude data of the optoelectronic device and the trajectory data stream of the maritime target from time c to time b. To determine the coordinate position of the pointing line and the target based on the attitude data of the optoelectronic device and the trajectory data stream of the maritime target, the first-level cache can be added to the second-level cache (i.e., the second-level cache already caches two hours of data, that is, all data. Based on the current time n, the first-level cache has already cached the data from time n to time n+5, and the data from time n to time n+5 is included in the second-level cache).

[0047] In a certain application scenario, if the L2 cache contains data from time 1 to time n, and the current video playback time is time 1, then the L1 cache contains data from time 1 to time 5 (n is greater than or equal to 5, meaning the L1 cache only caches data for a fixed time period and does not cache all data). The trajectory data stream of the maritime target cached in the L1 cache is [identifier 1, time 1, coordinate 1], [identifier 1, time 2, coordinate 2], [identifier 1, time 3, coordinate 3], [identifier 1, time 5, coordinate 5], and the posture data of the optoelectronic device cached in the L1 cache is [identifier 1, time 1, posture data 1], [identifier 1, time 2, posture data 2], [identifier 1, time 4, posture data 4], [identifier 1, time 5, posture data 5]. When the video playback time is time 1, the target coordinates and pointing line of target 1 in the currently playing video are loaded and calculated based on coordinate 1 and posture data 1 at time 1 (the current video playback time is time 1). When the video playback time reaches time 3, the coordinate data for time 3 is loaded, but the posture data for time 3 is missing. The posture data for time 3 is calculated based on the posture data for time 2 and time 4. That is, posture data 3 is calculated based on posture data 2 and posture data 4. The target coordinates and pointing line of target 1 in the currently playing video are then calculated based on coordinates 3 and posture data 3 (the current video playback time is again time 3). When the video playback time reaches time 4, the posture data for time 4 is loaded, but the coordinate data for time 4 is missing. The coordinates for time 4 are calculated based on the coordinate data for time 3 and time 5. That is, coordinates 4 is calculated based on coordinates 3 and coordinates 5. The target coordinates and pointing line of target 1 in the currently playing video are then calculated based on coordinates 4 and posture data 4. If the current level 1 cache time is less than the preset cache limit, the level 1 cache is re-set based on the current video playback time.

[0048] As an example, the posture data at the third moment can be determined by the following method:

[0049] in, , represents the third moment, Indicates the 4th moment, represents the second moment, Indicates the posture data at the second moment, Indicates the posture data at the 4th moment, Indicates the posture data at the third moment.

[0050] As an example, the coordinates of the 4th moment can be determined by:

[0051] in, , Indicates the 4th moment, Indicates the 5th moment, represents the third moment, represents the coordinates at the third moment, represents the coordinates at the 5th moment, Indicates the coordinates of time 4. This embodiment uses the first and second level caches to overcome the problem that the existing system may experience freezes, frame drops or synchronization deviations during high-speed playback and cannot support the smooth playback requirement of 16x speed.

[0052] refer to Figure 2 , Figure 2 A schematic diagram of an application scenario provided by the present invention.

[0053] First, the user selects the video to play and loads the video's metadata. There are two types of videos: standard videos (where the optoelectronic device is fixed and doesn't adjust to the target being filmed) and motion-tracking videos (where the optoelectronic device adjusts to the target's movement). For standard videos, the basic pose data is extracted; for motion-tracking videos, the basic pose data and target-related information (target identification and location) are extracted.

[0054] Then, the video player is initialized, the playback start time of the video is adjusted according to the basic posture data, and the basic posture data and / or target association information of each target in the target list is extracted.

[0055] If the user doesn't enable synchronization mode, only the video will be played. If the user does enable synchronization mode, the video will be played while the pointing line and target coordinates are displayed on the map. This requires initializing the target trajectory playback player, clearing the previous trajectory playback data, setting the trajectory replay time (in the example above, the trajectory playback time is from time c to time b), configuring the wedsocket parameters, and looping to check if the playback time of the current video has changed. If so, the draw function is called to update the target trajectory position (i.e., target-related information) and obtain the basic posture data of the optoelectronic device. To obtain the basic posture data and / or target-related information, it is necessary to determine whether the current basic posture data and / or target-related information are available in the secondary cache. If so, they are directly retrieved from the secondary cache. If not, the current basic posture data and / or target-related information can be calculated using a time difference algorithm and rendered on the map.

[0056] refer to Figure 3 、 Figure 4 and Figure 5, which is a schematic diagram of another application scenario provided by the present invention.

[0057] This embodiment provides a synchronous playback system based on multimodal data of maritime targets. The system includes a video playback controller, a target trajectory player, an optoelectronic pointing line renderer, a multi-speed time controller, a back-end interface, and a map rendering engine.

[0058] 1. Before playing a video, data must be stored. The system uses a hierarchical data structure to store the complete information of optoelectronic recordings. During optoelectronic device control, the system collects the device's three-dimensional posture parameters at regular intervals (typically 100 milliseconds). The core parameters collected include pan, tilt, and viewHorizon. Pan represents the device's horizontal turning angle, tilt represents the device's pitch angle, and viewHorizon represents the device's field of view. When the optoelectronic device is in tracking mode, the system additionally records the tracked target's unique identifier and MMSI (Maritime Identification Number). The system utilizes a time-series optimized storage strategy, using timestamp indexing to quickly locate posture data at a specific point in time. For long-duration recordings, a segmented storage mechanism is used, with each segment containing a fixed duration of continuous data, facilitating subsequent rapid retrieval and cache management.

[0059] 2. Multi-speed time controller. The system supports continuous adjustment from 0.25x to 16x speed, achieving smooth playback at high speeds through intelligent frame rate control and cache preloading mechanisms. When the playback speed exceeds 4x, the system automatically activates high-efficiency rendering mode, adaptively adjusting the time step to ensure millisecond-level time synchronization accuracy even at 16x speed.

[0060] During high-speed playback, the system employs predictive data loading and rendering optimization mechanisms. By dynamically adjusting the cache window size (up to several hours of data before and after the current playback time), it pre-loads the upcoming electro-optical attitude data and target trajectory information. Furthermore, keyframe rendering technology intelligently skips the detailed calculation of some intermediate frames while ensuring synchronization accuracy.

[0061] The system monitors rendering performance in real time and automatically adjusts rendering accuracy and update frequency when it detects a drop in frame rate or an increase in latency. When playing at 16x speed, time synchronization accuracy is prioritized over visual smoothness, ensuring accurate display of key electro-optical pointing line changes and target motion trajectories.

[0062] 3. Target Trajectory Replay. The system uses a layered cache architecture to achieve real-time rendering and display of targets. The first-level cache stores complete target data, the second-level cache extracts the target subset within the current playback time window, and the index cache records the target position, achieving efficient time-driven rendering.

[0063] The drawByTime function is called based on the playback time. A temporal interpolation algorithm is used to calculate the target's precise coordinates between historical locations. Combined with field of view detection and adaptive performance adjustment, this enables smooth dynamic target display. The system automatically manages the target lifecycle, promptly clearing expired targets to ensure rendering performance and display accuracy. This rendering mechanism provides a stable underlying foundation for the synchronized display of electro-optical pointing lines and target trajectories, ensuring a unified time base and display consistency for multimodal data.

[0064] 4. Photoelectric pointing line renderer. The system uses a layered rendering architecture based on OpenLayers, creating a separate vector layer for the photoelectric pointing lines. Setting a high layer level ensures that the pointing lines always appear above the map content. The system maintains a separate vector data source to manage the geometry of the pointing lines and coverage areas.

[0065] Because gesture data collection occurs over time, the system implements a precise time interpolation algorithm to determine the device's gesture at any given time. The algorithm first locates the time segment within the gesture data array that contains the target time, then performs linear interpolation between the two preceding and succeeding data points. For time points outside the data range, the system returns the nearest boundary data point.

[0066] The system calculates the spatial geometry of the pointing line in real time based on the three-dimensional posture parameters of the optoelectronic device. The calculation process includes: calculating the visible distance based on the tilt angle and device height, determining the precise direction of the pointing based on the horizontal angle, and calculating the sector boundary of the coverage range using the horizontal field of view angle (refer to Figure 5 ). The system converts these parameters into geometric shapes in a geographic coordinate system and generates vector features that can be displayed on a map.

[0067] During each rendering cycle, the system checks whether the current time point requires an update of the pointing line display. This avoids unnecessary recalculation by comparing the current time with the last drawing time. When the pose data changes, the system clears the previous graphic elements and recalculates and draws the new pointing line and coverage.

[0068] 5. Video playback controller. When a user selects a specific recording for playback, the system first parses the recording's metadata, including the time range and associated target information. The system then sets the video player's source address and initial playback position. It also extracts all target identifiers involved in the recording and constructs a playback target list.

[0069] The system initializes the target track playback module based on the recorded time information. It clears the previous track cache data, sets the track playback start time to the start time of the playback period, configures a WebSocket connection, and passes in the start time, end time, and current chart range to obtain the track data stream. The system ensures that the video player and track playback module use the same time base.

[0070] The system initiates a high-frequency synchronous rendering loop (60 frames per second) and continuously monitors the current time position of the video player. When a time change is detected, a synchronous update mechanism is immediately triggered. The system calculates the current playback time by adding the elapsed playback time to the video segment's start time. This time is then passed to the target trajectory player and optoelectronic pointing line renderer, ensuring that all visual elements remain precisely synchronized.

[0071] The system implements a unified playback control interface that supports operations such as play, pause, fast forward, rewind, and jump. Any playback control operation will simultaneously act on the video player, target trajectory player, and photoelectric pointing line renderer, ensuring that the three always maintain a consistent playback state.

[0072] 6. Map rendering engine. The system analyzes target association information in video metadata and automatically identifies maritime targets tracked by optoelectronic devices during recording. By parsing target identifiers and MMSI codes, the system highlights the target pointed at by optoelectronic devices with a red checkbox, establishing an association between the optoelectronic device and the specific target.

[0073] For identified moving targets, the system simultaneously acquires the target's historical trajectory data during the recording period. This trajectory data maintains a precise time correspondence with the photoelectric recording, ensuring that the target position and photoelectric pointing direction are accurately matched.

[0074] The system displays the optoelectronic pointing line and target trajectory simultaneously on the map interface, and highlights the relationship between the two through visual means such as color coding and line type differentiation.

[0075] 7. Backend Interface. The system implements a time-window-based intelligent caching mechanism through the backend interface. A two-hour data query window and a 30-minute cache update threshold are preset. When the current playback time approaches the cache limit, the system automatically triggers data preload to ensure smooth playback. The cache manager maintains data caches for multiple time periods and uses a least-recently-used (LRU) algorithm for cache eviction.

[0076] This embodiment, through a multi-level synchronization mechanism, intelligent caching strategy and precise geometric calculation, achieves high-precision synchronous display of optoelectronic historical video and maritime target trajectory, providing strong technical support for maritime monitoring and situation analysis.

[0077] Figure 6 This is a schematic structural diagram of an embodiment of a synchronous playback device based on multimodal data of marine targets proposed by the present invention, the device comprising: An acquisition module 601 is used to acquire video data of the movement of targets at sea collected by optoelectronic equipment; The processing module 602 is used to parse the metadata of the video data to obtain the video time information and a target list consisting of at least one sea target included in the video data; determine the start time of the trajectory playback of the sea target in the target list as the start time of the video playback in the video time information according to the video time information; obtain the trajectory data stream of the sea target within the playback time from the start time to the end time according to the end time of the video playback in the video time information; when playing the video data within the playback time, the trajectory data stream of the sea target within the playback time is played according to the start time of the trajectory playback time. Optionally, the processing module 602 is specifically configured to: The metadata of the video data is parsed to obtain the start time, end time, video file address and video detail object of the video, wherein the video detail object includes: a time series array of the photoelectric device posture data.

[0078] Optionally, the processing module 602 is specifically configured to: parsing metadata of the video data to obtain a target identifier of at least one marine target captured during the video recording process; A target list is obtained according to the target identifier of the at least one maritime target.

[0079] Optionally, the processing module 602 is specifically configured to: According to the video recording time information, the trajectory playback data in the trajectory playback cache is cleared, and the start time of the trajectory playback cache is set to the start time of the video recording playback in the video recording time information.

[0080] Optionally, the processing module 602 is specifically configured to: Determine the playback duration according to the end time of the video playback and the start time of the video playback in the video playback time information; Starting from the start time of the trajectory playback, the trajectory data stream of the sea target within the playback duration is obtained.

[0081] Optionally, the processing module 602 is specifically configured to: generating a pointing line of the optoelectronic device pointing to the marine target based on the attitude data of the optoelectronic device; According to the starting time of the trajectory playback time, the direction line and the trajectory data stream of the sea target are synchronously displayed on the chart display interface.

[0082] Optionally, the processing module 602 is specifically configured to: Extract sea targets within the playback duration through the secondary cache; Loading the trajectory data stream of the maritime target through the first-level cache and displaying it on the chart display interface; The pointing line of the photoelectric device is displayed in the form of a vector diagram on the map content of the nautical chart display interface.

[0083] It should be noted that the device is a device corresponding to the above method, and all implementation methods in the above method embodiments are applicable to the embodiments of the device and can achieve the same technical effects.

[0084] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the synchronous playback method based on multimodal data of marine targets provided in the above embodiments.

[0085] Furthermore, the logic instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0086] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform a synchronous playback method based on multimodal data of marine targets provided in the above embodiments.

[0087] The device embodiments described above are merely illustrative. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0088] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods of each embodiment or certain portions of the embodiments.

[0089] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A synchronous playback method based on multimodal data of marine targets, characterized in that: include: Acquire video data of maritime target movement collected by optoelectronic equipment; parsing metadata of the video data to obtain video recording time information and a target list consisting of at least one marine target included in the video data; According to the video time information, determining the start time of the trajectory playback of the sea target in the target list as the start time of the video playback in the video time information; According to the end time of the video playback in the video time information, a trajectory data stream of the sea target with a playback duration from the start time to the end time is obtained; When the video data within the playback time is played, the trajectory data stream of the sea target within the playback time is played back according to the starting time of the trajectory playback time.

2. The synchronous playback method based on multimodal data of marine targets according to claim 1, characterized in that: Parse the metadata of the video data to obtain the video recording time information, including: The metadata of the video data is parsed to obtain the start time, end time, video file address and video detail object of the video, wherein the video detail object includes: a time series array of the photoelectric device posture data.

3. The synchronous playback method based on multimodal data of marine targets according to claim 1, characterized in that: Parsing metadata of the video data to obtain a target list consisting of at least one marine target included in the video data includes: parsing metadata of the video data to obtain a target identifier of at least one marine target captured during the video recording process; A target list is obtained according to the target identifier of the at least one maritime target.

4. The synchronous playback method based on multimodal data of marine targets according to claim 1, characterized in that: Determining, based on the video recording time information, that the start time of the trajectory playback of the sea target in the target list is the start time of the video playback in the video recording time information comprises: According to the video recording time information, the trajectory playback data in the trajectory playback cache is cleared, and the start time of the trajectory playback cache is set to the start time of the video recording playback in the video recording time information.

5. The synchronous playback method based on multimodal data of marine targets according to claim 1, characterized in that: According to the end time of the video playback in the video time information, obtaining the trajectory data stream of the sea target with the playback time length from the start time to the end time, including: Determine the playback duration according to the end time of the video playback and the start time of the video playback in the video playback time information; Starting from the start time of the trajectory playback, the trajectory data stream of the sea target within the playback duration is obtained.

6. The synchronous playback method based on multimodal data of marine targets according to claim 2, characterized in that: When replaying the trajectory data stream of the sea target within the playback time according to the starting time of the trajectory playback time, it also includes: generating a pointing line of the optoelectronic device pointing to the marine target based on the attitude data of the optoelectronic device; According to the starting time of the trajectory playback time, the direction line and the trajectory data stream of the sea target are synchronously displayed on the chart display interface.

7. The synchronous playback method based on multimodal data of marine targets according to claim 6, characterized in that: Synchronously displaying the direction line and the trajectory data stream of the sea target on a nautical chart display interface includes: Extract sea targets within the playback duration through the secondary cache; Loading the trajectory data stream of the maritime target through the first-level cache and displaying it on the chart display interface; The pointing line of the photoelectric device is displayed in the form of a vector diagram on the map content of the nautical chart display interface.

8. A synchronous playback device based on multimodal data of marine targets, characterized in that: include: An acquisition module is used to acquire video data of the movement of targets at sea collected by optoelectronic equipment; The processing module is configured to parse metadata of the video data to obtain video time information and a target list consisting of at least one maritime target included in the video data; determine, based on the video time information, a start time for trajectory playback of the maritime targets in the target list as the start time of the video playback in the video time information; obtain, based on the end time of the video playback in the video time information, a trajectory data stream of the maritime targets for a playback duration between the start time and the end time; and, when playing video data within the playback duration, replay the trajectory data stream of the maritime targets within the playback duration according to the start time of the trajectory playback time.

9. An electronic device, characterized in that: include: A processor, a memory, and a bus, wherein the processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the steps of the method according to any one of claims 1 to 7 by calling the program instructions.

10. A computer-readable storage medium, characterized in that The device stores instructions, which, when executed on a computer, enable the computer to execute the steps of the method according to any one of claims 1 to 7.

Images