Marine scene multi-source heterogeneous data web end spatio-temporal fusion rendering system and method

Abstract

The application provides a marine scene multi-source heterogeneous data Web end space-time fusion rendering system and method, the system comprises: a unified space-time index layer, a dynamic tile request layer, a time alignment processing layer and a WebGL hybrid rendering layer; the method comprises the following steps: step one, constructing a global space-time index in space and time dimensions; step two, dynamically updating tiles according to changes in the viewport and the zoom level; step three, realizing time sequence synchronization based on the timestamp in the global space-time index; step four, fusing and drawing three types of data in the same picture; step five, in response to user zooming or dragging operations, updating the viewport range and the zoom level and jumping to step two, and simultaneously performing picture redrawing of step four based on the updated tile data. Advantageous effects: the problems of non-uniform space-time reference, data misplacement and time sequence asynchronization of multi-source heterogeneous marine data are solved, the running fluency is improved, the layer fragmentation problem is solved, and stepless drilling interaction is realized.

Description

Technical Field

[0001] This invention relates to the field of image modeling and 3D graphics rendering technology for geospatial data, and in particular to a spatiotemporal fusion rendering system and method for multi-source heterogeneous data of marine scenes on the Web. Background Technology

[0002] Marine monitoring scenarios involve the extensive deployment of sonar, video, and AIS sensors, generating massive amounts of heterogeneous data. Web-based platforms, with their cross-terminal and deployment-free advantages, have become the mainstream carrier for marine data visualization. However, the timing and spatial references of data acquisition from various sensors are independent, and traditional web visualization architectures cannot achieve integrated display of multi-source data. The industry urgently needs a dedicated rendering engine with spatiotemporal alignment, dynamic loading, and hybrid rendering capabilities.

[0003] Currently, mainstream web-based marine data visualization solutions generally adopt an independent layer overlay mode, that is, various sensor data are loaded and rendered independently in separate layers, using their own original coordinate systems and acquisition sequences; after the platform starts, a fixed layer of the entire domain is loaded at once, and layer switching depends on manual operation; at the rendering level, existing solutions usually implement two-dimensional or three-dimensional display based on HTML5 Canvas or basic WebGL interfaces.

[0004] This results in inconsistent spatiotemporal benchmarks, data misalignment, and inability to link data; after platform startup, a fixed layer is loaded across the entire domain, which consumes a lot of resources and is prone to lag during manual operation; the layers are independent and isolated from each other, and various types of data are only superimposed on the surface of the image, making it impossible to achieve in-depth fusion and display. Users find it difficult to comprehensively judge the correlation between sonar targets, video images, and AIS trajectories in the same view, and it does not support fine-grained drill-down analysis, lacking in-depth fusion and stepless drill-down capabilities. Summary of the Invention

[0005] The present invention was made in view of the above-mentioned problems, and its purpose is to provide a spatiotemporal fusion rendering system and method for multi-source heterogeneous data of marine scenes on the Web.

[0006] The objective of this invention is achieved through the following technical solution: This invention provides a web-based spatiotemporal fusion rendering system for multi-source heterogeneous data in marine scenes, comprising: a unified spatiotemporal index layer, a dynamic tile request layer, a time alignment processing layer, and a WebGL hybrid rendering layer; The unified spatiotemporal index layer is used to acquire multi-source heterogeneous data, then unify the multi-source heterogeneous data into a spatial coordinate system, and bind a unified time base timestamp to each data to construct a two-dimensional index of space and time. The dynamic tile request layer listens for changes in the viewport range and responds to zoom level adjustments. The time alignment processing layer is based on the global timestamp established by the unified spatiotemporal index layer, and uses a time-series matching algorithm to keep different sensor data and video stream data synchronized and linked when played back on the same time axis. The WebGL hybrid rendering layer is used to distinguish between three types of data in the WebGL rendering pipeline: sonar point cloud data, video stream data, and AIS trajectory data. By controlling the transparency and rendering depth of each type of data, it merges and renders multiple types of data in the same screen. It also enables stepless scaling and drag-and-drop interaction.

[0007] Furthermore: the multi-source heterogeneous data includes raw geographic coordinate data from sonar sensors and AIS sensors; The spatial coordinate system includes: a coordinate system with the monitoring platform coordinates as the origin, and a global latitude and longitude coordinate system; The coordinate system with the monitoring platform coordinates as the origin is constructed by a self-developed coordinate transformation matrix; The global latitude and longitude coordinate system is activated when the self-developed coordinate transformation matrix is ​​unusable because the sensor cannot transmit its own coordinates in real time. By calling a third-party geographic information service interface, the latitude and longitude coordinates of the sensor are directly mapped to the global coordinate system, ensuring the system's data spatial alignment capability when the platform coordinate system is unavailable.

[0008] Furthermore, the self-developed coordinate transformation matrix is ​​characterized by the following: For different sensor data types, corresponding coordinate transformation formulas are used: For sonar point cloud data, a three-dimensional rigid body transformation matrix is ​​used; For AIS trajectory data, a two-dimensional projection transformation matrix is ​​used.

[0009] Furthermore, the data scheduling mode of the dynamic tile request layer is as follows: Mode 1: Dynamically request tile scheduling based on viewport and zoom level. Specifically, when the view zoom level is high, request high-resolution tiles; when the view zoom level is low, request low-resolution overview tiles; and when the viewport moves, automatically request tiles that newly enter the viewport range and release tiles that leave the viewport range. Mode 2 employs preloading, incremental updates, and asynchronous batch loading, specifically as follows: Preloading: Based on the current viewport movement trend, predict the tile area that will soon enter the viewport range, send a request to the server in advance and cache the tile data in the browser memory. When the viewport actually moves to the area, it is directly read from the cache for rendering, eliminating tile loading delay. Incremental update: When the viewport moves, only the tile data newly entering the viewport is requested, the loaded tiles that are still in the viewport are reused, and the memory occupied by the tiles that leave the viewport range is released to avoid full reload; Asynchronous loading in batches: The tile request queue is divided into multiple batches according to spatial priority. Within each batch, tiles are requested in parallel using Promise.all. The execution is scheduled serially between batches to control the number of concurrent requests in the browser to be less than or equal to 6, so as to avoid the main thread rendering being blocked due to concurrency overload.

[0010] Furthermore, the global matching algorithm used in the time alignment processing layer includes at least one time-series matching algorithm.

[0011] Furthermore, the temporal matching algorithm includes: an interpolation algorithm and a nearest neighbor matching algorithm; The interpolation algorithm is used to fill in missing nodes in the time axis of video stream data and AIS trajectory data; The nearest neighbor matching algorithm is used to match sonar point cloud data to the closest global timestamp.

[0012] Furthermore: In the WebGL hybrid rendering layer, the method for controlling transparency is: to control transparency by adjusting the alpha channel value to achieve semi-transparent overlay; The method for controlling the rendering depth is to determine the foreground and background occlusion relationships of various data types in three-dimensional space by adjusting the Z-buffer value.

[0013] Further: The stepless scaling is: in response to the user's scaling operation, dynamically changing the screen scaling level and triggering the dynamic tile request layer to update the tile data of the corresponding precision; The drag-and-drill interaction is as follows: in response to the user's drag operation, the viewport space range is changed and the dynamic tile request layer is triggered to update the tile data of the corresponding area. At the same time, the WebGL hybrid rendering layer redraws the surface in real time based on the updated tile data.

[0014] This invention also provides a web-based spatiotemporal fusion rendering method for multi-source heterogeneous data of marine scenes, applied to the aforementioned web-based spatiotemporal fusion rendering system for multi-source heterogeneous data of marine scenes, comprising the following steps: Step 1: Acquire multi-source heterogeneous data from sonar, video, and AIS sensors, unify the multi-source heterogeneous data into the same spatial coordinate system, unify the timestamp of the time base, and construct a global spatiotemporal index with both spatial and temporal dimensions. Step 2: Monitor the browser viewport range and screen zoom level, request data tiles of the corresponding spatial range and resolution from the server, and dynamically update the tiles according to the changes in viewport and zoom level. Step 3: Based on the timestamps in the global spatiotemporal index, interpolation algorithms are used to complete the time-series nodes of the video stream data and AIS trajectory data, and the nearest neighbor matching algorithm is used to match the time-series data of the sonar point cloud data to achieve time-series synchronization. Step 4: In the WebGL rendering pipeline, identify sonar point cloud data, video stream data, and AIS trajectory data, and blend the three types of data into the same image by adjusting the transparency and rendering depth values. Step 5: In response to user zoom or drag operations, update the viewport range and zoom level and jump to Step 2. At the same time, perform screen redrawing in Step 4 based on the updated tile data.

[0015] In summary, the beneficial effects of the present invention are as follows: 1. Solve the problems of inconsistent spatiotemporal benchmarks, data misalignment, and time synchronization issues in multi-source heterogeneous marine data.

[0016] 2. Dynamic tile loading significantly reduces browser load and improves smoothness of operation.

[0017] 3. By leveraging WebGL deep blending rendering, the problem of layer fragmentation is solved, and with the help of stepless drill-down interaction, the data visualization effect and comprehensive analysis capability are greatly improved. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this drawing or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this drawing. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0019] Figure 1 This is a system flowchart of the present invention.

[0020] The purpose, features, and advantages of this accompanying drawing will be further explained in conjunction with the embodiments and with reference to the accompanying drawing. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments provided by this invention without inventive effort are within the scope of protection of this invention.

[0022] Obviously, the accompanying drawings described below are merely some examples or embodiments of the present invention. Those skilled in the art can apply the present invention to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this invention, modifications to design, manufacturing, or production based on the technical content disclosed in this invention are merely conventional technical means and should not be construed as insufficient disclosure of the present invention.

[0023] Unless otherwise specified, all embodiments and optional embodiments of the present invention can be combined with each other to form new technical solutions.

[0024] Unless otherwise specified, all technical features and optional technical features of this invention can be combined to form new technical solutions.

[0025] Unless otherwise specified, all steps of the present invention may be performed sequentially or randomly, preferably sequentially. For example, the method includes steps (a) and (b), indicating that the method may include steps (a) and (b) performed sequentially, or it may include steps (b) and (a) performed sequentially. For example, the mention that the method may also include step (c) indicates that step (c) may be added to the method in any order. For example, the method may include steps (a), (b), and (c), or it may include steps (a), (c), and (b), or it may include steps (c), (a), and (b), etc.

[0026] Unless otherwise specified, the terms "comprising" and "including" as used in this invention can be open-ended or closed-ended. For example, "comprising" and "including" can mean that other components not listed may also be included, or that only the listed components may be included.

[0027] Unless otherwise specified, the term "or" is inclusive in this invention. For example, the phrase "A or B" means "A, B, or both A and B". More specifically, the condition "A or B" is satisfied by any of the following conditions: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); or both A and B are true (or exist).

[0028] Please see Figure 1 The system flowchart of this invention illustrates its working principle as follows: Collect multi-source heterogeneous data, specifically the raw geographic coordinate data of sonar sensors and AIS sensors, as well as video stream data (with the deployment location and field of view parameters of the acquired cameras). Multi-source heterogeneous data is input into a unified spatiotemporal index layer for processing. A coordinate transformation matrix is ​​used to unify the original geographic coordinate data of sonar sensors and AIS sensors into a single spatial coordinate system. For video stream data, the known deployment locations of the acquisition cameras are used as spatial anchor points. The geographic locations of the cameras are unified into the same spatial coordinate system through a coordinate transformation matrix. The video frame images are projected and mapped to the corresponding geographic areas according to the camera field of view (FOV) parameters. The video frames themselves do not need to undergo pixel-by-pixel coordinate transformation. A unified time base timestamp is then bound to each data point to construct a two-dimensional index of space and time. The data processed by the unified spatiotemporal index layer is passed to the dynamic tile request layer. The dynamic tile request layer listens to the browser viewport range and screen zoom level in real time, requests data tiles with corresponding spatial range and resolution from the server, and dynamically updates the tiles according to the changes in viewport and zoom level. The tile data output by the dynamic tile request layer enters the time alignment processing layer. Based on the timestamp in the global spatiotemporal index, the interpolation algorithm is used to complete the time sequence nodes of the video stream data and AIS trajectory data, and the nearest neighbor matching algorithm is used to match the time sequence data of the sonar point cloud data to achieve time synchronization. The time-synchronized data output from the time alignment processing layer enters the WebGL hybrid rendering layer. In the WebGL rendering pipeline, sonar point cloud data, video stream data, and AIS trajectory data are identified. By adjusting the transparency and rendering depth values, the three types of data are merged and drawn into the same screen. At the same time, in response to user zooming or dragging operations, the viewport range and zoom level are updated, and the dynamic tile request layer is triggered to update the tile data. Based on the updated tile data, the screen is redrawn, realizing stepless zooming and drag-and-drill interaction.

[0029] There are two methods to unify the raw geographic coordinate data of sonar sensors and AIS sensors into a spatial coordinate system. One is a more accurate method, which uses the monitoring platform as the origin of the spatial coordinate system. The other is a backup method for sensors that cannot transmit their own coordinates in real time. The first method uses a self-developed coordinate transformation matrix, specifically: For the sonar point cloud data transmitted back by the sonar sensor, the three-dimensional rigid body transformation matrix M1=[R 3x3 |T 3x1 ], where R 3x3 T is a 3×3 rotation matrix determined by the sensor mounting azimuth angles (α, β, γ). 3x1 The translation vector is determined by the sensor's relative installation offset (Δx, Δy, Δz) from the monitoring platform. For the AIS trajectory data transmitted back by the AIS sensor, a two-dimensional projection transformation matrix M2=[R 2x2 |T 2x1The latitude and longitude coordinates are converted into planar projected coordinates with the monitoring platform as the origin, where R 2x2 T is a 2×2 rotation and scaling matrix. 2x1 It is a 2×1 translation vector; The second method is to directly call a third-party geographic information service interface to map the sensor's latitude and longitude coordinates directly to the global coordinate system (matching the geographical location recorded when the sensor was first placed, not real-time transmission), ensuring the system's data spatial alignment capability when the platform coordinate system is unavailable.

[0030] The data scheduling modes for the dynamic tile request layer are as follows: Mode 1: Dynamically request tile scheduling based on viewport and zoom level. Specifically, when the view zoom level is high, request high-resolution tiles; when the view zoom level is low, request low-resolution overview tiles; and when the viewport moves, automatically request tiles that newly enter the viewport range and release tiles that leave the viewport range. Mode 2 employs preloading, incremental updates, and asynchronous batch loading, specifically as follows: Preloading: Based on the current viewport movement trend (including drag direction and speed), predict the tile area that will soon enter the viewport range, send a request to the server in advance and cache the tile data in the browser memory. When the viewport actually moves to the area, it is directly read from the cache for rendering, eliminating tile loading delay. Incremental update: When the viewport moves, only the tile data newly entering the viewport is requested, the loaded tiles that are still in the viewport are reused, and the memory occupied by the tiles that leave the viewport range is released to avoid full reload; Asynchronous loading in batches: The tile request queue is divided into multiple batches according to spatial priority (the center area of ​​the viewport takes precedence over the edge areas). Within each batch, tiles are requested in parallel using Promise.all, and the execution is scheduled serially between batches. This controls the number of concurrent requests in the browser to be less than or equal to 6, avoiding main thread rendering blockage due to concurrency overload.

[0031] The time alignment processing layer completes time synchronization based on global second-level timestamps. It typically uses time matching algorithms to complete or align the received data. This application uses interpolation algorithms to complete time nodes for continuous video stream data and AIS trajectory data (video frame sequences and AIS trajectory points); for discrete sonar point cloud data, it uses nearest neighbor matching algorithms to match time-series data to ensure that all data are synchronized during timeline playback.

[0032] In the WebGL hybrid rendering layer, semi-transparent overlay is controlled by adjusting the alpha channel value, and the foreground and background occlusion relationships of various data types in 3D space are determined by adjusting the Z-buffer value. Both the alpha channel value adjustment and the Z-buffer value adjustment adopt an adaptive adjustment mechanism. Specifically: The adaptive adjustment logic of the Alpha channel is as follows: Based on the spatial density distribution of data points of each type in the current viewport, the transparency weight coefficient of each type of data is dynamically calculated. When the number of data points of a certain type in a unit viewport area exceeds the preset density threshold, the global Alpha value of that type of data is automatically reduced (transparency is increased) to reduce its occlusion effect and ensure that other types of data can be observed at the same time. The Z-buffer adaptive adjustment logic is as follows: based on the data type of the user's current interactive focus (determined by hovering the mouse or clicking), the rendering depth priority of the focused data type is dynamically increased, and its Z-buffer offset value is moved forward by a preset depth increment, so that the interactive focus data is always rendered in front of the visual data of other data types, while the other data types remain unchanged according to the default depth sorting rules.

[0033] The WebGL hybrid rendering layer's stepless scaling is as follows: in response to user zooming operations, it dynamically changes the image zoom level and triggers the dynamic tile request layer to update the tile data of the corresponding precision. The drag-and-drop interaction is as follows: In response to the user's drag operation, the viewport space range is changed and the dynamic tile request layer is triggered to update the tile data of the corresponding area. At the same time, the WebGL hybrid rendering layer redraws the surface in real time based on the updated tile data.

[0034] It should be noted that the present invention is not limited to the above-described embodiments. The above embodiments are merely examples, and any embodiments that have the same structure and perform the same effects as the technical concept within the scope of the present invention are included within the scope of the present invention. Furthermore, various modifications that can be conceived by those skilled in the art to the embodiments, and other ways of constructing by combining some of the constituent elements of the embodiments, without departing from the spirit of the present invention, are also included within the scope of the present invention.

Claims

1. A web-based spatiotemporal fusion rendering system for multi-source heterogeneous data in marine scenes, characterized in that: include: Unified spatiotemporal indexing layer, dynamic tile request layer, time alignment processing layer, and WebGL hybrid rendering layer; The unified spatiotemporal index layer is used to acquire multi-source heterogeneous data, then unify the multi-source heterogeneous data into a spatial coordinate system, and bind a unified time base timestamp to each data to construct a two-dimensional index of space and time. The dynamic tile request layer listens for changes in the viewport range and responds to zoom level adjustments. The time alignment processing layer uses a time-series matching algorithm to ensure that data from different sensors and video streams remain synchronized and linked when played back on the same timeline, based on the global timestamp established by the unified spatiotemporal index layer. The WebGL hybrid rendering layer is used to distinguish between three types of data in the WebGL rendering pipeline: sonar point cloud data, video stream data, and AIS trajectory data. By controlling the transparency and rendering depth of each type of data, it merges and renders multiple types of data in the same screen. It also enables stepless scaling and drag-and-drop interaction.

2. The marine scene multi-source heterogeneous data web-based spatiotemporal fusion rendering system according to claim 1, characterized in that, The multi-source heterogeneous data includes raw geographic coordinate data from sonar sensors and AIS sensors; The spatial coordinate system includes: a coordinate system with the monitoring platform coordinates as the origin, and a global latitude and longitude coordinate system; The coordinate system with the monitoring platform coordinates as the origin is constructed by a self-developed coordinate transformation matrix; The global latitude and longitude coordinate system is activated when the self-developed coordinate transformation matrix is ​​unusable because the sensor cannot transmit its own coordinates in real time. The sensor's latitude and longitude coordinates are directly mapped to the global coordinate system by calling a third-party geographic information service interface.

3. The marine scene multi-source heterogeneous data web-based spatiotemporal fusion rendering system according to claim 2, characterized in that, The self-developed coordinate transformation matrix is ​​as follows: For different sensor data types, corresponding coordinate transformation formulas are used: For sonar point cloud data, a three-dimensional rigid body transformation matrix is ​​used; For AIS trajectory data, a two-dimensional projection transformation matrix is ​​used.

4. The marine scene multi-source heterogeneous data web-based spatiotemporal fusion rendering system according to claim 1, characterized in that, The data scheduling modes of the dynamic tile request layer are as follows: Mode 1: Dynamically request tile scheduling based on viewport and zoom level. Specifically, when the view zoom level is high, request high-resolution tiles; when the view zoom level is low, request low-resolution overview tiles; and when the viewport moves, automatically request tiles that newly enter the viewport range and release tiles that leave the viewport range. Mode 2 employs preloading, incremental updates, and asynchronous batch loading, specifically as follows: Preloading: Based on the current viewport movement trend, predict the tile area that will soon enter the viewport range, send a request to the server in advance and cache the tile data in the browser memory. When the viewport actually moves to the area, it is directly read from the cache for rendering, eliminating tile loading delay. Incremental update: When the viewport moves, only the tile data newly entering the viewport is requested, the loaded tiles that are still in the viewport are reused, and the memory occupied by the tiles that leave the viewport range is released to avoid full reload; Asynchronous loading in batches: The tile request queue is divided into multiple batches according to spatial priority. Within each batch, tiles are requested in parallel using Promise.all, and the execution is scheduled serially between batches, controlling the number of concurrent requests in the browser to be less than or equal to 6.

5. The marine scene multi-source heterogeneous data web-based spatiotemporal fusion rendering system according to claim 1, characterized in that, The global matching algorithm used in the time alignment processing layer includes at least one time-series matching algorithm.

6. The marine scene multi-source heterogeneous data web-based spatiotemporal fusion rendering system according to claim 5, characterized in that, The time-series matching algorithm includes: interpolation algorithm and nearest neighbor matching algorithm; The interpolation algorithm is used to fill in missing nodes in the time axis of video stream data and AIS trajectory data; The nearest neighbor matching algorithm is used to match sonar point cloud data to the closest global timestamp.

7. The marine scene multi-source heterogeneous data web-based spatiotemporal fusion rendering system according to claim 1, characterized in that, In the WebGL hybrid rendering layer, the method for controlling transparency is: to control transparency by adjusting the Alpha channel value to achieve semi-transparent overlay; The method for controlling the rendering depth is to determine the foreground and background occlusion relationships of various types of data in three-dimensional space by adjusting the Z-buffer value.

8. The marine scene multi-source heterogeneous data web-based spatiotemporal fusion rendering system according to claim 1, characterized in that, The stepless scaling refers to: responding to the user's scaling operation, dynamically changing the screen scaling level and triggering the dynamic tile request layer to update the tile data of the corresponding precision; The drag-and-drill interaction is as follows: in response to the user's drag operation, the viewport space range is changed and the dynamic tile request layer is triggered to update the tile data of the corresponding area. At the same time, the WebGL hybrid rendering layer redraws the surface in real time based on the updated tile data.

9. A web-based spatiotemporal fusion interactive rendering method for multi-source heterogeneous marine sensor data, applied to the web-based spatiotemporal fusion rendering system for multi-source heterogeneous marine scene data as described in claims 1 to 8, characterized in that, Includes the following steps: Step 1: Acquire multi-source heterogeneous data from sonar, video, and AIS sensors, unify the multi-source heterogeneous data into the same spatial coordinate system, unify the timestamp of the time base, and construct a global spatiotemporal index with both spatial and temporal dimensions. Step 2: Monitor the browser viewport range and screen zoom level, request data tiles of the corresponding spatial range and resolution from the server, and dynamically update the tiles according to the changes in viewport and zoom level. Step 3: Based on the timestamps in the global spatiotemporal index, interpolation algorithms are used to complete the time-series nodes of the video stream data and AIS trajectory data, and the nearest neighbor matching algorithm is used to match the time-series data of the sonar point cloud data to achieve time-series synchronization. Step 4: In the WebGL rendering pipeline, identify sonar point cloud data, video stream data, and AIS trajectory data, and blend the three types of data into the same image by adjusting the transparency and rendering depth values. Step 5: In response to user zoom or drag operations, update the viewport range and zoom level and jump to Step 2. At the same time, perform screen redrawing in Step 4 based on the updated tile data.

Images