A spatiotemporal data volume rendering method for unreal engine

Abstract

This invention provides a spatiotemporal data volume rendering method for Unreal Engine, belonging to the field of spatiotemporal data volume rendering technology. This invention acquires NetCDF format ocean spatiotemporal data and generates two-dimensional volumetric textures. It establishes a memory partitioning management system to divide GPU memory into upper and lower memory grids, employs a two-layer game theory algorithm to achieve differentiated resource allocation, creates volumetric rendering material objects in Unreal Engine, and establishes a volumetric shading framework based on ray stepping. It dynamically adjusts the sampling step size through an adaptive ray stepping algorithm to achieve efficient volumetric texture sampling based on the memory grid, performs ray casting calculations to generate three-dimensional visualization effects, and establishes dynamic interactive control in the spatiotemporal dimension to achieve synchronous display of multi-source ocean data. This invention solves the technical problem of low rendering efficiency caused by unreasonable allocation of memory resources during the three-dimensional visualization rendering of ocean spatiotemporal data.

Description

Technical Field

[0001] This invention belongs to the field of spatiotemporal data volume rendering technology, and more specifically, relates to a spatiotemporal data volume rendering method for Unreal Engine. Background Technology

[0002] 3D visualization rendering technology for ocean spatiotemporal data is an important technical means for marine scientific research and marine environmental monitoring. Traditional volumetric rendering technology uses ray-stepping algorithms to sample and render 3D scalar fields, enabling visualization of the spatial distribution of environmental parameters such as ocean temperature, salinity, and current velocity. In current ocean data visualization applications, graphics rendering engines such as Unreal Engine are widely used to build high-quality 3D visualization systems. Through volumetric rendering materials and ray casting techniques, NetCDF format ocean observation data is converted into intuitive 3D images, providing oceanographers and environmental monitoring personnel with effective data analysis tools. However, traditional volumetric rendering technology faces the problem of inadequate memory resource management when processing large-scale ocean spatiotemporal data. Existing methods typically employ a uniform memory allocation strategy, failing to fully consider the differences in data density, access frequency, and processing complexity between surface and deep ocean data, resulting in low memory utilization and decreased rendering performance. In existing technologies, due to the lack of an adaptive memory management mechanism tailored to the characteristics of ocean spatiotemporal data, traditional volumetric rendering systems cannot effectively handle the different computational needs of surface and deep ocean data, resulting in unbalanced memory resource allocation and consequently affecting overall rendering efficiency and visual quality. In other words, existing technologies suffer from low rendering efficiency due to unreasonable allocation of video memory resources during the 3D visualization rendering of marine spatiotemporal data. Summary of the Invention

[0003] In view of this, the present invention provides a spatiotemporal data volume rendering method for Unreal Engine, which can solve the technical problem of low rendering efficiency caused by unreasonable allocation of video memory resources in the process of 3D visualization rendering of marine spatiotemporal data in the prior art.

[0004] This invention is implemented as follows: It provides a spatiotemporal data volume rendering method for Unreal Engine, including acquiring ocean spatiotemporal data based on NetCDF format and generating two-dimensional volumetric textures; establishing a memory partitioning management system, dividing GPU memory into upper and lower memory grids, with the upper grid processing surface ocean data and the lower grid processing deep ocean data; employing a two-layer game theory algorithm for memory grid hierarchical partitioning and using a memory allocation game model to calculate the optimal allocation scheme; creating volumetric rendering material objects in Unreal Engine and establishing a volumetric shading framework based on ray stepping; establishing an adaptive ray stepping algorithm and using a spatiotemporal coherence optimization function to calculate the dynamic step size; implementing volumetric texture sampling based on the memory grid and optimizing texture sampling using an ocean data adaptive prediction model; performing ray casting calculations for volumetric rendering; establishing dynamic interactive control in the spatiotemporal dimension to achieve synchronous visualization of multi-source ocean data; and achieving differentiated configuration of memory resources by establishing a hierarchical memory management mechanism based on ocean data feature differences and using a two-layer game theory algorithm to accurately capture the differences in data density, access frequency, and processing complexity between surface and deep ocean data.

[0005] Specifically, the step of generating a two-dimensional volumetric texture involves determining the number of layers in the ocean spatiotemporal data in the depth direction, extracting the depth index array, traversing the original ocean spatiotemporal dataset in depth order, stitching together the two-dimensional ocean environment data matrix corresponding to each depth layer in row-first-column order, calculating the number of rows and columns of the stitched matrix, filling null values ​​in areas exceeding the depth layer index value, and normalizing the stitched complete data matrix to generate a two-dimensional volumetric texture.

[0006] Specifically, the step of establishing a video memory partition management system involves dividing the GPU video memory into multiple video memory grids, with each video memory grid used for streaming processing of individual ocean spatiotemporal data, and dividing the video memory grid into two layers: an upper video memory grid and a lower video memory grid.

[0007] Specifically, the step of establishing a volume shading framework based on ray stepping involves setting the blending mode to semi-transparent, setting the shading mode to no lighting, calculating the intersection point of the ray emitted from the camera with the bounding box of the volume rendering object, obtaining the coordinates of the maximum and minimum vertices of the bounding box, calculating the minimum and maximum intersection points of the ray and the bounding box, and determining the effective intersection state of the ray and the bounding box through scene depth control.

[0008] Specifically, the adaptive ray stepping algorithm calculates the nearest frame number and the current intra-frame phase based on the current ray position. The total number of frames is equal to the square of the number of rows and columns of the splicing matrix. A large step size is used in areas with low data density, and a small step size is used in areas with high data density. Spatial jump technology is used to skip null value areas to improve rendering efficiency.

[0009] Specifically, the volumetric texture sampling step based on the memory grid involves calculating the corresponding texture coordinates using the horizontal and vertical coordinate components of the current position of the light ray, obtaining the current frame sampling value and the next frame sampling value, and performing linear interpolation calculation based on the phase within the current frame to obtain the final sampling value of the current sampling point of the light ray. This process accelerates the sampling and processing of large-scale ocean spatiotemporal data through a parallel computing shader.

[0010] Specifically, the light projection calculation step involves setting the light to start from the camera position and move forward along the direction vector according to a dynamic step size. At each step position, data values ​​are obtained through volumetric texture sampling. The maximum sampling value of each light ray during the light stepping process is accumulated. Based on the sampling data, the corresponding color value is obtained from a preset linear color mapping curve to achieve the final three-dimensional volumetric rendering visualization effect.

[0011] Specifically, the dynamic interactive control steps in the spatiotemporal dimension are achieved by dynamically updating the texture parameter object to realize the change effect of volume rendering in the time dimension. The coordinates of the maximum and minimum vertices of the bounding box are set as dynamic parameters, and the spatial clipping range of volume rendering is adjusted by controlling the corresponding parameter values ​​in real time. This allows users to observe the cross-sectional effect of the internal structure of ocean spatiotemporal data through interactive operations.

[0012] Specifically, the spatiotemporal coherence optimization function is used to calculate the optimal step size based on the data density characteristics of the current ray position. The inputs include the current sampling point data value, the gradient change rate of adjacent sampling points, the ray direction vector, the historical step size sequence, and the data density distribution characteristics. The output is the dynamically adjusted ray step size value.

[0013] Specifically, the memory allocation game model includes an upper-level game model that aims to minimize the computation latency of the upper-level memory grid and a lower-level game model that aims to maximize the processing throughput of the lower-level memory grid. The upper-level game model has an upper-level objective function and upper-level constraints, and the lower-level game model has a lower-level objective function and lower-level constraints. The upper-level objective function and the lower-level objective function have memory allocation coupling terms.

[0014] The upper-level objective function is specifically used to calculate the optimal allocation strategy of the upper-level memory grid. The inputs include the scale of ocean surface data, memory bandwidth utilization, data transmission latency, processor core utilization, and memory fragmentation degree. The output is the upper-level memory grid allocation weight value.

[0015] The lower-level objective function is specifically used to calculate the optimal allocation strategy of the lower-level video memory grid. The inputs include the complexity of deep ocean data, the number of parallel processing threads, the data access frequency, the video memory read / write speed, and the cache hit rate. The output is the weight value of the lower-level video memory grid allocation.

[0016] Specifically, the ocean data adaptive prediction model is a spatiotemporal sequence prediction network based on the Transformer-XL architecture, which includes a multi-head self-attention mechanism, a position encoding layer, a feedforward neural network layer, and a memory fragment storage module. The ocean data adaptive prediction model controls the perception range of the spatiotemporal context through the memory length parameter. The number of memory fragments is dynamically determined based on three parameters: the spatial resolution of the current rendering area, the time series length, and the complexity of data changes.

[0017] Specifically, the steps for establishing the training dataset for the ocean data adaptive prediction model include collecting global ocean observation data and numerical model output data, standardizing them according to a spatiotemporal grid, extracting multi-scale spatiotemporal feature sequences, constructing a mapping relationship between the input sequence and the target prediction value, expanding the diversity of training samples through data augmentation techniques, and establishing a comprehensive training dataset containing different sea areas, seasons, and depth levels.

[0018] Specifically, the training steps of the ocean data adaptive prediction model involve a hierarchical training strategy. First, pre-training is performed on large-scale ocean reanalysis data to learn the basic spatiotemporal patterns of ocean data. Then, fine-tuning training is performed on regional high-resolution observation data to optimize the prediction ability of the ocean data adaptive prediction model for local ocean phenomena. Gradient pruning and learning rate decay strategies are used during the training process to prevent overfitting. Cross-validation is used to evaluate the prediction accuracy of the ocean data adaptive prediction model at different spatiotemporal scales.

[0019] Specifically, the memory segment adjustment function is used to dynamically adjust the memory length parameter of the ocean data adaptive prediction model according to rendering requirements. The memory segment adjustment function is calculated based on four data points: spatial resolution, time series length, data variation complexity, and rendering accuracy requirements, to obtain a memory weight value. When the memory weight value is in the range of 0 to 0.3, it is used to reduce computational overhead by adopting a short-term memory mode. When the memory weight value is in the range of 0.3 to 0.7, it is used to balance accuracy and efficiency by adopting a medium-term memory mode. When the memory weight value is in the range of 0.7 to 1.0, it is used to maximize prediction accuracy by adopting a long-term memory mode and adjusting the number of memory segments of the ocean data adaptive prediction model.

[0020] This invention employs a memory partitioning management system and a two-layer game theory algorithm for hierarchical memory grid partitioning. This rationally divides GPU memory into upper and lower memory grids, optimizing the processing of surface and deep ocean data respectively, effectively solving the problem of improper memory resource allocation in traditional methods. By constructing a memory allocation game theory model, aiming to minimize the computational latency of the upper memory grid and maximize the processing throughput of the lower memory grid, this invention establishes a memory resource optimization allocation mechanism that considers the differences in ocean data characteristics. This overcomes the uneven resource utilization caused by the uniform allocation strategy in existing technologies, significantly improving memory bandwidth utilization and data processing efficiency. Furthermore, by combining an adaptive ray stepping algorithm and memory grid-based volumetric texture sampling technology, this invention establishes a complete 3D visualization rendering workflow for ocean spatiotemporal data, fundamentally solving the technical problem of low rendering efficiency caused by unreasonable memory resource allocation in the 3D visualization rendering process of ocean spatiotemporal data, as mentioned in the background technology. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention.

[0022] Figure 2 This is a schematic diagram of the network structure of the adaptive prediction model for marine data involved in the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0024] like Figure 1 The diagram shown is a flowchart of a spatiotemporal data volume rendering method for Unreal Engine provided by this invention. This method includes the following steps:

[0025] S01. Obtain ocean spatiotemporal data based on NetCDF format, determine the number of layers in the ocean spatiotemporal data in the depth direction, extract the depth index array, traverse the original ocean spatiotemporal dataset in depth order, stitch the two-dimensional ocean environment data matrix corresponding to each depth layer in row-first-column order, calculate the number of rows and columns of the stitched matrix, fill the region exceeding the depth layer index value with null values, and normalize the stitched complete data matrix to generate a two-dimensional volumetric texture.

[0026] S02. Establish a video memory partitioning management system to divide the GPU video memory into multiple video memory grids. Each video memory grid is used for streaming processing of individual ocean spatiotemporal data. The video memory grid is divided into two layers: an upper video memory grid and a lower video memory grid. The upper video memory grid processes ocean surface data, and the lower video memory grid processes ocean deep data. A two-layer game algorithm is used to divide the video memory grid into layers. The optimal allocation scheme of the upper and lower video memory grids is calculated using a video memory allocation game model.

[0027] S03. Create a volumetric rendering material object in Unreal Engine, set the blending mode to semi-transparent, set the shading mode to no lighting, establish a volumetric shading framework based on ray stepping, calculate the intersection point of the ray emitted from the camera and the bounding box of the volumetric rendering object, obtain the coordinates of the maximum vertex and the minimum vertex of the bounding box, calculate the minimum and maximum intersection points of the ray and the bounding box, and determine the effective intersection state of the ray and the bounding box through scene depth control.

[0028] S04. Establish an adaptive ray stepping algorithm. Calculate the nearest frame number and the current intra-frame phase based on the current ray position. The total number of frames is equal to the square of the number of rows and columns of the splicing matrix. Calculate the dynamic step size of the ray stepping using the spatiotemporal coherence optimization function. Use a large step size in areas with low data density and a small step size in areas with high data density. Improve rendering efficiency by skipping null value areas using spatial jump technology.

[0029] S05. Implement volumetric texture sampling based on memory grid. Calculate the corresponding texture coordinates using the horizontal and vertical coordinate components of the current position of the light source. Obtain the current frame sampling value and the next frame sampling value. Perform linear interpolation based on the phase within the current frame to obtain the final sampling value of the current sampling point of the light source. Use an adaptive prediction model for ocean data to optimize the accuracy and efficiency of texture sampling. Accelerate the sampling and processing of large-scale ocean spatiotemporal data through a parallel computing shader.

[0030] S06. Perform ray casting calculation for volumetric rendering. Set the ray to start from the camera position and move forward along the movement direction vector according to the dynamic step size. At each step position, obtain data values ​​through volumetric texture sampling. Accumulate the maximum sampling value of each ray during the ray stepping process. Obtain the corresponding color value from the preset linear color mapping curve based on the sampling data to achieve the final 3D volumetric rendering visualization effect.

[0031] S07. Establish dynamic interactive control in the spatiotemporal dimension. Realize the change effect of volume rendering in the time dimension by dynamically updating the texture parameter object. Set the coordinates of the maximum vertex and the minimum vertex of the bounding box as dynamic parameters. Adjust the spatial clipping range of volume rendering by controlling the corresponding parameter values ​​in real time. Support users to observe the profile effect of the internal structure of ocean spatiotemporal data through interactive operation and realize the synchronous visualization of multi-source ocean data.

[0032] The spatiotemporal coherence optimization function is used to calculate the optimal step size based on the data density characteristics of the current ray position. The inputs include the current sampling point data value, the gradient change rate of adjacent sampling points, the ray direction vector, the historical step size sequence, and the data density distribution characteristics. The output is the dynamically adjusted ray step size value.

[0033] The memory allocation game model includes an upper-level game model that aims to minimize the computation latency of the upper-level memory grid and a lower-level game model that aims to maximize the processing throughput of the lower-level memory grid. The upper-level game model has an upper-level objective function and upper-level constraints, and the lower-level game model has a lower-level objective function and lower-level constraints. The upper-level objective function and the lower-level objective function have memory allocation coupling terms.

[0034] The upper-level objective function is used to calculate the optimal allocation strategy of the upper-level memory grid. The inputs include the scale of ocean surface data, memory bandwidth utilization, data transmission latency, processor core utilization, and memory fragmentation degree. The output is the upper-level memory grid allocation weight value.

[0035] The lower-level objective function is used to calculate the optimal allocation strategy of the lower-level memory grid. The inputs include the complexity of deep ocean data, the number of parallel processing threads, the data access frequency, the memory read / write speed, and the cache hit rate. The output is the lower-level memory grid allocation weight value.

[0036] like Figure 2As shown, the specific structure of the ocean data adaptive prediction model is a spatiotemporal sequence prediction network based on the Transformer-XL architecture, including a multi-head self-attention mechanism, a position encoding layer, a feedforward neural network layer, and a memory fragment storage module. The ocean data adaptive prediction model controls the perception range of the spatiotemporal context through the memory length parameter. The number of memory fragments is dynamically determined based on three parameters: the spatial resolution of the current rendering area, the length of the time series, and the complexity of data changes. The training dataset establishment steps of the ocean data adaptive prediction model specifically include collecting global ocean observation data and numerical model output data, standardizing them according to the spatiotemporal grid, and extracting multi-scale spatiotemporal feature sequences. A mapping relationship between input sequences and target predicted values ​​is constructed. Data augmentation techniques are used to expand the diversity of training samples, and a comprehensive training dataset containing different sea areas, seasons, and depths is established. The training steps of the ocean data adaptive prediction model specifically include: first, pre-training on large-scale ocean reanalysis data using a hierarchical training strategy to learn the basic spatiotemporal patterns of ocean data; then, fine-tuning training on regional high-resolution observation data to optimize the prediction ability of the ocean data adaptive prediction model for local ocean phenomena; gradient pruning and learning rate decay strategies are used during training to prevent overfitting; and cross-validation is used to evaluate the prediction accuracy of the ocean data adaptive prediction model at different spatiotemporal scales.

[0037] The memory segment adjustment function is used to dynamically adjust the memory length parameter of the ocean data adaptive prediction model according to rendering requirements. The memory segment adjustment function is calculated based on four data points: spatial resolution, time series length, data variation complexity, and rendering accuracy requirements, to obtain a memory weight value. When the memory weight value is in the range of 0 to 0.3, it is used to reduce computational overhead by adopting short-term memory mode. When the memory weight value is in the range of 0.3 to 0.7, it is used to balance accuracy and efficiency by adopting medium-term memory mode. When the memory weight value is in the range of 0.7 to 1.0, it is used to maximize prediction accuracy by adopting long-term memory mode and adjusting the number of memory segments of the ocean data adaptive prediction model.

[0038] The NetCDF format is a general-purpose network data format, an array-oriented scientific data storage format, widely used for the storage and exchange of oceanographic, meteorological, and climate data.

[0039] The depth layers represent a vertically layered structure of ocean data, with each layer containing corresponding ocean environmental parameters. The depth index array is a sequence of indices arranged in depth order, used to identify the location information of different depth layers. The two-dimensional ocean environmental data matrix is ​​a planar data structure corresponding to a single depth layer, containing ocean environmental parameters such as temperature, salinity, and current velocity. The stitching matrix is ​​a complete data structure formed by sequentially connecting two-dimensional ocean environmental data matrices from multiple depth layers. The two-dimensional volumetric texture is a graphical data structure storing three-dimensional scalar field data, allowing access to data values ​​at any location in space via three-dimensional coordinate indices. The GPU memory grid is an independent storage area partitioned within the GPU memory, used for parallel processing of different parts of the ocean spatiotemporal data.

[0040] The upper-layer memory grid is a memory partition for processing surface ocean data, responsible for processing marine environmental information at shallower depths. The lower-layer memory grid is a memory partition for processing deep ocean data, responsible for processing marine environmental information at greater depths.

[0041] The volumetric rendering material object is a material resource in Unreal Engine used to achieve 3D volumetric rendering effects. The ray stepping is the core algorithm in volumetric rendering, achieving 3D visualization by emitting rays from the viewpoint and sampling volume data along the ray direction. The bounding box is the smallest cubic region that encloses the 3D object, used to optimize the intersection detection calculation between rays and the object. The maximum vertex coordinates of the bounding box are the largest coordinates of the bounding box in 3D space. The minimum vertex coordinates of the bounding box are the smallest coordinates of the bounding box in 3D space. The minimum intersection point is the starting position where the ray intersects the bounding box. The maximum intersection point is the ending position where the ray intersects the bounding box.

[0042] The adaptive ray stepping algorithm is a ray sampling algorithm that dynamically adjusts the step size based on data characteristics. The nearest frame number is the index of the closest data frame corresponding to the current ray position. The current intra-frame phase is the relative position information of the current position within the frame. The total number of frames is the total number of data slices in the depth direction. The dynamic step size is the ray travel distance adaptively adjusted according to data density.

[0043] The spatial jump technique is a volumetric rendering optimization method that improves rendering performance by detecting null regions and skipping invalid sampling points. The texture coordinates are a two-dimensional coordinate system used to access texture data. The ray casting calculation is the computational process in volumetric rendering where images are generated through the interaction of rays and volume data.

[0044] The linear color mapping curve is a mapping function that converts numerical values ​​into colors, and the texture parameter object is a set of parameters that control texture attributes. The spatial clipping range is the spatial boundary of the display area in volumetric rendering. The spatiotemporal coherence is the correlation characteristic of data between consecutive frames, and the correlation is used to reuse calculation results to reduce redundant operations.

[0045] The data density feature refers to the spatial density of volumetric data. The gradient rate of change is the rate of numerical change between adjacent sampling points. The ray direction vector is the direction of ray propagation in three-dimensional space. The data density distribution feature is the density distribution pattern of volumetric data throughout the entire space.

[0046] The ocean surface data scale refers to the total volume of data within the ocean surface region. The GPU memory bandwidth utilization rate is the efficiency of GPU memory data transfer bandwidth usage. The data transfer latency is the time delay of data transmission between storage and processing units. The processor core utilization rate is the percentage of GPU computing cores used. The memory fragmentation level is the degree of discretization in memory space partitioning. The upper-layer memory grid allocation weight value is the weight coefficient of upper-layer memory grid resource allocation. The ocean deep-sea data complexity is the complexity of the data structure in the ocean deep-sea region. The number of parallel processing threads is the total number of parallel computing threads executing simultaneously. The data access frequency is the number of times data is accessed per unit time. The memory read / write speed is the speed at which the GPU reads and writes data to memory. The cache hit rate is the percentage of data successfully retrieved from the cache during data access. The lower-layer memory grid allocation weight value is the weight coefficient of lower-layer memory grid resource allocation.

[0047] The multi-head self-attention mechanism is a core component of the Transformer architecture, processing feature relationships in different subspaces in parallel through multiple attention heads. The position encoding layer is a network layer in a neural network used to encode positional information. The feedforward neural network layer is a fully connected network layer containing linear transformations and nonlinear activations. The memory fragment storage module is a network component that stores historical information. The memory length parameter controls the storage capacity of memory fragments. Spatiotemporal context refers to relevant background information in both time and space dimensions. The number of memory fragments is the total number of fragments stored in the memory fragment storage module. Spatial resolution is the level of detail of the data in the spatial dimension. Time series length is the length of the data sequence in the time dimension. Data variation complexity is the complexity of data changes over time. Numerical model output data is the output data generated by the ocean numerical model calculation. The spatiotemporal grid is a grid structure composed of time and space dimensions. Multi-scale spatiotemporal feature sequences are feature sequences extracted at different spatiotemporal scales. Ocean reanalysis data is historical ocean data generated through data assimilation techniques. The basic spatiotemporal pattern is the fundamental spatiotemporal variation law in the ocean data. Regional high-resolution observation data is high-precision observation data within a regional area.

[0048] The prediction accuracy refers to the degree of matching between the model's prediction and the actual value. Rendering requirements refer to the user's demands for rendering effects and performance. Memory weights are weight parameters that control the selection of memory modes. Short-term memory mode is a processing mode that uses fewer memory segments. Medium-term memory mode is a processing mode that uses a moderate number of memory segments. Long-term memory mode is a processing mode that uses a large number of memory segments. Computational overhead refers to the resource consumption required to complete the computational task.

[0049] The specific implementation methods of the above steps are described in detail below.

[0050] The specific implementation of step S01 involves first reading the NetCDF file header information using a general network data format parser to extract data dimensional structure parameters, including the size information of four dimensions: longitude, latitude, depth, and time. Based on the depth dimension information, the total number of ocean data layers in the vertical direction is determined; this number is typically between 20 and 100 layers. A depth index array is constructed in ascending order of depth, with the array length equal to the total number of depth layers. A layer-by-layer reading algorithm is used to sequentially extract the two-dimensional ocean environmental data matrix corresponding to each depth layer. This matrix contains the distribution information of parameters such as temperature, salinity, and current velocity on the horizontal plane. The two-dimensional matrices of each depth layer are concatenated in row-major order to form a single large data matrix. The total number of rows and columns of the concatenated matrix is ​​calculated, where the number of rows equals the number of rows in a single layer multiplied by the number of depth layers. For areas where the depth index exceeds the actual data range, a preset null value identifier is filled in; this null value is typically set to -999 or NaN. The min-max normalization algorithm is used to map all valid values ​​in the stitched matrix to the range of 0 to 1, generating a two-dimensional volumetric texture data structure suitable for graphics processors.

[0051] The specific implementation of step S02 involves first establishing a memory partitioning management framework based on the graphics processor's memory capacity and bandwidth characteristics, dividing the available memory space into multiple independent memory grid units according to a preset grid size. The size of each memory grid is dynamically determined based on the spatial resolution and time series length of the ocean data, typically between 512MB and 2GB. A two-layer structure is adopted, dividing the memory grid into an upper-layer and a lower-layer memory grid. The upper-layer grid specifically handles ocean surface data with a depth less than 200m, while the lower-layer grid specifically handles ocean deep-sea data with a depth greater than 200m. A two-layer game theory algorithm model is established. This algorithm, based on Nash equilibrium theory, achieves optimal allocation of memory resources through the interactive optimization of the upper-layer and lower-layer games. The upper-layer game model uses minimizing computational latency as the objective function, with input parameters including the ocean surface data scale, memory bandwidth utilization, data transmission latency, processor core utilization, and memory fragmentation degree. The optimal solution is obtained using the Lagrange multiplier method. The lower-level game theory model aims to maximize processing throughput. Input parameters include the complexity of deep-sea data, the number of parallel processing threads, data access frequency, GPU memory read / write speed, and cache hit rate. The two-level game theory model achieves coordinated optimization through a GPU memory allocation coupling term, ultimately outputting the allocation weights of the upper and lower-level GPU memory grids.

[0052] The specific implementation of step S03 involves creating a new material resource object in the Unreal Engine material editor and setting the material's blending mode parameter to semi-transparent to support transparency blending effects for volume data. The shading mode parameter is configured to be unlit to avoid interference from traditional lighting models on volume rendering effects. A volume shading calculation framework is established based on ray stepping theory, employing a front-to-back traversal strategy to process volume data. The intersection coordinates of the line of sight emanating from the camera position and the bounding box of the volume rendering object are calculated using a ray-bounding box intersection algorithm. The maximum and minimum vertex coordinates of the bounding box in 3D space are obtained; these two coordinates define the effective spatial range of the volume data. The minimum and maximum intersection positions of the ray and the bounding box are calculated using a parametric ray representation method; this calculation process is based on solving the simultaneous equations of the ray and the plane. The effective intersection state of the ray and the bounding box is determined using scene depth buffer information, and invalid ray segments blocked by other objects are eliminated to improve rendering efficiency.

[0053] The specific implementation of step S04 is to establish a dynamic ray stepping algorithm based on data-driven adaptive sampling theory. The nearest frame number is calculated based on the current ray's position coordinates in three-dimensional space, obtained through a mapping function from spatial coordinates to a temporal index. The relative phase value of the current position within the frame is calculated, and this phase value is used for subsequent temporal interpolation operations. The total number of frames is calculated based on the number of rows and columns of the splicing matrix; the total number of frames equals the product of the number of rows and columns of the matrix. A spatiotemporal coherence optimization function is established, based on Markov process theory, utilizing the data correlation between consecutive frames to predict the optimal step size. The function input parameters include the data value of the current sampling point, the gradient change rate of adjacent sampling points, the ray direction vector, the historical step size sequence, and the data density distribution characteristics. The severity of data change is determined by gradient magnitude analysis. In low-density regions with gradients less than 0.01, a large step size is used for rapid advancement, ranging from 0.05 to 0.2. In high-density regions with gradients greater than 0.05, a small step size is used for fine sampling, ranging from 0.001 to 0.01. The integrated spatial skipping technology skips consecutive null value regions by pre-detecting the spatial distribution of null value regions, thus avoiding invalid sampling operations.

[0054] The specific implementation of step S05 involves establishing a parallel texture sampling framework based on video memory grids. This framework supports simultaneous texture data access operations across multiple video memory grids. The corresponding two-dimensional texture coordinates are calculated based on the three-dimensional coordinate components of the current ray's position, and a bilinear mapping function is used to convert the three-dimensional spatial coordinates into texture spatial coordinates. The sampled values ​​of the current frame and the next frame are obtained using the texture coordinates and time parameters; these two sampled values ​​are used for interpolation calculations in the time dimension. The final value of the current sampling point of the ray is calculated using a linear interpolation algorithm based on the phase value within the current frame. An integrated adaptive prediction model for ocean data optimizes the accuracy and efficiency of texture sampling. This model, based on deep learning theory, can predict data values ​​in unsampled areas, reducing the actual number of texture accesses. Efficient sampling processing of large-scale ocean spatiotemporal data is achieved through parallel computing shaders. The shaders employ a single-instruction, multiple-data-stream architecture, supporting simultaneous processing of thousands of sampling points. A texture caching optimization strategy is adopted, storing frequently accessed texture data in a high-speed cache to improve data access efficiency.

[0055] Step S06 is specifically implemented based on the volumetric ray projection theory to perform 3D visualization calculations. This theory generates the final image by simulating the propagation process of light in a volumetric medium. The starting position of the ray is set as the camera's position in the world coordinate system. The ray moves along the calculated direction vector according to the dynamic step size determined in step S04. At each step position, the corresponding data value is obtained using the texture sampling method established in step S05. This value represents the intensity of the marine environmental parameters at the current position. A maximum value projection algorithm is used to accumulate all sampled values ​​during the ray's stepping process, retaining the maximum sampled value on each ray path as the final value of that pixel. A linear color mapping mechanism is established to map the normalized sampled values ​​to a preset color space, typically using a gradient color spectrum from blue to red to represent the change in values ​​from small to large. A semi-transparent volumetric rendering effect is achieved through an alpha blending algorithm, allowing the observer to see the data distribution structure inside the volume. A depth testing optimization mechanism is used to ensure the correct occlusion relationship between the volumetric rendering result and other 3D objects in the scene.

[0056] The specific implementation of step S07 involves establishing a multi-dimensional dynamic interactive control system to support real-time operation and observation of spatiotemporal data. Animation effects of volume rendering in the time dimension are achieved through a dynamic texture parameter object, which includes control parameters such as time index, playback speed, and loop mode. The maximum and minimum vertex coordinates of the bounding box are set as dynamically adjustable parameters, allowing users to modify the spatial display range of the volume rendering in real time via sliders or numerical input boxes. A spatial clipping control mechanism is established, allowing users to observe volume data in sections along any direction; the position and direction of the sections are defined by plane equation parameters. A multi-source ocean data synchronous playback function is integrated, supporting the simultaneous display of ocean data from different sources or with different parameters, enabling data comparison and analysis through color overlay or split-screen display. An interactive camera control system is adopted, allowing users to change the observation perspective through mouse or touch operations, including basic operations such as rotation, zoom, and pan. A parameter linkage mechanism is established, so when a user adjusts one parameter, other related parameters are automatically adjusted accordingly, maintaining the consistency and rationality of the rendering effect.

[0057] The adaptive prediction model for ocean data employs a spatiotemporal sequence prediction network structure based on the Transformer-XL architecture. This architecture processes long-sequence spatiotemporal data through extended self-attention and memory mechanisms. The model includes a multi-head self-attention module, using 8 to 16 attention heads to process feature relationships in different subspaces in parallel. Each attention head focuses on capturing spatiotemporal dependencies at different scales. The location encoding layer uses a combination of sine and cosine functions to encode spatiotemporal coordinates, ensuring the model understands both absolute and relative positional information in the data. The feedforward neural network layer contains two linear transformation layers and a ReLU activation function, with the hidden layer dimension set to four times the input dimension to enhance the model's nonlinear expressive power. The memory fragment storage module maintains a variable-length memory buffer to store hidden state information from historical moments, supporting the modeling of long-distance spatiotemporal dependencies. The memory length parameter is dynamically adjusted based on the spatial resolution of the rendering region, the length of the time series, and the complexity of data variations, ranging from 64 to 512.

[0058] The training dataset creation process includes collecting real-time observational data from a global ocean observation network and acquiring simulated output data from ocean numerical models. The collected raw data was standardized using a unified spatiotemporal grid with a resolution of 0.25 degrees × 0.25 degrees and a time interval of 1 hour. Multi-scale feature extraction methods were employed to extract feature sequences at different spatiotemporal scales from the raw data, including features at multiple time scales such as diurnal, monthly, seasonal, and interannual variations. A mapping relationship between the input sequences and target predictions was constructed, with the input sequence length set to 168 hours and the target prediction duration set to 24 hours. Data augmentation techniques, including time translation, spatial rotation, and noise addition, were used to expand the diversity of training samples and enhance the model's generalization ability. A comprehensive training dataset containing different sea areas, seasons, and depths was established, exceeding 10TB in size and covering the marine environmental change characteristics of major global sea areas.

[0059] The model training employs a hierarchical training strategy. First, it is pre-trained on large-scale ocean reanalysis data to learn the fundamental spatiotemporal variation patterns and laws of ocean data. During the pre-training phase, a self-supervised learning method is used to train the model to understand the intrinsic structure of ocean data through masked prediction tasks. Then, fine-tuning training is performed on regional high-resolution observation data to optimize the model's predictive ability and accuracy for local ocean phenomena. Gradient pruning is used during training to prevent gradient explosion, with a pruning threshold set to 1.0. A cosine annealing learning rate decay strategy is employed, with an initial learning rate of 0.001 and a minimum learning rate of 0.00001. Five-fold cross-validation is used to evaluate the model's prediction accuracy at different spatiotemporal scales, ensuring the model's stability and reliability.

[0060] The adaptive prediction model for ocean data effectively handles the high dimensionality, nonlinearity, and long-distance dependencies of ocean spatiotemporal data. Compared with traditional interpolation methods and statistical prediction models, this model has significant advantages. Traditional bilinear interpolation methods can only perform simple linear combinations based on neighboring spatial points, failing to capture the complex spatiotemporal variations of ocean data and exhibiting low prediction accuracy in sparse data regions. While statistical regression models can handle nonlinear relationships, they are limited by fixed time windows and feature representations, making it difficult to adapt to the multi-scale variations of ocean data. The adaptive prediction model of this invention dynamically captures dependencies at different spatiotemporal scales through memory and self-attention mechanisms, enabling high-quality predictions in sparse data regions through long-distance memory information. The model's adaptability is reflected in the dynamic adjustment of the memory length parameter, automatically selecting the most suitable memory mode based on rendering requirements and data characteristics, optimizing computational efficiency while ensuring prediction accuracy.

[0061] The key technical concepts of this invention include a two-layer game theory algorithm for memory partitioning management, an adaptive ray stepping mechanism, an integrated ocean data prediction model, and spatiotemporal dynamic interactive control. The two-layer game theory algorithm for memory partitioning management offers advantages in dynamic optimization and load balancing compared to traditional static memory allocation methods. It achieves optimal utilization of memory resources through the coordination of upper and lower layer game models, avoiding memory fragmentation and resource waste problems inherent in traditional methods. The adaptive ray stepping mechanism, compared to traditional ray stepping methods with fixed step sizes, dynamically adjusts the sampling density based on data density characteristics, significantly improving rendering efficiency while ensuring rendering quality. It avoids the problems of oversampling in sparse areas and undersampling in dense areas found in traditional methods. The integrated ocean data prediction model, compared to traditional pure interpolation methods, leverages the powerful fitting capabilities of deep learning to predict data values ​​in unsampled areas, reducing the number of actual texture accesses and improving sampling accuracy and efficiency. The spatiotemporal dynamic interactive control provides a more flexible and intuitive way to explore data compared to traditional static visualization methods, supporting users to adjust observation parameters and cross-sectional positions in real time. The synergistic effect of these technical approaches forms a complete spatiotemporal data volume rendering solution. Through the organic combination of memory optimization, rendering acceleration, prediction enhancement, and interactive control, it achieves a comprehensive improvement in performance and effect compared to existing volume rendering technologies, and is particularly suitable for real-time visualization application scenarios of large-scale marine spatiotemporal data.

[0062] It should be noted that this invention also solves the following technical problems: First, this invention addresses the difficulty in balancing ray stepping sampling accuracy and computational efficiency in ocean spatiotemporal data volume rendering. Traditional fixed-step ray stepping algorithms face a trade-off between accuracy and efficiency when processing ocean data. Using small step sizes in areas with uniform data density wastes computational resources, while using large step sizes in areas with drastic data changes leads to the loss of important details. This invention establishes an adaptive ray stepping algorithm that dynamically calculates the optimal step size based on the data density characteristics and gradient change rate of the current ray position. Large step sizes are used in areas with low data density to reduce invalid sampling, while small step sizes are used in areas with high data density to ensure sampling accuracy. Simultaneously, spatial jump technology is used to skip null value areas, achieving an optimal balance between sampling accuracy and computational efficiency. Second, this invention solves the technical problems of discontinuous display of temporal dimension changes and insufficient prediction accuracy in ocean spatiotemporal data visualization. Existing ocean data visualization systems typically only display static data at discrete time points, failing to provide continuous temporal change effects and lacking the ability to predict future ocean conditions. This invention constructs an adaptive prediction model for ocean data based on the Transformer-XL architecture. It utilizes a multi-head self-attention mechanism and a memory fragment storage module to capture the long-term spatiotemporal dependencies of ocean data. By dynamically adjusting the memory length parameter according to rendering requirements through a memory fragment adjustment function, it intelligently switches between short-term, medium-term, and long-term memory modes. This not only ensures the accuracy of time series prediction but also enables continuous visualization of the spatiotemporal evolution of ocean data, providing reliable technical support for the analysis of ocean environmental change trends.

[0063] Specifically, the principle of this invention is as follows: The core principle that solves the problem of unreasonable allocation of video memory resources in marine spatiotemporal data rendering lies in establishing a hierarchical video memory management mechanism based on the differences in marine data characteristics. Ocean surface data typically has a high frequency of spatiotemporal changes and complex environmental dynamics, requiring fast response and low-latency processing. In contrast, deep ocean data, although massive in volume, changes relatively slowly, making it more suitable for high-throughput batch processing. This invention accurately captures these data characteristic differences through a two-layer game theory algorithm, constructing an upper-layer game theory model and a lower-layer game theory model, with the optimization objectives of minimizing computational latency and maximizing processing throughput, respectively, thus achieving differentiated allocation of video memory resources. The key to the logical consistency of this invention's technical solution lies in its complete systematic design concept, forming a closed-loop technical chain from the generation of two-dimensional volumetric textures in the data preprocessing stage, to the establishment of a hierarchical architecture for the video memory management system, and then to adaptive ray stepping and volumetric texture sampling in the rendering stage. The video memory allocation game theory model establishes a coordination mechanism between the upper and lower layer video memory grids through coupling terms, ensuring the optimal allocation of overall system resources. The adaptive ray stepping algorithm dynamically adjusts the step size based on data density characteristics, creating a synergistic effect with the layered processing of the video memory grid, further improving rendering efficiency. The ocean data adaptive prediction model adopted in this invention is based on the Transformer-XL architecture. Through memory fragment storage modules and spatiotemporal coherence optimization, it can predict the spatiotemporal variation trends of ocean data, providing a scientific basis for the pre-allocation of video memory resources. This prediction-driven resource management strategy is a fundamental way to solve the problem of traditional passive resource allocation. The entire technical solution, through multi-level optimization algorithms and adaptive mechanisms, achieves intelligent management from data feature identification to dynamic resource configuration, thereby effectively solving the core problem of unreasonable video memory resource allocation in existing technologies.

[0064] The following provides a specific embodiment 1 of the present invention, and the specific implementation of each step in this embodiment 1 is described in detail below.

[0065] The specific implementation of step S01 is to first read the NetCDF file using a network general data format parser and extract the depth layer number. A depth index array is constructed, and the original ocean spatiotemporal dataset is traversed in depth order. The two-dimensional ocean environment data matrix corresponding to each depth layer is then concatenated. The formula for calculating the concatenated matrix is ​​as follows: In the formula, This is the complete data matrix after splicing; For the first Two-dimensional marine environmental data matrix corresponding to each depth layer ; This represents the total number of layers in the depth direction of the ocean data. The number of rows and columns of the stitched matrix is ​​calculated as follows: , In the formula, This represents the total number of rows in the concatenated matrix. This represents the total number of columns in the concatenated matrix. This represents the number of rows in a single-level data matrix. This represents the number of columns in a single-layer data matrix. The formula for normalizing the concatenated data matrix is ​​as follows: In the formula, Normalized position The value at that location; Original position The value at that location; It is the minimum value of all valid values ​​in the concatenated matrix; This represents the maximum value of all valid values ​​in the concatenated matrix. The depth dimension information is obtained by reading NetCDF files, ranging from 20 to 100 layers. and It is calculated by traversing all non-empty elements in the concatenated matrix.

[0066] The specific implementation of step S02 involves establishing a video memory partitioning management system and using a two-layer game theory algorithm to partition the video memory grid into layers. The formula for calculating the upper-layer objective function is: In the formula, The dimensionless objective function value of the upper-level memory grid; The normalized scale of ocean surface data; This represents the video memory bandwidth utilization rate, with a value ranging from 0 to 1. Normalized data transmission delay; This represents the processor core utilization rate, with a value ranging from 0 to 1. Normalized memory fragmentation level; For the first Each upper-layer memory grid is assigned a weight value, ranging from 0 to 1; These are dimensionless weighting coefficients; It is a Lagrange multiplier; This represents the total number of memory grid cells in the upper layer. The formula for calculating the objective function in the lower layer is: In the formula, The dimensionless objective function value of the lower-level memory grid; Normalized ocean deep data complexity; The normalized number of parallel processing threads; Normalized data access frequency; This represents the normalized video memory read / write speed. This represents the cache hit rate, with a value ranging from 0 to 1. For the first Each lower-level memory grid is assigned a weight value, ranging from 0 to 1; These are dimensionless weighting coefficients; It is a Lagrange multiplier; This represents the total number of lower-level memory grid cells. Among them, The data was obtained by normalizing the data volume with a depth of less than 200m by dividing it by the total data volume; The gradient rate of change variance of data with a depth greater than 200m is calculated and normalized. The calculation formula is as follows: ,in For deep data, Let Variance be the variance function, and the calculation formula is: , For the sample size, For the first Each sample value The sample mean; weighting coefficients to The value ranges from 0.1 to 0.3. to The value ranges from 0.1 to 0.3, and satisfies... and .

[0067] The specific implementation of step S03 involves creating a volumetric rendering material object in Unreal Engine and establishing a volumetric shading framework based on ray stepping. The formula for calculating the intersection point of the ray and the bounding box is: , In the formula, The minimum intersection point parameter between the ray and the bounding box; The parameter represents the maximum intersection point between the ray and the bounding box; The coordinates of the smallest vertex of the bounding box; The coordinates of the largest vertex of the bounding box; The coordinates of the ray's origin; Let be the direction vector of the light ray. and Determined by the spatial range of volume data; This represents the camera's position in the world coordinate system. It is a normalized ray direction vector.

[0068] The specific implementation of step S04 involves establishing an adaptive ray pacing algorithm and using a spatiotemporal coherence optimization function to calculate the dynamic step size. The formula for calculating the total number of frames is: In the formula, This represents the total number of frames corresponding to the splicing matrix. The formula for calculating the spatiotemporal coherence optimization function is: In the formula, This represents the dynamically adjusted step size of the light beam, expressed in meters. The basic step size is in meters. The gradient vector of the current sampling point is calculated using the finite difference method. The partial derivatives are calculated using the central difference scheme. , , ; It is a dimensionless spatiotemporal coherence coefficient; Let be the normalized ray direction vector, satisfying ; This is a dimensionless adjustment coefficient; This represents the spatial difference step size, in meters. The formula for calculating the most recent frame number is: In the formula, The most recent frame number; The depth coordinates of the current ray position, in meters; This is the floor function. The formula for calculating the phase within the current frame is: In the formula, This represents the phase value within the current frame, ranging from 0 to 1. The default value is 0.01m; The value ranges from 0.1 to 1.0. The value ranges from 0.1 to 1.0. The value ranges from 0.1 to 1.0; The default value is 0.001m; The correlation between consecutive frames is obtained by calculating the Pearson correlation coefficient. The calculation formula is as follows: ,in For the first Frame number The value of each sampling point For the first The mean of all sampled points in the frame. For the first Frame number The value of each sampling point For the first The mean of all sampled points in the frame. The total number of sampling points. For time frame index.

[0069] The specific implementation of step S05 involves performing volumetric texture sampling based on the video memory grid, calculating texture coordinates based on the current position of the ray, and then interpolating the results. The formula for calculating texture coordinates is: , In the formula, For the texture horizontal coordinate component; The texture's ordinate component; Let be the current coordinates of the ray. The formula for linear interpolation is: In the formula, This is the final sampled value of the light at the current sampling point; The sampled value of the current frame; This is the sampled value for the next frame. and Through texture coordinates Sampled from a two-dimensional volumetric texture.

[0070] The specific implementation of step S06 involves performing ray casting calculations for volumetric rendering, using a maximum value projection algorithm for ray accumulation. The calculation formula for the ray stepping process is as follows: In the formula, For the next step, position the light source; The current position of the light source; The ray step index is incremented from 0. The formula for calculating the maximum projection is: In the formula, This represents the maximum sample value along the ray path; These are all sampled values ​​during the light ray stepping process; This represents the total number of sampling points. The formula for calculating color mapping is: In the formula, This is the final color value; The minimum color value; This is the maximum color value. and The endpoint color values ​​of the preset linear color mapping curve.

[0071] The specific implementation method of step S07 is the same as described above, and will not be repeated in detail here.

[0072] It should be explained that the formula for calculating the memory segment adjustment function is as follows: In the formula, The weights are dimensionless memory values; These are the normalized spatial resolution parameters; The normalized time series length parameter; For normalized data variation complexity parameters; Normalized rendering accuracy requirements parameters; The weighting coefficients are dimensionless and satisfy the following conditions: The formula for adjusting the number of memory segments is: In the formula, This is a parameter representing the number of memory segments. The number of segments in the short-term memory pattern; The number of segments in the intermediate memory pattern; This represents the number of segments in the long-term memory pattern. Among them, The pixel density of the rendered area is obtained by normalizing it by dividing it by the maximum pixel density; The time series was obtained by normalizing the number of frames by the maximum number of frames. The standard deviation of the gradient of data over time is calculated and normalized. The calculation formula is as follows: ,in It is a function of standard deviation. The rate of change of the data over time is calculated using forward differencing. , For a moment Data values, For time step; Determined by normalization based on the rendering quality level set by the user; The default value is 0.25; The value ranges from 64 to 128. The value range is from 128 to 256. The value ranges from 256 to 512.

[0073] It needs to be explained that the formula for calculating the spliced ​​matrix is... The principle is based on the theory of vertical matrix stitching, which concatenates two-dimensional data matrices from different depth layers in depth order to form a unified data structure. The effect of this formula is to convert three-dimensional ocean data into a two-dimensional texture format suitable for graphics processors, reducing memory usage and improving data access efficiency compared to traditional three-dimensional texture storage methods. Normalization formula Based on linear transformation theory, the original data is mapped to a standardized range of 0 to 1, eliminating the influence of differences in the dimensions of different parameters. This improves the consistency of rendering effects and the accuracy of color mapping compared to directly using the original numerical values.

[0074] It needs to be explained that the upper-level objective function of the two-level game model... Based on Nash equilibrium theory, optimal allocation of video memory resources is achieved through multi-objective optimization. This function includes a data scale term. Used to measure the processing load of surface data, bandwidth utilization. Used to evaluate video memory transfer efficiency and latency. Used to control response time, core resource usage. Used to balance computing resources, fragmented items Constraints used to reduce memory waste This formula is used to ensure the rationality of weight allocation. Its effect is to achieve dynamic optimization of video memory resource allocation, significantly improving video memory utilization and processing efficiency compared to traditional static allocation methods.

[0075] It needs to be explained that the spatiotemporal coherence optimization function Based on adaptive sampling theory, the ray step size is dynamically adjusted by analyzing data gradients and spatiotemporal correlations. This function includes a gradient term. Used to detect the degree of drastic change in data, including gradient calculation. Central difference scheme Achieving high-precision spatial variation detection, coherence term Used to utilize spatiotemporal continuity, where the coherence coefficient The Pearson correlation coefficient is used to quantify the strength of data association between consecutive frames, where the mean is calculated. To ensure the accuracy of correlation calculations, the direction term... This formula takes into account the characteristics of light propagation. Its effect is to maximize rendering efficiency while maintaining rendering quality, reducing invalid sampling and improving detail capture compared to traditional methods with fixed step sizes.

[0076] It needs to be explained that the memory segment regulation function Based on multi-dimensional evaluation theory, the optimal memory pattern is determined by comprehensively considering spatial resolution, time duration, complexity, and accuracy requirements. The complexity term in this function... Through the standard deviation function Quantifying the irregularity of data changes over time, including the rate of change over time. The calculation is performed using a forward difference scheme, with the following constraints: Ensure the weight allocation is reasonable. The function's effect is to adaptively adjust the memory capacity of the prediction model, optimizing computational overhead and response speed while maintaining prediction accuracy compared to traditional methods with fixed memory length.

[0077] To better understand and implement this invention, the following is a specific application scenario of the invention, Example 2: A technical team needs to perform real-time visualization analysis of the three-dimensional marine environment of a certain sea area. This sea area covers approximately 15,000 square kilometers, with a depth ranging from the sea surface to 3,000 meters, and includes complex ocean current structures, temperature stratification, and salinity gradient changes. Traditional two-dimensional cross-sectional display methods cannot meet the need for intuitive observation of the three-dimensional structure inside the ocean. The team decided to use spatiotemporal data volume rendering technology based on Unreal Engine to solve this problem.

[0078] First, the technical team acquired a marine spatiotemporal dataset based on the NetCDF format. This dataset contains marine observation data for the entire year of 2024, with a spatial resolution of 0.25°×0.25° and a temporal resolution of 3 hours, divided into 42 depth layers vertically. By reading the file header information using a NetCDF parser, the data dimensional structure was determined to be 360 ​​grid points for longitude, 180 grid points for latitude, 42 depth layers, and 2920 time points. A depth index array was constructed in order of depth from 0m to 3000m, with an array length of 42. A layer-by-layer reading algorithm was used to extract a 360×180 two-dimensional marine environmental data matrix corresponding to each depth layer. This matrix contains three parameters: temperature, salinity, and current velocity. The two-dimensional matrices of the 42 depth layers were concatenated in row-major order to form a large data matrix of 15120×180. For regions where the depth index exceeds the actual data range, -999 null values ​​were filled in. A min-max normalization algorithm was used to map all valid values ​​to the range of 0 to 1, generating a two-dimensional volumetric texture suitable for GPU processing.

[0079] Next, a memory partitioning management system was established, dividing the 8GB GPU memory into 8 independent memory grid units with a 1GB grid size. A two-layer architecture was adopted: the upper memory grid allocated 4 units specifically for processing ocean surface data with depths less than 200m, and the lower memory grid allocated 4 units for processing ocean deep-sea data with depths greater than 200m. A two-layer game theory algorithm model was established. The objective function of the upper-layer game model was set to minimize computational latency, with input parameters including ocean surface data size of 482MB, memory bandwidth utilization of 73%, data transfer latency of 2.3ms, processor core utilization of 68%, and memory fragmentation of 0.15. The objective function of the lower-layer game model was set to maximize processing throughput, with input parameters including ocean deep-sea data complexity of 1.24, number of parallel processing threads of 2048, data access frequency of 156 times / s, memory read / write speed of 847GB / s, and cache hit rate of 86%. The Lagrange multiplier method was used to solve the problem, and the weight values ​​for the upper memory grid allocation were found to be 0.62 and 0.38, respectively.

[0080] Create a volumetric rendering material object in Unreal Engine, setting its blending mode to semi-transparent and its shading mode to no lighting. Establish a volumetric shading framework based on ray stepping, calculating the intersection points of the line of sight emanating from the camera position and the bounding box of the volumetric rendering object. Set the maximum vertex coordinates of the bounding box to (500, 500, 300) and the minimum vertex coordinates to (-500, -500, -300), in kilometers. Calculate the minimum and maximum intersection points of the rays using a ray-bounding box intersection algorithm, and use the scene depth buffer to determine the effective intersection status of the rays and the bounding box.

[0081] An adaptive ray stepping algorithm is established to calculate the nearest frame number and the current intra-frame phase based on the current ray position. The total number of frames is calculated based on the 15120×180 size of the stitching matrix. A spatiotemporal coherence optimization function is established, with input parameters including the current sampling point data value, the gradient change rate of adjacent sampling points, the ray direction vector, the historical step size sequence, and the data density distribution characteristics. A large step size of 0.12 is used in low-density regions with gradients less than 0.01, and a small step size of 0.008 is used in high-density regions with gradients greater than 0.05. A spatial skipping technique is integrated to pre-detect the distribution of null value regions and directly skip consecutive null value regions.

[0082] Volumetric texture sampling based on GPU memory grids is implemented, calculating the corresponding texture coordinates using the x and y coordinate components of the current ray position. The current frame's sampled value and the next frame's sampled value are obtained, and linear interpolation is performed based on the phase within the current frame. An adaptive prediction model for ocean data based on the Transformer-XL architecture is used to optimize texture sampling accuracy. This model includes an 8-head self-attention mechanism, a 512-dimensional positional encoding layer, and a 2048-dimensional feedforward neural network layer. The memory fragment storage module capacity is set to 256 fragments, and the memory length parameter is dynamically adjusted according to the rendering area. The prediction model parameter configuration is shown in Table 1.

[0083] Table 1. Parameter Configuration Table for Adaptive Prediction Model for Ocean Data

[0084]

[0085] The parallel computing shader accelerates the sampling and processing of large-scale ocean spatiotemporal data, supporting simultaneous processing of 2048 sampling points. A texture caching optimization strategy is employed, storing frequently accessed texture data in a 512MB cache.

[0086] The ray casting calculation for volumetric rendering is performed, with rays originating from the camera position and moving along the direction vector in dynamic steps. At each step, data values ​​are obtained through volumetric texture sampling, and a maximum value projection algorithm is used to accumulate the maximum sampled values ​​during the ray's movement. A linear color mapping mechanism is established, using a blue-to-red gradient spectrum to represent numerical changes, and an alpha blending algorithm is used to achieve a semi-transparent volumetric rendering effect. Rendering performance metrics are shown in Table 2.

[0087] Table 2. Statistics of Volumetric Rendering Performance Indicators

[0088]

[0089] A dynamic, interactive control system is established to control volumetric rendering over time using dynamic texture parameter objects. Bounding box coordinates are set as dynamic parameters, allowing users to adjust the spatial clipping range of volumetric rendering in real time. A spatial clipping control mechanism is established, allowing users to view volumetric data in sections along any direction. Multi-source ocean data synchronization is integrated, supporting simultaneous display and comparative analysis of temperature, salinity, and current velocity.

[0090] The memory fragment adjustment function dynamically adjusts the memory length parameter of the prediction model according to rendering requirements. The memory weight value is calculated based on four parameters: spatial resolution of 0.25°, time series length of 2920, data variation complexity of 1.24, and rendering accuracy requirement of 0.95. When the memory weight value is 0.28, a short-term memory mode is used, with the number of memory fragments set to 64, reducing computational overhead by 35%. When the memory weight value is 0.56, a medium-term memory mode is used, with the number of memory fragments set to 128, balancing accuracy and efficiency. When the memory weight value is 0.84, a long-term memory mode is used, with the number of memory fragments set to 256, maximizing prediction accuracy. The memory mode configuration parameters are shown in Table 3.

[0091] Table 3 Comparison of Memory Mode Configuration Parameters

[0092]

[0093] During the establishment of the training dataset, the team collected global ocean observation data and numerical model output data, including multi-source data such as Argo buoy data, satellite remote sensing data, and ship observation data. The data was standardized according to a 0.25°×0.25° spatiotemporal grid to extract multi-scale spatiotemporal feature sequences. A mapping relationship between the input sequences and target prediction values ​​was constructed, and data augmentation techniques were used to expand the training samples, resulting in a comprehensive training dataset containing different sea areas, seasons, and depths, with a total size of 487GB.

[0094] The predictive model employs a hierarchical training strategy. First, it is pre-trained on large-scale ocean reanalysis data to learn the basic spatiotemporal patterns of the ocean data. Then, it is fine-tuned on regional high-resolution observational data to optimize the model's predictive ability for local ocean phenomena. During training, a gradient pruning threshold of 1.0 and a learning rate decay strategy are used to prevent overfitting. Five-fold cross-validation is used to evaluate the model's predictive accuracy at different spatiotemporal scales.

[0095] After three months of development and debugging, the technical team successfully implemented a real-time 3D marine environment visualization system for the Xisha Islands area of ​​the South China Sea. The system can smoothly display 3D marine data at a frame rate of 42fps, allowing users to interactively observe temperature stratification, salinity gradients, and current field structures within the ocean. Compared to traditional 2D cross-sectional display methods, this system improves data display completeness by 18%, user interaction experience by 15%, and rendering efficiency by 12%. Traditional methods can only display the 2D distribution of a single depth layer, failing to intuitively demonstrate the continuity and hierarchy of the ocean's 3D structure. The new system, however, can completely display the continuous 3D data distribution from the sea surface to the seabed, providing a more intuitive and efficient visualization and analysis tool for marine scientific research and marine engineering applications.

[0096] It should be noted that the variables involved in this invention are explained in detail in Tables 4 and 5.

[0097] Table 4. Variable Explanation Table (Part 1)

[0098]

[0099] Table 5. Variable Explanation Table (Part Two)

[0100]

[0101] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for rendering spatiotemporal data volumes for Unreal Engine, characterized in that, This includes acquiring ocean spatiotemporal data based on NetCDF format and generating 2D volumetric textures; establishing a memory partitioning management system that divides GPU memory into upper and lower memory grids, with the upper grid processing surface ocean data and the lower grid processing deep ocean data, employing a two-layer game theory algorithm for memory grid hierarchy partitioning and using a memory allocation game model to calculate the optimal allocation scheme; creating volumetric rendering material objects in Unreal Engine and establishing a volumetric shading framework based on ray stepping; establishing an adaptive ray stepping algorithm and using a spatiotemporal coherence optimization function to calculate the dynamic step size; implementing volumetric texture sampling based on memory grids and optimizing texture sampling using an ocean data adaptive prediction model; performing ray casting calculations for volumetric rendering; establishing dynamic interactive control in the spatiotemporal dimension to achieve synchronous visualization of multi-source ocean data; and accurately capturing the differences in data density, access frequency, and processing complexity between surface and deep ocean data through a hierarchical memory management mechanism based on ocean data feature differences and a two-layer game theory algorithm to achieve differentiated configuration of memory resources.

2. The spatiotemporal data volume rendering method for Unreal Engine according to claim 1, characterized in that, The steps for generating a two-dimensional volumetric texture are as follows: determine the number of layers in the ocean spatiotemporal data in the depth direction, extract the depth index array, traverse the original ocean spatiotemporal dataset in depth order, stitch together the two-dimensional ocean environment data matrix corresponding to each depth layer in row-first-column order, calculate the number of rows and columns of the stitched matrix, fill null values ​​in areas exceeding the depth layer index value, and normalize the stitched complete data matrix to generate a two-dimensional volumetric texture.

3. The spatiotemporal data volume rendering method for Unreal Engine according to claim 2, characterized in that, The steps of establishing a video memory partition management system specifically involve dividing the GPU video memory into multiple video memory grids, with each video memory grid used for streaming processing of individual ocean spatiotemporal data, and dividing the video memory grid into two layers: an upper video memory grid and a lower video memory grid.

4. The spatiotemporal data volume rendering method for Unreal Engine according to claim 3, characterized in that, The steps for establishing a volume shading framework based on ray stepping specifically involve setting the blending mode to semi-transparent, setting the shading mode to no lighting, calculating the intersection point of the ray emitted from the camera with the bounding box of the volume rendering object, obtaining the coordinates of the maximum and minimum vertices of the bounding box, calculating the minimum and maximum intersection points of the ray and the bounding box, and determining the effective intersection state of the ray and the bounding box through scene depth control.

5. The spatiotemporal data volume rendering method for Unreal Engine according to claim 4, characterized in that, The adaptive ray stepping algorithm specifically calculates the nearest frame number and the current intra-frame phase based on the current ray position. The total number of frames is equal to the square of the number of rows and columns of the splicing matrix. A large step size is used in areas with low data density, and a small step size is used in areas with high data density. Spatial jump technology is used to skip empty value areas to improve rendering efficiency.

6. The spatiotemporal data volume rendering method for Unreal Engine according to claim 5, characterized in that, The volumetric texture sampling step based on the memory grid specifically involves calculating the corresponding texture coordinates using the horizontal and vertical coordinate components of the current position of the light ray, obtaining the current frame sampling value and the next frame sampling value, performing linear interpolation based on the phase within the current frame to obtain the final sampling value of the current sampling point of the light ray, and accelerating the sampling processing of large-scale ocean spatiotemporal data through a parallel computing shader.

7. The spatiotemporal data volume rendering method for Unreal Engine according to claim 6, characterized in that, The specific steps of the light projection calculation are as follows: the light starts from the camera position and moves forward along the direction vector according to the dynamic step size. At each step position, data values ​​are obtained by sampling the volume texture. The maximum value of each light ray during the light stepping process is accumulated. The corresponding color value is obtained from the preset linear color mapping curve based on the sampling data to achieve the final three-dimensional volume rendering visualization effect.

8. The spatiotemporal data volume rendering method for Unreal Engine according to claim 7, characterized in that, The steps of dynamic interactive control in the spatiotemporal dimension are specifically to realize the change effect of volume rendering in the time dimension by dynamically updating the texture parameter object, setting the coordinates of the maximum vertex of the bounding box and the coordinates of the minimum vertex of the bounding box as dynamic parameters, and adjusting the spatial clipping range of volume rendering by controlling the corresponding parameter values ​​in real time, allowing users to observe the cross-sectional effect of the internal structure of ocean spatiotemporal data through interactive operations.

9. The spatiotemporal data volume rendering method for Unreal Engine according to claim 8, characterized in that, The spatiotemporal coherence optimization function is specifically used to calculate the optimal step size based on the data density characteristics of the current ray position. The inputs include the current sampling point data value, the gradient change rate of adjacent sampling points, the ray direction vector, the historical step size sequence, and the data density distribution characteristics. The output is the dynamically adjusted ray step size value.

10. The spatiotemporal data volume rendering method for Unreal Engine according to claim 9, characterized in that, The memory allocation game model specifically includes an upper-level game model that aims to minimize the computation latency of the upper-level memory grid and a lower-level game model that aims to maximize the processing throughput of the lower-level memory grid. The upper-level game model has an upper-level objective function and upper-level constraints, and the lower-level game model has a lower-level objective function and lower-level constraints. The upper-level objective function and the lower-level objective function have memory allocation coupling terms.

Images