Meteorological visualization engine based on space-time database

Abstract

The invention relates to the technical field of meteorological visualization, in particular to a meteorological visualization engine based on a space-time database, which comprises a meteorological data storage and processing module, a data processing algorithm cluster module, a data query analysis module and a WebGL-based visualization rendering module, and is characterized in that the meteorological data storage and processing module comprises the space-time database and a cloud storage; the data processing algorithm cluster module is used for analyzing, processing and processing various kinds of meteorological data, and has the following beneficial effects that the problems of performance bottleneck and expansibility when large-scale meteorological data is processed in a traditional storage mode can be solved, and the data processing algorithm cluster module can perform real-time or batch processing on the meteorological data; according to the technical scheme, the meteorological data are analyzed and processed to meet different scenes and requirements, and accurate and high-quality data support can be provided for subsequent data query, analysis and visualization by analyzing and processing the meteorological data.

Description

Technical Field

[0001] The present invention relates to the technical field of meteorological visualization, and in particular to a meteorological visualization engine based on a spatiotemporal database. Background Art

[0002] With the continuous growth of meteorological data, traditional storage technology can no longer meet the storage needs of large-scale meteorological data. At present, the storage of meteorological data mainly relies on relational databases. Although relational databases perform well in data structuring, consistency and stability, their performance and scalability are greatly limited when storing large-scale meteorological data. In addition, relational databases often require additional processing logic when processing spatiotemporal data, which undoubtedly increases the complexity and maintenance cost of the system. When processing large-scale meteorological data, traditional storage systems often encounter IO and memory bottlenecks. Due to the huge amount of meteorological data and frequent read and write operations, the IO performance of the storage system is insufficient, affecting the data processing speed. At the same time, since meteorological data needs to be analyzed in real time, the demand for memory is also extremely high. Existing storage systems often cannot provide sufficient memory resources, resulting in further reduction in data processing efficiency. The visualization of meteorological data is an important part of meteorological services. However, in the rendering process of meteorological data, traditional rendering technologies often encounter performance bottlenecks. Due to the high dimension and large scale of meteorological data, the rendering process requires a large amount of computing resources and memory resources. Existing rendering technologies often cannot complete the rendering of meteorological data within a limited time, resulting in poor rendering effects and affecting the user experience of meteorological services. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the present invention provides a meteorological visualization engine based on a spatiotemporal database to solve the problems of low data processing efficiency and poor rendering effect.

[0004] To solve the above problems, the technical solution of the present invention is as follows: A meteorological visualization engine based on a spatiotemporal database, comprising:

[0005] A meteorological data storage and processing module, the meteorological data storage and processing module includes a spatiotemporal database and a cloud storage, the spatiotemporal database is used to store and process meteorological data, and the cloud storage is used to support data storage and access;

[0006] Data processing algorithm cluster module, used to parse, process and process various meteorological data;

[0007] Data query and analysis module, used to analyze meteorological data after being processed and stored in the spatiotemporal database, and supports efficient query of database data;

[0008] A WebGL-based visualization rendering module is used to process and render meteorological data.

[0009] Furthermore, the spatiotemporal database includes a time series sub-database and a space sub-database, the time series sub-database includes a Timescale plug-in for expanding the statistical query and analysis capabilities of the spatiotemporal database, and the space sub-database includes a PostGis plug-in for expanding the query and analysis capabilities of the spatiotemporal database.

[0010] Furthermore, the spatiotemporal database also includes a grid model plug-in built based on Rust, and the grid model plug-in built based on Rust is used to uniformly process grid query, statistics, and inversion.

[0011] Furthermore, the meteorological data storage and processing module adopts horizontal distribution, high availability, and multi-core CPU parallel computing.

[0012] Furthermore, the data processing algorithm cluster module adopts one or more of a data cleaning algorithm, a data compression algorithm, a feature extraction and dimensionality reduction algorithm, and a prediction and interpolation algorithm.

[0013] Furthermore, the data processing algorithm cluster module realizes data processing based on the algorithm cluster, including the following steps:

[0014] (1) Data aggregation: Aggregate meteorological data according to business needs, including calculating daily averages, monthly averages, and annual cumulative values;

[0015] (2) Data interpolation: For missing data points, interpolation algorithms are used to fill them, including linear interpolation and Kriging interpolation, to ensure the integrity and continuity of the data;

[0016] (3) Data compression: compress large-scale meteorological data;

[0017] (4) Feature extraction: Extract key features from meteorological data for subsequent data analysis and prediction.

[0018] Furthermore, the WebGL-based meteorological visualization rendering module is used to:

[0019] ① WebGL environment construction;

[0020] ②Create visual elements;

[0021] ③Shader programming;

[0022] ④ Texture mapping and color mapping: Map meteorological data onto textures to achieve intuitive display of data. Textures include simple color gradients and complex images. The color or transparency of the texture is adjusted according to the value of the meteorological data to reflect the actual status of the data.

[0023] Furthermore, the WebGL environment construction includes building a WebGL rendering context in a web page environment, through HTML5 <canvas>The element implements and uses the WebGL API to create and manage WebGL rendering contexts.

[0024] Furthermore, the creation of visual elements includes creating corresponding visual elements in WebGL according to meteorological data, the visual elements include points, lines, surfaces, and bodies, the visual elements are defined and controlled by vertex shaders and fragment shaders, the vertex shader is responsible for processing vertex data, the vertex data includes position, color, and texture coordinates, and the fragment shader is responsible for processing pixel data between vertices, and finally generating a rendered image.

[0025] Furthermore, the shader programming is used to write GLSL (OpenGL Shading Language), write vertex shader and fragment shader codes to achieve specific rendering effects. In the shader, mathematical functions and texture mapping technology are used to enhance the rendering effect and realize gradient color and isosurface switching display.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. The spatiotemporal database has the ability to store and manage large-scale spatiotemporal data and can effectively store and process meteorological data. Cloud storage provides flexible scalability and high availability and can support the storage and access of massive data. The combination of the spatiotemporal database and cloud storage solves the performance bottleneck and scalability problems of traditional storage methods when processing large-scale meteorological data.

[0028] 2. The data processing algorithm cluster module can process meteorological data in real time or in batches to meet different scenarios and needs. By analyzing and processing meteorological data, it can provide accurate and high-quality data support for subsequent data query, analysis and visualization. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the business architecture of the present invention;

[0030] Figure 2 This is a schematic diagram of the meteorological data storage and processing module of the present invention. DETAILED DESCRIPTION

[0031] like Figure 1 As shown, a meteorological visualization engine based on spatiotemporal database includes:

[0032] Meteorological data storage and processing module

[0033] like Figure 2 As shown in the figure, based on the meteorological data storage and processing module, an efficient and scalable meteorological data storage and processing module is constructed. The spatiotemporal database has the ability to store and manage large-scale spatiotemporal data, and can effectively store and process meteorological data. Cloud storage provides flexible scalability and high availability, and can support the storage and access of massive data. Through the combination of spatiotemporal database and cloud storage, the performance bottleneck and scalability problems of traditional storage methods in processing large-scale meteorological data can be solved. Meteorological data includes NC files, numerical grids, and radar grids. The spatiotemporal database includes a time series sub-database and a space sub-database. The time series sub-database includes a Timescale plug-in for expanding the statistical query and analysis capabilities of the spatiotemporal database. The spatial sub-database includes a PostGis plug-in for expanding the query and analysis capabilities of the spatiotemporal database. The spatiotemporal database also includes a grid model plug-in based on Rust. The grid model plug-in based on Rust is used to uniformly process the query, statistics, and inversion capabilities of the grid. At the same time, the horizontal distribution, high availability, and multi-core CPU parallel computing capabilities can be reused to improve data processing capabilities.

[0034] Data processing algorithm cluster module

[0035] Used to parse, process and process various types of meteorological data, the data processing algorithm cluster module can process meteorological data in real time or in batches to meet different scenarios and needs. By parsing and processing meteorological data, it can provide accurate and high-quality data support for subsequent data query, analysis and visualization.

[0036] The data processing algorithm cluster module adopts one or more of the following algorithms: data cleaning algorithm, data compression algorithm, feature extraction and dimension reduction algorithm, prediction and interpolation algorithm.

[0037] Data cleaning algorithm, missing value processing: identify and fill or delete missing values ​​in the data set to ensure data integrity, outlier detection and correction: identify and process outliers through statistical methods or machine learning models to prevent them from having adverse effects on subsequent analysis.

[0038] Data compression algorithm, spatiotemporal data compression: utilizing the spatiotemporal correlation of meteorological data, using algorithms such as run-length encoding and Huffman coding to compress data, reducing the demand for storage space and transmission bandwidth, combining lossy compression with lossless compression: flexibly selecting compression methods based on the importance and accuracy requirements of the data, balancing data quality and storage efficiency.

[0039] Feature extraction and dimensionality reduction algorithm, principal component analysis (PCA): by extracting the main characteristic components in the data, reducing the dimension of the data and improving processing efficiency.

[0040] Prediction and interpolation algorithms, spatial interpolation: such as inverse distance weighted interpolation (IDW), Kriging interpolation (Kriging), etc., are used to fill in the missing meteorological data points in space.

[0041] The data processing algorithm cluster module realizes data processing based on the algorithm cluster, including the following steps:

[0042] (1) Data aggregation: Aggregate meteorological data according to business needs, such as calculating daily averages, monthly averages, and annual cumulative values.

[0043] (2) Data interpolation: For missing data points, interpolation algorithms (such as linear interpolation, Kriging interpolation, etc.) are used to fill them in to ensure the integrity and continuity of the data.

[0044] (3) Data compression: Compress large-scale meteorological data to reduce storage space requirements while maintaining data accuracy and availability.

[0045] (4) Feature extraction: Extract key features from meteorological data for subsequent data analysis and prediction.

[0046] Data storage and query module

[0047] The processed meteorological data is stored in the spatiotemporal database, and efficient query of the database data is supported. The core data service provides customized support for query, analysis, visualization and other functions of the database data to meet the needs of different users and businesses. In addition, this technical solution also supports the implementation of dynamic raster slicing services, dynamic vector slicing services and dynamic grid slicing services, thereby providing users with richer data display and analysis functions.

[0048] Weather visualization rendering module based on WebGL

[0049] In order to achieve intuitive display of meteorological data, a grid meteorological visualization rendering module based on WebGL is adopted. WebGL is a Web-based graphics rendering technology with the advantages of high performance, cross-platform and good compatibility. By binding WebGL technology with the background visualization service, this technical solution can handle the data rendering process as transparently as possible and improve the user experience. At the same time, the WebGL-based visualization engine can also support multi-dimensional meteorological data display and analysis, providing users with richer and more accurate data visualization services.

[0050] WebGL-based meteorological visualization rendering module is used

[0051] ①WebGL environment construction:

[0052] It is necessary to build a WebGL rendering context in the web environment, through HTML5 <canvas>The element implements and uses the WebGL API to create and manage WebGL rendering contexts.

[0053] ②Create visual elements:

[0054] In WebGL, corresponding visual elements such as points, lines, surfaces, and volumes are created based on meteorological data. These elements are defined and controlled by vertex and fragment shaders.

[0055] The vertex shader is responsible for processing vertex data, such as position, color, texture coordinates, etc., while the fragment shader is responsible for processing the pixel data between vertices and ultimately generating a rendered image.

[0056] ③Shader programming:

[0057] Write vertex and fragment shader codes written in GLSL (OpenGL Shading Language) to achieve specific rendering effects. These codes define the data processing method and rendering logic. In the shader, various mathematical functions and texture mapping techniques can be used to enhance the rendering effect, such as realizing gradient color, isosurface switching display, etc.

[0058] ④Texture mapping and color mapping:

[0059] Meteorological data is mapped onto a texture to achieve an intuitive display of the data. The texture can be a simple color gradient or a complex image. The color or transparency of the texture is adjusted according to the value of the meteorological data to reflect the actual status of the data.

[0060] The above specific implementation methods are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, a person skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.< / canvas> < / canvas>

Claims

1. A meteorological visualization engine based on spatiotemporal database, characterized in that: include: A meteorological data storage and processing module, the meteorological data storage and processing module includes a spatiotemporal database and a cloud storage, the spatiotemporal database is used to store and process meteorological data, and the cloud storage is used to support data storage and access; Data processing algorithm cluster module, used to parse, process and process various meteorological data; Data query and analysis module, used to analyze meteorological data after being processed and stored in the spatiotemporal database, and supports efficient query of database data; A WebGL-based visualization rendering module is used to process and render meteorological data.

2. The meteorological visualization engine according to claim 1, characterized in that: The spatiotemporal database includes a time series sub-database and a space sub-database. The time series sub-database includes a Timescale plug-in for expanding the statistical query and analysis capabilities of the spatiotemporal database, and the space sub-database includes a PostGis plug-in for expanding the query and analysis capabilities of the spatiotemporal database.

3. The meteorological visualization engine according to claim 2, characterized in that: The spatiotemporal database also includes a grid model plug-in built based on Rust, and the grid model plug-in built based on Rust is used to uniformly process grid query, statistics, and inversion.

4. The meteorological visualization engine according to claim 3, characterized in that: The meteorological data storage and processing module adopts horizontal distribution, high availability, and multi-core CPU parallel computing.

5. The meteorological visualization engine according to claim 4, characterized in that: The data processing algorithm cluster module adopts one or more of a data cleaning algorithm, a data compression algorithm, a feature extraction and dimensionality reduction algorithm, and a prediction and interpolation algorithm.

6. The meteorological visualization engine according to claim 5, characterized in that: The data processing algorithm cluster module realizes data processing based on the algorithm cluster, including the following steps: (1) Data aggregation: Aggregate meteorological data according to business needs, including calculating daily averages, monthly averages, and annual cumulative values; (2) Data interpolation: For missing data points, interpolation algorithms are used to fill them, including linear interpolation and Kriging interpolation, to ensure the integrity and continuity of the data; (3) Data compression: compress large-scale meteorological data; (4) Feature extraction: Extract key features from meteorological data for subsequent data analysis and prediction.

7. The meteorological visualization engine according to claim 6, characterized in that: The WebGL-based meteorological visualization rendering module is used to: ① WebGL environment construction; ②Create visual elements; ③Shader programming; ④ Texture mapping and color mapping: Map meteorological data onto textures to achieve intuitive display of data. Textures include simple color gradients and complex images. The color or transparency of the texture is adjusted according to the value of the meteorological data to reflect the actual status of the data.

8. The meteorological visualization engine according to claim 7, characterized in that: The WebGL environment construction includes building a WebGL rendering context in a web page environment, through HTML5 <canvas> The element implements and uses the WebGL API to create and manage WebGL rendering contexts.< / canvas> 9. The meteorological visualization engine according to claim 8, characterized in that: The creation of visual elements includes creating corresponding visual elements in WebGL according to meteorological data, the visual elements include points, lines, surfaces, and bodies, the visual elements are defined and controlled by vertex shaders and fragment shaders, the vertex shader is responsible for processing vertex data, the vertex data includes position, color, and texture coordinates, and the fragment shader is responsible for processing pixel data between vertices, and finally generating a rendered image.

10. The meteorological visualization engine according to claim 9, characterized in that: The shader programming is used to write GLSL (OpenGL Shading Language), write vertex shader and fragment shader codes to achieve specific rendering effects. In the shader, mathematical functions and texture mapping technology are used to enhance the rendering effect and realize gradient color and isosurface switching display.

Images