Method and system for parsing and storing marine data

Abstract

The application relates to a NetCDF marine data analysis and storage method and system based on a localization platform. According to the multi-dimensional and multi-variable data characteristics of a NetCDF file, a set of storage structures suitable for a Dreambase database is designed, including table structure design, table association relationship and query analysis storage process. Based on the storage design scheme, data analysis is self-defined according to three hierarchical structures of general metadata, grid information and grid point data. A variable slicing method is adopted to efficiently extract three-dimensional and four-dimensional data. Further, the marine data analysis and storage method is applied to the localization platform, and the system integrates a whole-process solution scheme from data access, analysis, storage to query analysis, so that efficient management and utilization of marine environment data are ensured. The application effectively fills the blank of integrated technology of multi-source NetCDF file data processing and storage under the localization platform, and further promotes the localization process of the national information system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ocean environment data processing and database storage, in particular to a NetCDF ocean data analysis and storage method and system based on a domestic platform. BACKGROUND

[0002] With the continuous development of ocean observation means, from observation platform to numerical simulation, ocean environment data presents obvious multi-source heterogeneous characteristics. Among them, NetCDF (Network Common Data Form, NC file for short) as a common scientific data format is widely used. However, these format file data need to be parsed and processed before they can be effectively used, and efficient storage and management of a large amount of parsed ocean data also becomes a current problem to be solved. At the same time, the country is vigorously promoting the information system localization process, and it is urgent to establish an efficient and reliable technical solution on the national localization platform. Current data processing and storage technology is mostly deployed on commercial operating systems and non-domestic database platforms. Under the background of promoting the localization of national information systems, the integrated solution of NetCDF ocean data analysis and storage based on the national localization environment is still relatively scarce.

[0003] The current parsing method for NC file format data mainly includes: using NetCDF standard library for programming analysis, using third-party middleware visualization tools. But these methods have certain deficiencies in use, lack of enough flexibility, especially before encoding analysis, it is often necessary to view file metadata information (such as variables, attributes, dimensions, etc.) first.

[0004] In terms of data storage, there are not many database storage technology solutions for NC data parsing results. Most technical personnel in the business application development in the fields of meteorology, ocean, etc. still generally use the traditional way of directly reading the gridded NetCDF data file, and lack of means for deep processing and efficient storage and management of original observation data.

[0005] In the business application development in the field of ocean, there are few integrated application systems or solutions from multi-source NC file data parsing processing to efficient storage and storage. At present, there is no product-level solution that can truly meet the industry demand. SUMMARY

[0006] In view of the problems of insufficient flexibility of current NC file data analysis methods, relatively insufficient data storage technical solutions, and lack of integrated system application under the domestic platform, the application designs an integrated technical solution for efficiently analyzing and processing multi-source NC files under the national platform and rapidly storing the analysis results into a domestic database, so as to meet the actual business needs of ocean environment data integration, quality control and application development, support data application in the field of ocean, and help the process of domesticization. In order to achieve the above object, the application adopts the following technical solution:

[0007] In the first aspect, the application provides a method for NetCDF ocean data analysis and storage based on a domestic platform, specifically comprising:

[0008] 1. According to the data characteristics of the NetCDF file, the storage structure of the Dream database is designed, including constructing a data table, creating a table association relationship, and creating a storage process.

[0009] (1) Create a data table, the structure design of the dataset table (dataset): datasetID, grid starting longitude and latitude, grid longitude and latitude increment, longitude and latitude number, i.e. grid width and height number, grid value block number; the structure design of the grid table (grid): gridID, belonging datasetID (datasetID), measured physical quantity, time, height (not limited to height, but also depth, pressure, level, etc.); the structure design of the grid data table (data): belonging gridID, grid block serial number, grid block data.

[0010] The grid block data in the grid data table (data) is the grid array data. The large array composed under each grid is split into continuous small arrays for storage. For example: a 100*100 grid, a total of 10000 grid point values, split into groups of 2500, and the actual data table will store four rows of data, each row records the belonging gridID, grouping serial number, and grouping array.

[0011] (2) Create a table association relationship, one dataset corresponds to multiple grid plane data, each grid plane data is marked with a datasetID, and each grid plane data corresponds to multiple grid point arrays.

[0012] (3) Create a storage process, datasetID and gridID are used as storage process input parameters, the process method retrieves dataset table metadata information through datasetID, calculates the longitude and latitude of each grid point through the longitude and latitude range, increment data and count of the metadata information, and inserts into a temporary table; the process method retrieves grid point array values in the grid data table through gridID; the dynamic view containing longitude, latitude and corresponding data values is generated by joining the longitude and latitude data in the temporary table with the processed result data in the grid data table.

[0013] Wherein the creation of temporary table (temp_ncplace), record ID, longitude, latitude. ID value is obtained by calling the sequence generator. Longitude and latitude are obtained by double loop through the index value of longitude and latitude.

[0014] Latitude = Latitude Increment * i + Latitude Min (i Latitude index, range from 0 to Latitude number - 1)

[0015] Longitude = Longitude Increment * j + Longitude Min (j Longitude index, range from 0 to Longitude number - 1)

[0016] Wherein the dynamic view, according to the input parameter grid ID from the grid data table (data) to extract the grid array data (datas), and through a series of operations such as data processing, hierarchical sequence construction, string segmentation, hierarchical matching, finally generate temporary data column temp_val for matching, while recording the row number ROWNUM of each data segmentation and its belonging level lvl; the generated temporary data column temp_val and the temporary table temp_ncplace containing longitude and latitude data are associated, and the data is matched piece by piece through the row number ROWNUM, ensuring that each longitude and latitude record corresponds to the corresponding segmented data value; dynamic view contains the unique identification id of each record, longitude and latitude information, and the data value val calculated by data processing.

[0017] 2. According to the storage structure design of dameng database, custom programming is used to parse the data of NetCDF file. Mainly includes:

[0018] (1) parse dataset information: open NetCDF file, construct dataset object. Extract coordinate dimension variable, including longitude, latitude, time, height, etc. (not limited to height), and calculate the data boundary range according to the longitude and latitude information of coordinate dimension.

[0019] Minimum value calculation: coordinate_min = start value - (increment / 2)

[0020] Maximum value calculation: coordinate_max = coordinate_min + increment * (coordinate axis size)

[0021] The above data information is integrated into the structured dataset format according to the dataset table (dataset) attribute.

[0022] (2) Analyze grid information: Extract data dimension variables, i.e., physical quantities (such as temperature, salinity, rainfall, etc.). Traverse the data variables and extract them according to their corresponding dimensions, including physical quantities, time, and height (not limited to height). Integrate the above data information into a structured dataset format according to the grid table attributes.

[0023] (3) Analyze grid data information: Extract data dimension variables, i.e., physical quantities (such as temperature, salinity, rainfall, etc.). Traverse the data variables and extract them according to their corresponding dimensions. For three- and four-dimensional data containing time or height (not limited to height) dimensions, use the slicing method to extract them. For dimensional data that does not contain time or height, read them all directly. The slicing method:

[0024] getDataOfSlice(Variable variable,Integer positionOf0,IntegerpositionOf1)

[0025] Extracts slices of a specified dimension from 3D or 4D variable data. For 3D data, use the positionOf0 parameter for slicing; for 4D data, use both the positionOf0 and positionOf1 parameters. The time and height subscripts are used as slicing parameters for data processing. After reading the sliced ​​data, it converts it into a one-dimensional array using dimensionality reduction. The data is then divided into arrays of equal lengths based on the number of grid blocks. The number of grid blocks, the block sequence, and the grid point array values ​​are then integrated into a structured dataset format according to the grid point data table (data) attributes.

[0026] Secondly, the present invention provides a system for parsing and storing NetCDF ocean data based on a domestic platform. The system integrates the access, parsing, integration, storage, visualization and system deployment of NetCDF format ocean data under the domestic platform to form a complete integrated solution.

[0027] 1. Select import NetCDF file through data interface, file information includes temperature, salinity, sea current, wind speed, wind direction and other marine environmental element information. Call backend data parsing method, record file parsing progress and parsing state through front and rear real-time communication and update to front page in real time. The parsed result data is formatted and converted according to the database storage structure and integrated into a unified complete data result set, and stored according to the preset database structure. The system page displays the parsed result data. The result data supports data list display and graphical display. The graphics include heat map, line chart and vector diagram. The system can be automatically processed through data listening. Start listening mode, real-time listen to NetCDF file in specified position, complete automatic reading, parsing and storage operation of listening file. The page displays the listening message and data processing state in real time during processing.

[0028] 2. System deployment of the domestic platform: install Kylin operating system, dream database, Nginx and Java Runtime Environment (JRE) on the server; create database tables, views and stored procedures; compile and package Java code into JAR files using Maven, and start the backend service on the Kylin operating system; compile the Vue.js front-end program using npm, and deploy the generated static files to the specified directory. Set the service port, static resource path and proxy rules. Start nginx as the front-end service.

[0029] The beneficial effects of the present application are:

[0030] 1. By optimizing the parsing algorithm and process, higher flexibility and automation degree of parsing are realized, and multi-source NC files can be efficiently parsed without manual intervention.

[0031] 2. By referring to the characteristics of the original grid data storage method, a database structure suitable for the characteristics of multi-source marine environmental data is designed. An efficient and reliable mass data storage solution is provided, which maximizes the performance advantages of the database and improves the storage and query efficiency of the data.

[0032] 3. The present application stores the parsed data into the dream database, and through the efficient indexing and querying mechanism of the database, the data access speed and processing efficiency are significantly improved, making the utilization of marine environmental data more efficient.

[0033] 4. The present application is deployed on a domestic platform, integrating data parsing, processing and storage, fully complying with the national information system localization strategy. Through this integrated solution, not only the self-controllability and data security of the system are improved, but also the processing flow of marine environmental data is simplified, meeting the actual business needs of data integration, quality control and business development. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a method and system flow diagram of NetCDF marine data parsing and storage provided by the present application;

[0035] Figure 2 is a database storage design topology provided by the present application;

[0036] Figure 3 is a java parsing NetCDF marine data flow diagram provided by the present application;

[0037] Figure 4 is a system workflow diagram of NetCDF marine data parsing and storage provided by the present application; DETAILED DESCRIPTION

[0038] In order to make the technical solutions of the present application clearer, the present application will be further described below in combination with the drawings. The present application is specifically implemented in the following steps:

[0039] Figure 1 is a method and system flow diagram of NetCDF marine data parsing and storage provided by the present application. It includes the following steps:

[0040] 1. S100, based on the data characteristics of the NetCDF file, a dream database storage scheme is designed, including table structure design, table association relationship, query analysis storage process.

[0041] In the S100 embodiment, it specifically includes the following steps:

[0042] (1) Table structure design: the present application provides a database design topology. As shown in Figure 2 , the storage design is designed according to the storage structure of the dream database of the original file format, and three-layer data tables are constructed. A dataset table (dataset) is created, which mainly describes general metadata. Table attribute fields are designed, including datasetID, grid starting longitude and latitude, grid longitude and latitude increment, grid width and height number, i.e. longitude and latitude number, grid value block number; a grid table (grid) is created, which mainly describes dimension metadata, and table attribute fields are designed, including gridID, belonging datasetID (i.e. datasetID), physical quantity, time, height (which can also be depth, pressure, level, etc.); a grid data (data) is created, which is stored in an array type, and table attribute fields are designed, including dataID, belonging gridID (i.e. gridID), grid data block serial number, and grid block data.

[0043] (2) Create table association: set the primary key of the dataset table as datasetID; set the primary key of the grid table as gridID, and the foreign key references datasetID in the dataset table; set the primary key of the grid data table as dataID, and the foreign key references gridID in the grid table.

[0044] (3) Create a storage procedure: create a storage procedure for three-layer table data integration. Given datasetID, gridID, get the grid point value under the grid. Corresponding to the measured physical quantity, time, and height in the actual business scenario. The original design form of the grid point file is retained.

[0045] Storage procedure creation steps:

[0046] Ensure that the database creates a temporary table temp_ncplace. Record ID, longitude, latitude. ID value is obtained by calling the sequence

[0047] Generator gets;

[0048] ① Define input parameters (datasetID: unique identifier of dataset; gridID: unique identifier of grid data)

[0049] ② Declare variables (used to store intermediate results)

[0050] ③ Get the latitude and longitude range, increment, and count of the dataset from the dataset table through the input parameter datasetId, store them in variables, and traverse the index values of latitude and longitude through double-loop, calculate the latitude and longitude of each grid point, and insert the result into the temporary table (temp_ncplace).

[0051] Latitude = latitude increment * i + latitude minimum value (i latitude index, range from 0 to latitude number - 1)

[0052] Longitude = longitude increment * j + longitude minimum value (j longitude index, range from 0 to longitude number - 1)

[0053] ④ Create dynamic view V_NC_GRID_DATA.

[0054] Dynamic view includes:

[0055] a) Extract grid point array data datas from the grid data table (data) according to the input parameter grid_id.

[0056] b) The datas field contains multiple data values separated by commas. By calculating the number of commas in each record, determine the number of data values contained in the record, and store it in the temporary field cnt.

[0057] c) Create a level sequence lvl with values ranging from 1 to the maximum number of data values in the record (max_len). The maximum number is obtained by calculating the maximum number of commas in all relevant records plus 1.

[0058] d) Use the SUBSTR function in combination with the INSTR function to split the data values in the datas field by commas. For each level lvl, extract the corresponding substring and use it as the value of the temp_val field. Each row of the temp_val field corresponds to a specific data value in the datas field.

[0059] e) Assign a unique row number to each extracted temp_val field by using ROWNUM and arrange the row numbers in order with the grid data block number field in the data table. The final result is an intermediate table containing the ROWNUM, gridID, grid data block number, lvl, and temp_val fields, with the temp_val field holding each data value split from the datas field.

[0060] f) Join the generated temporary data column temp_val intermediate table with the temporary table temp_ncplace containing the latitude and longitude information, and match the data row by row through the row number ROWNUM to ensure that each latitude and longitude data corresponds to the corresponding split data value.

[0061] ⑤ Execute and commit the transaction, dynamically create a dynamic view, and ensure that all operations of the stored procedure are persisted to the dream database. Finally, retrieve the required data by querying the view.

[0062] 2. S200, based on the above storage design scheme, structured custom data parsing, parsing from three levels of general metadata, grid information, and grid point data layer by layer.

[0063] In the S200 embodiment, the present application provides a flowchart for parsing a NetCDF file in java. As shown in Figure 3 , the details are as follows:

[0064] (1) Parse the first layer of data, map the dataset: the main steps include uploading the NetCDF file to save to a specified path. Open the NetCDF file and construct the dataset, extract the coordinate dimension variable (the coordinate dimension generally includes longitude, latitude, time, and height, but is not limited to height), and calculate the NetCDF data boundary range through the latitude and longitude information of the coordinate dimension variable.

[0065] Minimum value calculation: coordinate_min = start value - (increment / 2)

[0066] Max calculation: coordinate_max = coordinate_min + increment * (coordinate axis size)

[0067] According to the dataset of dameng database storage design table, integrate metadata and map to structured dataset format.

[0068] (2) Analyze the second layer data, map dataset grid: extract data dimension variable (data dimension variable is physical quantity such as temperature, salinity, rainfall, etc.), traverse data dimension variable, extract the corresponding Dimension of each data variable through analysis and judgment. According to the dataset of dameng database storage design table, integrate the required dimension variables and map to structured dataset format.

[0069] (3) Analyze the third layer data, map dataset data: extract data dimension variable (data dimension variable is physical quantity such as temperature, salinity, rainfall, etc.), traverse data dimension variable, extract the corresponding Dimension of each data variable through analysis and judgment. For data variables including time dimension or height dimension (not limited to height dimension), data reading and extraction are carried out through slicing method. That is, three-dimensional or four-dimensional data including (longitude, latitude, time or height) in Dimension. Core slicing method:

[0070] getDataOfSlice(Variable variable, Integer positionOf0, Integer positionOf1)

[0071] Parameter variable is a specific variable.

[0072] Parameter positionOf0 if it is 3D data, the parameter represents the 0th dimension slice number; if it is 4D data, the parameter represents the 1st dimension slice number.

[0073] Parameter positionOf1 if it is 3D data, the parameter will not be used; if it is 4D data, the parameter represents the 0th dimension slice number.

[0074] Return value resultData if it is 3D data, take 0th dimension positionOf0 layer slice; if it is 4D data, take 0th dimension positionOf1 layer, 1st dimension positionOf0 layer slice.

[0075] For example, three-dimensional variable variable, read the 5th layer data slice of the 0th dimension:

[0076] getDataOfSlice(variable, 5, null);

[0077] Four-dimensional variable variable, read the data slice of the 5th position of the 0th dimension and the 10th position of the 1st dimension: getDataOfSlice(variable, 5, 10);

[0078] Finally, the multi-dimensional array data read is reduced to one-dimensional array data through the processing operations of obtaining the number of dimensions of the variable variable.getRank(), reading data variable.read(), and specifying the read data range Rang. According to the number of grid value blocks, a plurality of equal length arrays are cut, and finally the grid point data table grid structure is mapped to the structured data set format. The above three data sets are integrated into new mapping data. The parsing is completed and the NetCDF file is closed.

[0079] 3. S300, based on the above Dameng database storage design and NetCDF custom data parsing method, a set of integrated full-process integrated system solution from data access, parsing, storage to query analysis under the domestic platform is realized.

[0080] In the S300 embodiment, the following is described in detail:

[0081] (1) System flow: the system workflow diagram of the NetCDF marine data parsing and storage provided by the application. As shown in Figure 4 , mainly includes data access, data parsing, data integration, data storage, data visualization. The system selects marine environmental data (such as temperature, salinity, sea current, etc.) in NetCDF format from the local. The file is parsed and processed by calling the backend parsing method, and the parsed result data is formatted, converted, integrated according to the database storage structure design, and finally the data result set is stored into the Dameng database. By calling the Dameng database storage process and query view, the system will display the parsed result data in the form of data list and graphical visualization. Among them, the data parsing process, the backend defines global state variables, the front end sets the timer setInterval, the backend parsing state value is obtained in time, and the front end presents the real-time parsing progress and the real-time update of the parsing state.

[0082] Among them, the data visualization display part selects dataset file, measurement element, time, height as input parameters to call and execute the storage process, and then retrieves and displays the data through the query view. (Selecting dataset file can determine the datasetID of dataset table, selecting measurement element, time, height can determine the gridID of grid table)

[0083] The graphical visualization includes a heat map, a line chart, and a vector diagram.

[0084] The system supports data monitoring, opens a monitoring mode, and monitors a specified NetCDF file in real time. The system automatically reads, analyzes, stores and the like the monitored file.

[0085] (2) The NetCDF marine data analysis method and system of the application is deployed and used on a domestic platform. The main implementation steps include:

[0086] ① Environment preparation: install Kylin operating system, Dream database, Nginx and Java Runtime Environment (JRE) on the server.

[0087] ② Database configuration: create tables, stored procedures and views according to the above database structure design.

[0088] ③ Backend deployment: use Maven to compile and package the backend Java code into a JAR file, and place the JAR file in the specified directory of the Kylin operating system. Start the backend service by using the "java-jar / path / to / your-app.jar" command.

[0089] ④ Front-end deployment: use npm to compile and build the Vue.js project, and generate static files to be packaged into a specified directory.

[0090] ⑤ Configure Nginx: correctly set the service port, front-end project static resource file path and the like, and configure the proxy rules to forward the API request to the backend service. Reload the Nginx configuration by using the "nginx-s reload" command and start the front-end service.

[0091] ⑥ Deployment verification: enter the system homepage address in the browser address bar to enter the system homepage, and the deployment is successful.

Claims

1. A NetCDF marine data parsing and storage method based on a localization platform, characterized in that: According to the data characteristics of the NetCDF file, the storage structure of the Dream database is designed, three-layer data tables are constructed, the table association relationship is created, the storage process is designed, and the data of the NetCDF file is parsed by self-defined programming combined with the storage structure design scheme; The specific steps of constructing three-layer data tables include: The design structure of the dataset table includes datasetID, grid starting longitude and latitude, grid longitude and latitude increment, longitude and latitude number, and grid value block number; The design structure of the grid table includes gridID, datasetID, measured physical quantity, time, and height; The design structure of the grid data table includes gridID, grid block serial number, and grid block data; The steps of creating a table association relationship include: Each dataset corresponds to multiple grid plane data, and the grid plane data is associated with the dataset through datasetID; Each grid plane data corresponds to multiple grid point arrays, and the grid point data is associated with the grid through gridID; The steps of designing the storage process include: The storage process takes datasetID and gridID as input parameters, retrieves metadata information including longitude and latitude range and increment information from the dataset table through datasetID; According to the longitude and latitude range and increment information of the metadata, the longitude and latitude of each grid point are calculated, and the results are inserted into a temporary table; The steps of creating the temporary table include: ID, longitude, and latitude information, wherein ID is generated by calling a sequence generator; The longitude and latitude are obtained by traversing the index values of longitude and latitude through double loops, and the calculation formula is: Longitude = longitude increment * j + minimum longitude value, j is the longitude index, ranging from 0 to longitude number - 1; Latitude = latitude increment * i + minimum latitude value, i is the latitude index, ranging from 0 to latitude number - 1; The grid point array value is extracted from the grid data table through gridID; The longitude and latitude data in the temporary table are connected with the data in the grid data table to generate a dynamic view containing longitude, latitude, and corresponding data values; The steps of creating the dynamic view include: Extract the grid point array data datas from the grid data table according to the input parameter gridID; The datas field contains multiple data values separated by commas, the number of data values contained in each record is determined by counting the number of commas in each record, and stored in the temporary field cnt; Create a hierarchical sequence lvl, whose value ranges from 1 to the maximum number of data values in the record; the maximum number is obtained by adding 1 to the maximum number of commas in all related records; Use the SUBSTR function combined with the INSTR function to separate the data values in the datas field by commas; for each level lvl, extract the corresponding substring and use it as the value of the temp_val field; each row of the temp_val field corresponds to a specific data value in the datas field; ​ A unique row number is assigned to each extracted temp_val field, and these row numbers are arranged in order with the grid data block number field in the grid data table; an intermediate table containing row numbers, gridID, grid data block number, lvl and temp_val fields is obtained, and the temp_val field saves each data value separated from the datas field; The generated intermediate table is connected with the temporary table containing latitude and longitude information, and the data is matched piece by piece through the row number, ensuring that each latitude and longitude data corresponds to the corresponding segmented data value; Execute and commit the transaction, and dynamically create a dynamic view.

2. A NetCDF marine data parsing and storing system based on a localization platform, characterized in that, The method and system for parsing and storing NetCDF marine data based on a localization platform, which integrates the access, parsing, integration, storage, visualization of NetCDF format marine data and system deployment under the localization platform, form a complete integrated solution: The system imports NetCDF files containing temperature, salinity, sea current, wind speed through the data interface, the back-end parsing method realizes automatic processing of data, and feedbacks the parsing progress in real time; the parsing result is formatted and stored according to the preset database structure, and the front-end visualization display includes data list, heat map, line chart and vector diagram; the system supports data monitoring function, automatically reads, parses and stores the NetCDF file at the specified position; The system is deployed on the localization platform, the server runs Kylin operating system and Dream database, Maven is used to package Java back-end service, and Vue.js front-end is compiled and deployed through npm, and Nginx is used for front-end and back-end service management.

Images