Marine data analysis and storage method and system

Abstract

The invention relates to a NetCDF ocean data analysis and storage method and system based on a localization platform. According to the method, a set of storage structure adaptive to a DM database is designed according to multi-dimensional and multivariable data characteristics of a NetCDF file, and the method comprises the processes of table structure design, table association relationship and query analysis storage. Based on the storage design scheme, data analysis is customized according to three hierarchical structures of general metadata, grid information and grid point data. A variable slicing method is adopted to efficiently extract three-dimensional and four-dimensional data. The marine data analysis and storage method is further applied to a domestic platform, the system integrates a whole-process solution from data access, analysis and storage to query analysis, and efficient management and utilization of marine environment data are ensured. The technical blank of integration of multi-source NetCDF file data processing and storage under a localization platform is effectively filled, and the localization process of a national information system is further promoted.

Description

Technical Field

[0001] The present invention relates to the fields of marine environment data processing technology and database storage technology, and in particular to a NetCDF marine data parsing and storage method and system based on a domestic platform. Background Art

[0002] With the continuous development of ocean observation methods, from observation platforms to numerical simulations, marine environmental data presents obvious multi-source heterogeneous characteristics. Among them, NetCDF (Network Common Data Form, referred to as NC file) is widely used as a common scientific data format. However, these format file data need to be parsed and processed before they can be effectively used. At the same time, efficient storage and management of a large amount of parsed ocean data has also become a problem that needs to be solved urgently. At the same time, the country is vigorously promoting the localization of information systems, and it is urgent to establish efficient and reliable technical solutions on the national platform. Most of the current data processing and storage technologies are deployed on commercial operating systems and non-domestic database platforms. In the context of promoting the localization of national information systems, the integrated solution of NetCDF ocean data parsing and storage based on the national environment is still relatively scarce.

[0003] The current parsing methods for NC file format data mainly include: using the NetCDF standard library for programming parsing and using third-party middleware visualization tools. However, these methods have certain shortcomings and lack sufficient flexibility, especially before encoding parsing, it is often necessary to check the file metadata information (such as variables, attributes, dimensions, etc.).

[0004] In terms of data storage, there are not many database storage technology solutions for the analysis results of NC data. Most technicians still generally use the traditional method of directly reading gridded NetCDF data files in the development of business applications in meteorology, oceanography and other fields, lacking the means to deeply process the original observation data and efficiently store and manage them.

[0005] In the development of business applications in the marine field, there are very few integrated application systems or solutions from multi-source NC file data analysis and processing to efficient storage and warehousing. Currently, there is no product-level solution that can truly meet the needs of the industry. Summary of the invention

[0006] In view of the problems of insufficient flexibility of current NC file data analysis methods, the relative lack of data storage technology solutions, and the lack of integrated system applications under the domestic platform, the present invention has designed a set of integrated technology solutions for efficiently analyzing and processing multi-source NC files under the domestic platform, and quickly storing the analysis results in the domestic database, so as to meet the actual business needs of marine environmental data integration, quality control and application development, support data applications in marine-related fields, and promote the localization process. To achieve the above purpose, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for parsing and storing NetCDF ocean data based on a domestic platform, specifically comprising:

[0008] 1. Based on the data characteristics of the NetCDF file, design the storage structure of the DAMO database, including building data tables, creating table relationships, and creating stored procedures.

[0009] (1) Create a data table. The structure design of the dataset table (dataset): datasetID, grid starting longitude and latitude, grid longitude and latitude increment, longitude and latitude quantity (i.e. grid width and height quantity), grid value block quantity; the structure design of the grid table (grid): gridID, dataset ID (datasetID), measured physical quantity, time, height (not limited to height but also depth, pressure, level, etc.); the structure design of the grid data table (data): grid ID (gridID), grid block number, grid block data.

[0010] The grid block data in the grid data table (data) is the grid array data. The large array under each grid is split into continuous small arrays for storage. For example: a 100*100 grid has a total of 10,000 grid values, which are split into groups of 2,500. The actual data table will store four rows of data, each of which records the grid ID, group number, and group array.

[0011] (2) Create a table association relationship. One data set corresponds to multiple grid plane data. Each grid plane data is marked with the data set ID to which it belongs. Each grid plane data corresponds to multiple grid point arrays.

[0012] (3) Create a stored procedure with the dataset ID and grid ID as the stored procedure input parameters. The procedure method retrieves the dataset table metadata information through the dataset ID, calculates the longitude and latitude of each grid point through the longitude and latitude range, incremental data, and count of the metadata information, and inserts them into the temporary table; the procedure method retrieves the point array value of the grid data table through the grid ID; and generates a dynamic view containing longitude, latitude, and corresponding data values ​​by connecting the longitude and longitude data in the temporary table with the result data after processing the grid data table.

[0013] The temporary table (temp_ncplace) is created to record ID, longitude and latitude. The ID value is obtained by calling the sequence generator. The longitude and latitude are obtained by traversing the index values ​​of the longitude and latitude through a double loop.

[0014] Latitude = latitude increment × i + minimum latitude (i is the latitude index, ranging from 0 to the number of latitudes - 1)

[0015] Longitude = longitude increment × j + longitude minimum (j longitude index, ranging from 0 to longitude number - 1)

[0016] The dynamic view extracts grid array data (datas) from the grid data table (data) according to the input parameter grid ID, and finally generates a temporary data column temp_val for matching through a series of operations including data processing, hierarchical sequence construction lvl, string segmentation, and hierarchical matching, and records the row number ROWNUM of each data segmentation and the hierarchical level lvl to which it belongs; associates the generated temporary data column temp_val with the temporary table temp_ncplace containing longitude and latitude data, and realizes data matching one by one through the row number ROWNUM, ensuring that each longitude and latitude record corresponds to the corresponding segmented data value; the dynamic view contains the unique identification id of each record, the longitude and latitude information, and the data value val calculated through data processing.

[0017] 2. Based on the storage structure design of DAMO database, custom programming is used to parse the data of NetCDF files. Mainly includes:

[0018] (1) Parsing dataset information: Open the NetCDF file and construct a dataset object. Extract coordinate dimension variables, including longitude, latitude, time, altitude, etc. (not limited to altitude), and calculate the data boundary range based on the longitude and latitude information of the coordinate dimension.

[0019] Minimum value calculation: coordinate_min = starting value - (increment / 2)

[0020] Maximum value calculation: coordinate_max = coordinate_min + increment * (coordinate axis size)

[0021] Integrate the above data information into a structured dataset format according to the dataset table attributes.

[0022] (2) Analyze grid information: extract data dimension variables, that is, physical quantities (such as temperature, salinity, rainfall, etc.). Traverse the data variables and extract them according to their corresponding dimensions, including physical quantities, time, and height (not limited to height). Integrate the above data information into a structured data set format according to the grid table attributes.

[0023] (3) Analyze grid data information: Extract data dimension variables, that is, physical quantities (such as temperature, salinity, rainfall, etc.). Traverse the data variables and extract them according to their corresponding dimensions. For three-dimensional and four-dimensional data containing time or height (not limited to height) dimensions, use the slicing method to extract them, and directly read all the dimensional data that does not contain time and height. The slicing method:

[0024] getDataOfSlice(Variable variable,Integer positionOf0,IntegerpositionOf1)

[0025] Extract the slice layer of the specified dimension in the three-dimensional or four-dimensional variable data. For three-dimensional data, use the positionOf0 parameter for slicing; for four-dimensional data, use both the positionOf0 and positionOf1 parameters for slicing; use the time dimension subscript and the height dimension subscript as slicing parameters for data processing. After reading the slice data, convert the data into a one-dimensional array for storage through the dimensionality reduction method, and then determine how many equal-length array data to slice according to the number of grid value blocks. Integrate the number of grid blocks, block sequence number, and grid array value according to the grid data table (data) attributes into a structured data set format.

[0026] In the second aspect, the present invention provides a system for parsing and storing NetCDF ocean data based on a domestic platform. The system integrates the access, parsing, integration, storage, visualization and system deployment of NetCDF format ocean data under a domestic platform to form a complete integrated solution.

[0027] 1. Select to import NetCDF file through the data interface. The file information includes ocean environment element information such as temperature, salinity, ocean current, wind speed, wind direction, etc. Call the back-end data parsing method, record the file parsing progress and parsing status through real-time communication between the front and back ends, and update it to the front-end page in real time. The parsing result data is formatted and converted according to the database storage structure and integrated into a unified and complete data result set, and stored according to the preset database structure. The system page displays the parsing result data. The result data supports data list display and graphical display. The graphics include heat maps, line graphs, and vector graphs. The system can automatically process through data monitoring. Turn on the monitoring mode, monitor the NetCDF file at the specified location in real time, and complete the automatic reading, parsing and storage operations of the monitoring file. During the processing, the page displays the monitoring message and data processing status in real time.

[0028] 2. System deployment of the localized platform: Install Kylin operating system, DAMO database, Nginx and Java Runtime Environment (JRE) on the server; create database tables, views, and stored procedures; use Maven to compile and package Java code into JAR files, and start the backend service on the Kylin operating system; use npm to compile the Vue.js front-end program and deploy the generated static files to the specified directory. Set the service port, static resource path, and proxy rules. Start nginx as the front-end service.

[0029] The beneficial effects of the present invention are:

[0030] 1. By optimizing the parsing algorithm and process, higher parsing flexibility and automation are achieved, and multi-source NC files can be efficiently parsed without manual intervention.

[0031] 2. By referring to the characteristics of the original grid data storage method, a database structure that adapts to the characteristics of multi-source marine environmental data is designed. It provides an efficient and reliable mass data storage solution, maximizes the performance advantages of the database, and improves the efficiency of data storage and query.

[0032] 3. The present invention stores the parsed data in the Dameng database. Through the efficient indexing and query mechanism of the database, the data access speed and processing efficiency are significantly improved, making the use of marine environmental data more efficient.

[0033] 4. The present invention is deployed and implemented on a localized platform, integrating data analysis, processing, and storage, which fully complies with the national information system localization strategy. Through this integrated solution, not only the system's autonomous controllability and data security are improved, but also the processing flow of marine environmental data is simplified, meeting the actual business needs of data integration, quality control, and business development. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the method and system flow of NetCDF ocean data parsing and storage provided by the present invention;

[0035] Figure 2 It is a database storage design topology diagram provided by the present invention;

[0036] Figure 3 The present invention provides a Java parsing NetCDF ocean data flow chart;

[0037] Figure 4 It is a system workflow diagram of NetCDF ocean data parsing and storage provided by the present invention; DETAILED DESCRIPTION

[0038] To make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. The present invention is specifically implemented according to the following steps:

[0039] Figure 1 The method and system flow diagram of NetCDF ocean data parsing and storage provided by the present invention includes the following steps:

[0040] 1.S100, based on the data characteristics of NetCDF files, design the DAMO database storage solution, including table structure design, table association, query analysis and storage procedures.

[0041] In the S100 embodiment, the following steps are specifically included:

[0042] (1) Table structure design: The present invention provides a database design topology diagram. Figure 2 As shown, the storage design designs the storage structure of the DAMO database according to the original file format, and constructs a three-layer data table. Create a dataset table (dataset), which mainly describes general metadata. Design table attribute fields, including datasetID, grid starting longitude and latitude, grid longitude and latitude increments, grid width and height quantity, i.e. longitude and latitude quantity, and grid value block quantity; create a grid table (grid) to mainly describe dimensional metadata, design table attribute fields, including gridID, dataset ID (i.e. datasetID), physical quantity, time, height (can also be depth, pressure, level, etc.); create grid data (data) to use array type to store grid value, design table attribute fields, including dataID, grid ID (i.e. gridID), grid data block sequence number, and grid block data.

[0043] (2) Create table associations: Set the primary key of the dataset table to datasetID; set the primary key of the grid table to gridID, and the foreign key references the datasetID in the dataset table; set the primary key of the grid data table to dataID, and the foreign key references the gridID in the grid table.

[0044] (3) Create a stored procedure: The method of creating a stored procedure is used for the three-layer table data. Given a dataset ID and a grid ID, obtain the grid point value under the grid. Corresponding to the measurement value of a certain measured physical quantity, a certain time, and a certain height in the actual business scenario. The original design form of the grid point file is retained.

[0045] Steps to create a stored procedure:

[0046] Make sure the database creates a temporary table temp_ncplace to record ID, longitude, and latitude. The ID value is obtained by calling the sequence

[0047] Generator acquisition;

[0048] ①Define input parameters (datasetID: unique identifier of dataset; gridID: unique identifier of grid data)

[0049] ②Declare variables (used to store intermediate results)

[0050] ③ Get the longitude and latitude range, increment and count of the dataset from the dataset table through the input parameter datasetId and store them into variables. Through a double loop, traverse the latitude and longitude index values, calculate the longitude and latitude of each grid point, and insert the results into the temporary table (temp_ncplace).

[0051] Latitude = latitude increment × i + minimum latitude (i is the latitude index, ranging from 0 to the number of latitudes - 1)

[0052] Longitude = longitude increment × j + longitude minimum (j longitude index, ranging from 0 to longitude number - 1)

[0053] ④Create the dynamic view V_NC_GRID_DATA.

[0054] Dynamic views include:

[0055] a) Extract the grid array data datas from the grid data table (data) according to the input parameter grid_id.

[0056] b) The datas field contains multiple data values ​​separated by commas. By counting the number of commas in each record, the number of data values ​​contained in the record is determined and stored in the temporary field cnt.

[0057] c) Create a hierarchical sequence lvl whose values ​​range from 1 to the maximum number of data values ​​in a record (max_len). The maximum number is calculated by adding 1 to the maximum number of commas in all related records.

[0058] d) Use the SUBSTR function in combination with the INSTR function to split the data values ​​in the datas field by commas. For each level lvl, extract the corresponding substring and use it as the value of the temp_val field. Each row of the temp_val field corresponds to a specific data value in the datas field.

[0059] e) Assign a unique row number to each extracted temp_val field by using ROWNUM, and align these row numbers with the grid data block number field in the data table. The final result is an intermediate table containing ROWNUM, gridID, grid data block number, lvl, and temp_val fields, where the temp_val field stores each data value split from the datas field.

[0060] f) Connect the generated temporary data column temp_val intermediate table with the temporary table temp_ncplace containing longitude and latitude information, and match the data one by one through the row number ROWNUM to ensure that each longitude and latitude data corresponds to the corresponding segmented data value;

[0061] ⑤Execute and commit the transaction, dynamically create dynamic views, and ensure that all operations of the stored procedure are persisted to the Dream database. Finally, retrieve the required data by querying the view.

[0062] 2. S200, based on the above storage design, structured custom data parsing, parsing layer by layer from three levels: general metadata, grid information, and grid point data.

[0063] In the S200 embodiment, the present invention provides a flowchart of Java parsing NetCDF files. Figure 3 As shown, the details are as follows:

[0064] (1) Parse the first layer of data and map the dataset dataset: The main steps include uploading the NetCDF file and saving it to the specified path. Open the NetCDF file and construct the dataset, extract the coordinate dimension variables (the coordinate dimension generally includes longitude, latitude, time, and altitude but is not limited to altitude), and calculate the NetCDF data boundary range through the longitude and latitude information of the coordinate dimension variables.

[0065] Minimum value calculation: coordinate_min = starting value - (increment / 2)

[0066] Maximum value calculation: coordinate_max = coordinate_min + increment * (coordinate axis size)

[0067] According to the design table dataset storage of DAMO database, the metadata is integrated and mapped into a structured dataset format.

[0068] (2) Analyze the second layer of data and map the data set grid: extract data dimension variables (data dimension variables are physical quantities such as temperature, salinity, rainfall, etc.), traverse the data dimension variables, and extract the Dimension corresponding to each data variable through analysis and judgment. According to the DAMO database storage design table grid, integrate the required dimension variables and map them into a structured data set format.

[0069] (3) Parse the third layer of data and map the data set data: extract data dimension variables (data dimension variables are physical quantities such as temperature, salinity, rainfall, etc.), traverse the data dimension variables, and extract the Dimension corresponding to each data variable through analysis and judgment. For data variables that include time dimension or height dimension (not limited to height dimension), data reading and extraction are performed through slicing method. That is, Dimension includes three-dimensional or four-dimensional data (longitude, latitude, time or height). Core slicing method:

[0070] getDataOfSlice(Variable variable,Integer positionOf0,IntegerpositionOf1)

[0071] The parameter variable is a specific variable.

[0072] If the parameter positionOf0 is 3D data, this parameter represents the number of slices in the 0th dimension; if it is 4D data, this parameter represents the number of slices in the 1st dimension.

[0073] If the parameter positionOf1 is 3D data, this parameter will not be used; if it is 4D data, this parameter represents the number of slice layers in the 0th dimension.

[0074] If the return value resultData is 3D data, take the 0-dimensional positionOf0 layer slice; if it is 4-dimensional data, take the 0-dimensional positionOf1 layer and 1-dimensional positionOf0 layer slice.

[0075] For example, for a three-dimensional variable, read the fifth layer of data slices in the 0th dimension:

[0076] getDataOfSlice(variable,5,null);

[0077] Four-dimensional variable variable, read the data slice at the 5th position of the 0th dimension and the 10th position of the 1st dimension: getDataOfSlice(variable,5,10);

[0078] Finally, the multi-dimensional array data is reduced to one-dimensional array data by obtaining the number of variables (variable.getRank()), reading data (variable.read()), and specifying the read data range (Rang). According to the number of grid value blocks, multiple arrays of equal length are cut and finally mapped to a structured data set format according to the structure of the grid data table (grid). The new mapping data is integrated uniformly through the above three data sets. After the parsing is completed, the NetCDF file is closed.

[0079] 3. S300, based on the above-mentioned DAMO database storage design and NetCDF custom data parsing method, implements a full-process integrated system solution from data access, parsing, storage to query and analysis on a domestic platform.

[0080] In the S300 embodiment, the details are as follows:

[0081] (1) System flow: The system flow chart of NetCDF ocean data analysis and storage provided by the present invention. Figure 4 As shown, it mainly includes data access, data analysis, data integration, data storage, and data visualization. The system selects marine environmental data (such as temperature, salinity, current, etc.) in NetCDF format from the local. The file is parsed and processed by calling the back-end parsing method, and the parsing result data is formatted, converted, and integrated according to the database storage structure design, and finally the data result set is stored in the DAMO database. By calling the DAMO database stored procedures and query views, the system will display the parsing result data in the form of data lists and graphical visualizations. In the data parsing process, the back-end defines global state variables, the front-end sets the timer setInterval, and periodically obtains the back-end parsing status value. The front-end presents real-time parsing progress and real-time updates of the parsing status.

[0082] In the data visualization display part, select the dataset file, measurement element, time, and height as input parameters to call and execute the stored procedure, and then retrieve and display the data by querying the view. (Selecting the dataset file can determine the datasetID of the dataset table, and selecting the measurement element, time, and height can determine the gridID of the grid table)

[0083] Graphic visualization includes heat maps, line graphs, and vector graphs.

[0084] The system supports data monitoring. When the monitoring mode is turned on, the NetCDF file in the specified location can be monitored in real time. The system automatically completes operations such as reading, parsing, and storing the monitored files.

[0085] (2) The NetCDF ocean data analysis method and system of the present invention is deployed and used on a domestic platform. The main implementation steps include:

[0086] ①Environment preparation: Install Kylin operating system, DAMO database, Nginx, and Java Runtime Environment (JRE) on the server.

[0087] ②Database configuration: Create tables, stored procedures, and views according to the above database structure design.

[0088] ③ Backend deployment: Use Maven to compile and package the backend Java code into a JAR file, and place the JAR file in the specified directory of the Kylin operating system, and start the backend service through the "java-jar / path / to / your-app.jar" command.

[0089] ④Front-end deployment: Use npm to compile and build the Vue.js project, generate static files and package them into the specified directory.

[0090] ⑤Configure Nginx: Set the service port, front-end project static resource file path, etc. correctly, and configure proxy rules to forward API requests to the back-end service. Reload the Nginx configuration and start the front-end service through the "nginx-s reload" command.

[0091] ⑥Deployment verification: Enter the system homepage address in the browser address bar and enter the system homepage to confirm the deployment is successful.

Claims

1. A NetCDF ocean data parsing and storage method based on a domestic platform, characterized by: According to the data characteristics of NetCDF files, the storage structure of the DAMO database is designed, three-layer data tables are constructed, table associations are created, and storage procedures are designed. Combined with the storage structure design, custom programming is used to parse the data of the NetCDF file.

2. According to claim 1, a method for analyzing and storing NetCDF ocean data based on a domestic platform is characterized in that: The specific steps of constructing the three-layer data table include: The design structure of the dataset table (dataset): including datasetID, grid starting longitude and latitude, grid longitude and latitude increment, number of longitude and latitude (i.e. grid width and height), and number of grid value blocks; The design structure of the grid table (grid): including gridID, dataset ID (datasetID), measured physical quantity, time, height (or depth, pressure, level, etc.); The design structure of the grid data table (data): includes the grid ID (gridID), grid block serial number, and grid block data.

3. The method for analyzing and storing NetCDF ocean data based on a domestic platform according to claim 1, characterized in that: The step of creating a table association relationship includes: Each dataset corresponds to multiple grid plane data, and the grid plane data is associated with the dataset to which it belongs through datasetID; Each grid plane data corresponds to multiple grid point arrays, and the grid point data is associated with the grid to which it belongs through gridID.

4. According to claim 1, a method for analyzing and storing NetCDF ocean data based on a domestic platform is characterized in that: The steps of creating a stored procedure include: The stored procedure takes the dataset ID and grid ID as input parameters, and retrieves metadata information from the dataset table through the dataset ID, including latitude and longitude ranges, increment information, etc. Calculate the longitude and latitude of each grid point according to the latitude and longitude range and increment information in the metadata, and insert the result into the temporary table (temp_ncplace); Extract the grid array value from the grid data table by grid ID; Connect the longitude and latitude data in the temporary table with the data in the grid data table to generate a dynamic view containing longitude, latitude and corresponding data values.

5. The method for analyzing and storing NetCDF ocean data based on a domestic platform according to claim 4 is characterized in that: The steps of creating the temporary table (temp_ncplace) include: Record ID, longitude, and latitude information, where ID is generated by calling the sequence generator; The longitude and latitude are obtained by traversing the index values ​​of the longitude and latitude through a double loop. The calculation formula is: Longitude = longitude increment × j + longitude minimum (j longitude index, ranging from 0 to longitude number - 1) Latitude = latitude increment × i + minimum latitude (i is the latitude index, ranging from 0 to the number of latitudes - 1) 6. The method for analyzing and storing NetCDF ocean data based on a domestic platform according to claim 4 is characterized in that: The steps of creating the dynamic view include: Extract grid array data from the grid data table according to the grid ID, perform data processing, generate a temporary data column (temp_val) for matching through hierarchical sequence construction, string segmentation and hierarchical matching, and record the row number (ROWNUM) and hierarchical information (lvl) of each data segmentation; Associate the generated temporary data column with the temporary table containing longitude and latitude, and ensure that each longitude and latitude record matches the corresponding segmentation data value by matching the row numbers; Finally, a dynamic view is generated, which contains the unique identification ID, longitude and latitude information of each record, and the value (val) calculated through data processing.

7. A NetCDF ocean data analysis and storage system based on a domestic platform, characterized in that: The system integrates the access, analysis, integration, storage, visualization and system deployment of NetCDF format ocean data on a domestic platform, forming a complete integrated solution.

8. The NetCDF ocean data analysis and storage system based on a domestic platform according to claim 7 is characterized in that: The system imports NetCDF files containing information such as temperature, salinity, ocean currents, and wind speed through a data interface. The back-end parsing method realizes automatic data processing and provides real-time feedback on the parsing progress. The parsing results are formatted and stored according to the preset database structure, and the front-end visualization includes data lists, heat maps, line graphs, and vector graphs. The system supports data monitoring functions, automatically reading, parsing, and storing NetCDF files in specified locations.

9. The NetCDF ocean data analysis and storage system based on a domestic platform according to claim 7, characterized in that: The system is deployed on a domestic platform. The server runs the Kylin operating system and the DAMO database. Maven is used to package the Java backend service, and the Vue.js front end is compiled and deployed through npm, combined with Nginx for front-end and back-end service management.

Images