Integrated processing system and method for marine data stored in netCDF file

Abstract

The application relates to the technical field of marine data processing, and particularly discloses an integrated processing system and method for marine data stored in a NetCDF file, which integrates and realizes the functions of slicing, gridding processing and contour line generation of marine data in the NetCDF file, supports data visualization, can generate data visualization graphs and contour line visualization graphs of specified depth layers, has the function of slicing marine data in the NetCDF file, can slice marine data in single or multiple NetCDF files, has the format conversion function of converting the NetCDF file into a TIFF file, can convert single or multiple NetCDF files into TIFF files, can automatically draw contour line graphs of single or multiple NetCDF files based on extracted data, and adopts parallel processing when processing the multiple NetCDF files, and view control and operation guidance can be provided in the processing process.

Description

Technical Field

[0001] This application relates to the field of marine data processing technology, and specifically discloses an integrated processing system and method for marine data stored in NetCDF files. Background Technology

[0002] With the rapid development of big data, remote sensing, and buoy monitoring technologies, integrated air-space-ground monitoring of various marine data has become possible, leading to a surge in the volume of marine data and a proliferation of diverse data formats. Therefore, research on marine data processing, computation, and rapid visualization has become crucial for researchers in marine data mining and analysis, rapid development of marine resources, and the advancement of the marine sector.

[0003] NetCDF, as one of the mainstream formats for existing marine data storage, possesses excellent data management capabilities, but researchers often lack an intuitive understanding of the marine data-related information elements it contains. Therefore, when acquiring marine data stored in this format, data preprocessing is necessary to improve data visualization efficiency. Existing methods for preprocessing marine data include:

[0004] (1) The ocean data is processed by grid slicing and the sliced ​​ocean data is rasterized. The main process is as follows: The ocean data is sliced ​​by grid in the longitude and latitude directions with a certain step size through the traditional serial method. At the same time, the sliced ​​data is rasterized through the GDAL library, so as to provide a certain basis for researchers to call it in the front end of the system.

[0005] (2) Another example is Chinese patent CN119254386B, which discloses a satellite-based method and system for processing ocean observation data. It uses an ocean observation device to acquire ocean observation data within a set time period and formats the observation data to obtain multiple slice data.

[0006] While the two methods described above can compress the amount of ocean observation data and reduce the amount of data that ocean observation devices need to transmit, thereby improving communication efficiency, they still have the following shortcomings:

[0007] The entire process of "reading, processing, transforming, and visualizing" has not been formed. It requires the combination of multiple independent tools, which is cumbersome and has low data flow efficiency. The marine data slicing process still uses a serial method, which is time-consuming when processing massive amounts of data. The rasterization processing of marine data relies on the GDAL library for data transformation, but its transformation efficiency is low when dealing with massive amounts of marine data. At the same time, the single solution has a narrow functional scope and lacks the display of the spatial distribution characteristics of marine data.

[0008] This invention provides an integrated processing system and method for ocean data stored in NetCDF files, in order to solve the above-mentioned problems. Summary of the Invention

[0009] To achieve the above objectives, the basic solution of this invention provides an integrated processing system for ocean data stored in NetCDF files, including:

[0010] The file opening module is used to receive user instructions to open and parse the NetCDF file containing ocean data to be processed, and extract the following ocean data information: variable information, dimension information, and attribute information;

[0011] The data visualization control module is used to receive the variable names, coordinate scale spacing, and contour line interval values ​​configured by the user for the NetCDF file to be processed, and to realize data visualization, contour line visualization, and local saving based on the extracted marine data information and the matplotlib library.

[0012] The contour generation module receives the variable names, contour interval values, time index, and depth index configured by the user for the NetCDF file to be processed. After rasterizing the ocean data, it generates vector contour data, visualizes the variable contours, and saves them locally in Shapefile format.

[0013] The slicing module receives the latitude and longitude slice size, compression level, and output directory configured by the user for the NetCDF file to be processed, performs data slicing on the ocean data in the NetCDF file, and saves it as a new NetCDF file after verifying the valid data.

[0014] The format conversion module receives the variable names, EPSG coordinate system codes, time index, depth index, and TIFF file output directory configured by the user for the NetCDF file to be processed. It extracts the specified variables, time slices, and depth layer 2D ocean data from the NetCDF file, and then converts it into a TIFF file that retains the original latitude and longitude grid structure and geographic coordinate association based on the GDAL library.

[0015] The view control module is used to manage and process logs;

[0016] The help module provides operation guidance.

[0017] Furthermore, the variable information of the ocean data includes a description of the physical meaning of the variable, the data unit, the length unit, and the effective value range. The dimensional information of the ocean data includes time, longitude, latitude, and depth.

[0018] Furthermore, the contour line generation module includes a single NetCDF file processing unit and multiple NetCDF file processing units;

[0019] The single NetCDF file processing unit, based on the user-configured single NetCDF file path, contour output directory, variable name, contour interval value, time index, and depth index, calls the contour calculation function to generate raster data from the ocean data in the NetCDF file through the GDAL library, and then calculates contours based on the raster data and saves them in SHP format;

[0020] The multiple NetCDF file processing units, based on the user-configured directories of multiple NetCDF files, contour output directories, variable names, contour interval values, time indexes, and depth indexes, call contour calculation functions in a parallel computing manner. Simultaneously, they generate raster data from the ocean data in all NetCDF files using the GDAL library, then calculate contours based on the raster data and save them in SHP format.

[0021] Furthermore, the processing procedure of the single NetCDF file processing unit is as follows:

[0022] Based on user-configured time and depth indexes, the ocean data of specified time and depth layers in the NetCDF file are filtered and the target variables are read. Invalid data and values ​​outside the valid range are identified and marked as missing values. Then, the preprocessed variables and latitude and longitude data are read by the GDAL library to generate a GeoTIFF file with EPSG:4326 geographic coordinate reference. The ContourGenerate function of the GDAL library is called to compare the raster values ​​with the user-defined contour line level pixel by pixel to generate vector contour lines. Finally, the elevation field of the Shapefile file is created to store the ocean data values ​​corresponding to the contour lines, and the contour lines are written as LineString geometric objects and the elevation field attribute values ​​together and saved in SHP format.

[0023] Furthermore, the slicing module includes a single NetCDF file slicing unit and multiple NetCDF file slicing units;

[0024] The individual NetCDF file slicing unit, based on the user-configured individual NetCDF file path, slicing output directory, latitude and longitude slice size, and compression level, calls the slicing calculation function to parse the metadata of the ocean data in the individual NetCDF file, divide the latitude and longitude grid, and slice iteratively, and saves the sliced ​​NetCDF file to the slicing output directory;

[0025] The multiple NetCDF file slicing units, based on the user-configured directories of multiple NetCDF files, slicing output directories, latitude and longitude slice sizes, and compression levels, call slicing calculation functions in a parallel computing manner to synchronously parse the metadata of the ocean data in each NetCDF file, divide the latitude and longitude grids, and cyclically slice the data, saving the sliced ​​NetCDF files to the slicing output directory.

[0026] Furthermore, the individual NetCDF file slicing unit processes each grid cell sequentially in a serial loop manner, while the multiple NetCDF file slicing units adopt a multi-level parallel computing architecture. Multiprocessing.Pool is used to achieve multi-process parallel computing, automatically allocating different NetCDF files to multiple identical processes, and processing them simultaneously based on the processing of the individual NetCDF file slicing unit.

[0027] Furthermore, the format conversion module includes a single file format conversion unit and multiple file format conversion units;

[0028] The single file format conversion unit, based on the user-configured single NetCDF file path, TIFF file output directory, variable name, EPSG coordinate system code, and time index, calls the TIFF file calculation function to parse the metadata of the ocean data in the NetCDF file, calculate geographic parameters, iteratively process the depth layer data and extract 2D data, and writes the 2D data into the TIFF file and saves it to the output directory through the GDAL library;

[0029] The multiple file format conversion units, based on the directories containing multiple NetCDF files, the output directory of TIFF files, variable names, EPSG coordinate system codes, and time indexes configured by the user, call the TIFF file calculation function in a parallel computing manner. They simultaneously parse the metadata of the ocean data in the NetCDF files, calculate geographic parameters, process the depth layer data in a loop, and extract 2D data. The 2D data is then written to the TIFF file and saved to the output directory through the GDAL library.

[0030] Furthermore, when converting the format of multiple NetCDF files, all NetCDF files are recursively searched and a thread pool is created for task allocation. Multiple worker threads execute the format conversion of a single NetCDF file in parallel based on the processing of a single file format conversion unit.

[0031] The basic solution of this invention also provides an integrated processing method for ocean data stored in NetCDF files, applied to the aforementioned integrated processing system for ocean data stored in NetCDF files, including the following steps:

[0032] The file opening module opens and parses the NetCDF file containing ocean data according to user instructions, and extracts the following ocean data information: variable information, dimension information, and attribute information.

[0033] The data visualization control module receives the variable names, coordinate scale spacing, and contour line interval values ​​configured by the user for the NetCDF file to be processed, and performs data visualization, contour line visualization, and local saving based on the extracted marine data information and the matplotlib library.

[0034] The contour generation module receives the variable names, contour interval values, time index, and depth index configured by the user for the NetCDF file to be processed. After rasterization of the ocean data, vector contour data is generated, and the variable contours are visualized and saved locally in Shapefile format.

[0035] The slicing module receives the latitude and longitude slice size, compression level and output directory configured by the user for the NetCDF file to be processed, performs data slicing on the ocean data in the NetCDF file, and saves it as a new NetCDF file after verifying the valid data.

[0036] The format conversion module receives the variable names, EPSG coordinate system codes, time index, depth index, and TIFF file output directory configured by the user for the NetCDF file to be processed. It extracts the specified variables, time slices, and depth layer 2D ocean data from the NetCDF file, and then converts it into a TIFF file that retains the original latitude and longitude grid structure and geographic coordinate association based on the GDAL library.

[0037] Logs are managed and processed by the view control module;

[0038] Operation guidance is provided by the help module.

[0039] The principle and effect of this solution are as follows:

[0040] 1. Compared with existing technologies, this invention integrates and implements the functions of slicing, rasterization, and contour generation for marine data stored in NetCDF files. It solves the problem that traditional tools for processing marine data stored in NetCDF files lack rapid slicing, rapid rasterization calculation, and rapid contour generation capabilities. Furthermore, due to the complexity and dynamic nature of marine environmental data, which exhibits multidimensional characteristics, this invention can achieve multidimensional data decomposition during rasterization and contour generation. It can transform marine data from three or four dimensions into two-dimensional data, and then use the GDAL library to generate raster images and contour SHP images from this two-dimensional data.

[0041] 2. Compared with the prior art, the present invention can also realize the custom setting of parameters. For different types of ocean data, the data type to be processed can be customized. When slicing, the size of the latitude and longitude slice range can be customized. When generating contour lines from NetCDF files, the spacing of contour lines can be customized.

[0042] 3. Compared with the prior art, the present invention simultaneously realizes parallel processing of marine data in NetCDF files. When performing the following processing on multiple NetCDF files: contour line generation, slicing, and format conversion, parallel computing is used to improve data processing efficiency. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A schematic diagram of an integrated processing system for marine data stored in NetCDF files, as proposed in an embodiment of this application, is shown. Detailed Implementation

[0045] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0046] For an integrated processing system for ocean data stored in NetCDF files, implement, for example Figure 1 As shown, it includes a file opening module, a data visualization control module, a contour line generation module, a slicing module, a format conversion module, a view control module, and a help module.

[0047] Users can individually call any one of the following modules: data visualization control module, contour generation module, tiling module, and format conversion module; or they can call multiple modules simultaneously. The core functional module cluster consists of these modules, and each module is invoked through this cluster.

[0048] In this embodiment, the file opening module is used to receive user instructions to open and parse the NetCDF file containing ocean data to be processed, and extract the following ocean data information: variable information, dimension information, and attribute information;

[0049] The data visualization control module is used to receive the variable names, coordinate scale spacing, and contour line interval values ​​configured by the user for the NetCDF file to be processed, and to realize data visualization, contour line visualization, and local saving based on the extracted information and the matplotlib library.

[0050] The contour generation module receives the variable names and contour interval values ​​configured by the user for the NetCDF file to be processed, generates vector contour data after rasterizing the ocean data, visualizes the variable contours, and saves them locally in Shapefile format.

[0051] The slicing module receives the latitude and longitude slice size, compression level, and output directory configured by the user for the NetCDF file to be processed, performs data slicing on the ocean data in the NetCDF file, and saves it as a new NetCDF file after verifying the valid data.

[0052] The format conversion module receives the variable names, EPSG coordinate system codes, time index, depth index, and TIFF file output directory configured by the user for the NetCDF file to be processed. It extracts the specified variables, time slices, and depth layer 2D ocean data from the NetCDF file, and then converts it into a TIFF file that retains the original latitude and longitude grid structure and geographic coordinate association based on the GDAL library.

[0053] The view control module is used to manage and process logs;

[0054] The help module provides operation guidance.

[0055] In this embodiment, the types of ocean data that can be processed include: ocean current data, sea temperature data, ocean wave data, salinity data, water level data, etc.

[0056] The extraction of variable, dimensional, and attribute information is based on the processing of metadata and data structures of ocean data in NetCDF files. Attribute information specifically refers to the source, time, standard, and range of the ocean data. Extracted variable information from the ocean data includes the physical meaning description of the variable, the data unit (units), the length unit (units_long), and the valid value range. The physical meaning description of the variable includes the variable's long name (long_name) and standard name (standard_name), and the valid value range includes valid_min and valid_max values. Specific variable information types include ocean elements such as temperature, salinity, and current velocity. Extracted dimensional information from the ocean data includes time, longitude, latitude, and depth.

[0057] The data visualization control module can receive user configuration parameters including the file path and depth index of the NetCDF file to be processed. After receiving the user configuration parameters, the data visualization control module calls the matplotlib library integrated within the module to generate data visualizations based on the extracted ocean data. It can also generate contour line visualizations by concatenating variables with the same values. Both the generated data visualizations and contour line visualizations are saved locally in PNG format.

[0058] The contour line generation module includes a single NetCDF file processing unit and multiple NetCDF file processing units.

[0059] A single NetCDF file processing unit, based on the user-configured single NetCDF file path, contour output directory, variable name, and contour interval value, calls the contour calculation function to generate raster data from the ocean data in the NetCDF file through the GDAL library, and then calculates contours based on the raster data and saves them in SHP format;

[0060] Multiple NetCDF file processing units, based on user-configured directories containing multiple NetCDF files, contour line output directories, variable names, and contour line interval values, call contour line calculation functions in a parallel computing manner. Simultaneously, they generate raster data from ocean data in all NetCDF files using the GDAL library, then calculate contour lines based on the raster data and save them in SHP format.

[0061] Specifically, it includes the following processes:

[0062] Based on user-configured time and depth indexes, the ocean data of specified time and depth layers in the NetCDF file are filtered and the target variables are read. _FillValue and values ​​outside the valid range are identified and replaced with NaN, where NaN indicates a missing value and _FillValue indicates invalid data. Then, the preprocessed variables and latitude / longitude data are read using the GDAL library to generate a GeoTIFF file with EPSG:4326 geographic coordinates. The ContourGenerate function of the GDAL library is called to compare raster values ​​pixel-by-pixel with the user-defined contour level to generate vector contour lines. Finally, an elevation field is created in the Shapefile file to store the ocean data values ​​corresponding to the contour lines, and the contour lines are written as LineString geometric objects along with the elevation field attribute values ​​and saved in SHP format.

[0063] The slicing module includes a single NetCDF file slicing unit and multiple NetCDF file slicing units.

[0064] A single NetCDF file tiling unit is based on the user-configured single NetCDF file path, tiling output directory, latitude and longitude tiling size, and compression level. It calls the tiling calculation function to parse the metadata of the ocean data in a single NetCDF file, divide the latitude and longitude grid, and tiling it in a loop. The tiled NetCDF file is then saved to the tiling output directory.

[0065] Multiple NetCDF file slicing units, based on the user-configured directories containing multiple NetCDF files, slicing output directories, latitude and longitude slice sizes, and compression levels, call slicing calculation functions in a parallel computing manner. They synchronously parse the metadata of the ocean data in each NetCDF file, divide the latitude and longitude grids, and slice in a loop. The sliced ​​NetCDF files are then saved to the slicing output directory.

[0066] Furthermore, in this embodiment, the processing of a single NetCDF file slice adopts a serial loop approach to process each grid cell sequentially, while the processing of multiple NetCDF file slices adopts a multi-level parallel computing architecture. Multiprocessing.Pool is used to achieve multi-process parallel computing, automatically allocating different NetCDF files to multiple identical processes, and processing them simultaneously based on the processing procedure of each individual NetCDF file slice cell. Here, multiprocessing.Pool is a multi-process pool.

[0067] The format conversion module includes a single file format conversion unit and multiple file format conversion units.

[0068] The single file format conversion unit, based on the user-configured single NetCDF file path, TIFF file output directory, variable name, EPSG coordinate system code, and time index, calls the TIFF file calculation function to parse the metadata of the ocean data in the NetCDF file, calculate geographic parameters, iteratively process the depth layer data and extract 2D data, and writes the 2D data into the TIFF file and saves it to the output directory through the GDAL library;

[0069] Multiple file format conversion units, based on user-configured directories containing multiple NetCDF files, TIFF file output directories, variable names, EPSG coordinate system codes, and time indices, call TIFF file calculation functions in a parallel computing manner. They simultaneously parse the metadata of marine data in the NetCDF files, calculate geographic parameters, process depth layer data in a loop, extract 2D data, and write the 2D data into TIFF files and save them to the output directory using the GDAL library.

[0070] Furthermore, in this embodiment, when converting multiple NetCDF files, all NetCDF files are first recursively searched and a thread pool is created for task allocation. Multiple worker threads then execute the conversion of a single NetCDF file in parallel based on the processing of a single file format conversion unit. For single-file processing, the conversion process is executed directly. The conversion process includes, in sequence, NetCDF file structure parsing, multi-depth layer loop processing, creation of temporary TIFF files, and GDAL raster data resampling optimization.

[0071] In this embodiment, the management functions that the view control module can implement for processing logs include real-time log display, hiding the log panel according to user instructions, and clearing historical logs with one click. During the real-time log display process, the operation time, processing object, progress, and result status are included.

[0072] The help module includes an "About" section and a "User Guide" section. The "About" section stores and displays system version information and developer information, while the "User Guide" section provides specific operating steps and process guidance for each functional module.

[0073] This invention also provides an integrated processing method for ocean data stored in NetCDF files, applied to the aforementioned integrated processing system for ocean data stored in NetCDF files, comprising the following steps:

[0074] The file opening module opens and parses the NetCDF file containing ocean data according to user instructions, and extracts the following ocean data information: variable information, dimension information, and attribute information;

[0075] The data visualization control module receives the variable names, coordinate scale spacing, and contour line interval values ​​configured by the user for the NetCDF file to be processed, and realizes data visualization, contour line visualization, and local saving based on the extracted information and the matplotlib library.

[0076] The contour generation module receives the variable names, contour interval values, time index, and depth index configured by the user for the NetCDF file to be processed. After rasterizing the ocean data, it generates vector contour data, visualizes the variable contours, and saves them locally in Shapefile format.

[0077] The slicing module receives the latitude and longitude slice size, compression level and output directory configured by the user for the NetCDF file to be processed, performs data slicing on the ocean data in the NetCDF file, and saves it as a new NetCDF file after verifying the valid data.

[0078] The format conversion module receives the variable names, EPSG coordinate system codes, time index, depth index, and TIFF file output directory configured by the user for the NetCDF file to be processed. It extracts the specified variables, time slices, and depth layer 2D ocean data from the NetCDF file, and then converts it into a TIFF file that retains the original latitude and longitude grid structure and geographic coordinate association based on the GDAL library.

[0079] The view control module manages and processes logs, which specifically refers to the log information generated during the processing of ocean data in NetCDF files.

[0080] The help module provides operation guidance, which includes: system version information, developer information, and usage instructions.

[0081] In this embodiment, the user invokes the data visualization control function through the data visualization module to generate data visualization displays and contour line visualization displays based on the following process:

[0082] Step A1: Configure the file path, variable name, depth index, coordinate tick spacing, and contour line interval value of the NetCDF file and save it. Open the saved NetCDF file. If it fails to open, record the error and return. Return the error information and write it to the processing log.

[0083] Step A2: Check if the variable name exists. If it exists, proceed to step A3 and select the time layer and depth index layer. If it does not exist, record the variable error and return. Return the error message and write it to the processing log.

[0084] Step A3: Obtain ocean feature values ​​and latitude / longitude coordinate data based on the variable and dimensional information of the ocean data in the NetCDF file;

[0085] Step A4: Determine the visualization type. If it is determined to be contour visualization, proceed as follows:

[0086] Calculate the latitude and longitude boundary range of the input NetCDF file and generate contour lines. Use the matplotlib library to visualize the contour line data and save it in PNG format.

[0087] When it is determined that data visualization is to be performed, the following process is followed:

[0088] Calculate the latitude and longitude boundary range of the input NetCDF file, call the matplotlib library to visualize the data and save it in PNG format.

[0089] In the contour generation module, based on a single NetCDF file processing unit or multiple NetCDF file processing units, the function of generating contour lines from a single NetCDF file and the function of generating contour lines from multiple NetCDF files can be selected.

[0090] Under the function of generating contour lines from a single NetCDF file, the user configures the input path of the single NetCDF file, the contour line output directory, variable names, contour line intervals, time index, and depth index. After the configuration is completed, the system uses the function of generating contour lines from a single NetCDF file to call the contour line calculation function to generate raster data from the ocean data using the GDAL library, and then performs contour line calculations based on the raster data and saves it as SHP format.

[0091] In the function of generating contour lines from multiple NetCDF files, the user configures the directories of multiple input NetCDF files, the output directory of contour lines, variable names, contour line intervals, time indexes, and depth indexes. After the configuration is completed, the system calls the contour line calculation function based on the function of generating contour lines from multiple NetCDF files, processes multiple NetCDF files in parallel in the same process, and saves the calculated contour lines in SHP format.

[0092] In this embodiment, for ocean data in a single NetCDF file, the user uses the contour generation module to call the contour generation function for a single NetCDF file and generates vector contour data based on the following process:

[0093] Step B1: Configure the path to the single NetCDF file and open the NetCDF file;

[0094] Step B2: Read the specified ocean data with specified variables, time index, and depth index, and obtain the minimum and maximum values ​​of each variable in the specified ocean data to set the contour range;

[0095] Step B3: Read latitude and longitude data and process invalid values, then replace the latitude and longitude data with NaN, where NaN is a missing value marker. Invalid values ​​in latitude and longitude data include filled invalid data (_FillValue) and values ​​outside the valid range;

[0096] Step B4: Calculate the geographic transformation parameters, including the top-left corner coordinates and cell size;

[0097] Step B5: Create a GeoTIFF file, set the georeference, write the specified ocean data into the GeoTIFF file, replace NaN with -9999.0 as the NoData value, and then close the GeoTIFF file. The georeference is EPSG:4326.

[0098] Step B6: Create a list of contour levels, a Shapefile data source, and a line type layer at intervals, from minimum to maximum value;

[0099] Step B7: Create a field called elevation to store contour values, and call the gdal.ContourGenerate function to generate a contour shapefile file;

[0100] In this embodiment, for ocean data in multiple NetCDF files, the user generates vector contour data using the contour generation module based on the following process:

[0101] Step C1: Configure the input directory for multiple NetCDF files, and open all NetCDF files;

[0102] Step C2 includes the following parallel processing procedures:

[0103] According to the process of steps B1 to B5, the specified ocean data in each NetCDF file is written into the corresponding GeoTIFF file. Then, according to the process of steps B6 to B7, contour line Shapefile files are generated for the specified ocean data in each NetCDF file.

[0104] Step C3: Traverse the input directory to find all NetCDF files. For each NetCDF file, create a corresponding output subdirectory according to the original input directory structure and store it in the generated contour shapefile file.

[0105] Under the slicing module, based on a single NetCDF file slicing unit or multiple NetCDF file slicing units, the single NetCDF file slicing function and the multiple NetCDF file slicing function can be selected.

[0106] Under the single NetCDF file slicing function, the user configures the input single NetCDF file path, slice output directory, slice parameters: latitude slice size and longitude slice size, compression level. After configuration, the system calls the slice calculation function to parse the NetCDF file according to the single NetCDF file slicing function, extracts the metadata of the ocean data in the NetCDF file, divides the latitude and longitude grid for cyclic slicing, and saves the sliced ​​NetCDF file in the slice output directory.

[0107] With multiple NetCDF file slicing functions, the user configures the input NetCDF file directory, slice output directory, slice parameters (latitude slice size and longitude slice size, compression level), and after configuration, the system calls the slice calculation function according to the multiple NetCDF file slicing functions, and processes multiple NetCDF slice files synchronously in parallel using the same process as under a single NetCDF file slicing function, and saves the sliced ​​NetCDF files in the slice output directory.

[0108] In this embodiment, for ocean data in a single NetCDF file, the user performs data slicing processing through the slicing module based on the following procedure:

[0109] Step D1: Configure the file path of the NetCDF file, open the NetCDF file using the netCDF4 library, obtain the file handle, and then verify whether the ocean data in the NetCDF file can be accessed and read normally. After it can be accessed and read normally, proceed to step D2.

[0110] Step D2: Parse the filename, extract the year, month, and date information from the filename, and use regular expressions to match the YYYYMMDD format. If the date information cannot be extracted, record the error and terminate the process.

[0111] Step D3: Read the latitude and longitude from the dimension information in the NetCDF file in sequence, obtain the latitude array and longitude array in sequence, and verify their completeness and validity after obtaining the latitude array and longitude array;

[0112] Step D4: Create the output directory according to the hierarchical structure of prefix / year / month / day. After creating the output directory, establish its directory organization system (output file storage path) corresponding to the original ocean data time information.

[0113] Step D5: Perform a tiling cycle based on the latitude and longitude grid, specifically including:

[0114] Establish a latitude cycle from -90° to 90°: from -90° to 90°, and increment by a specified latitude step, such as 10°;

[0115] Establish a longitude cycle from -180° to 180° and increment it by a specified longitude step, such as 10°.

[0116] Calculate the latitude and longitude boundaries of each slice, including the starting latitude, ending latitude, starting longitude, and ending longitude, and construct the starting coordinates and ending coordinates;

[0117] Step D6: Create a spatial mask and extract slice data, specifically including:

[0118] Create a Boolean mask for the current slice, making its latitude greater than or equal to the starting latitude and less than the ending latitude;

[0119] Create a longitude boolean mask for the current slice, making its longitude greater than or equal to the starting longitude and less than the ending longitude;

[0120] After creating the latitude and longitude Boolean masks, verify that there are valid data points within the slice to avoid generating empty slices;

[0121] Step D7: Create a new NetCDF file in NETCDF4 format for each slice, and ensure that the file name format is: slice_{start latitude}_{start longitude}.NetCDF. Then copy the global attributes of the corresponding original NetCDF file to the slice file.

[0122] Step D8: Create a latitude dimension for the slice file using the latitude values ​​within the slice, create a longitude dimension for the slice file using the longitude values ​​within the slice, and copy all remaining dimensions, such as time and depth, to maintain consistency with the original file;

[0123] Step D9: Iterate through all variables of the ocean data in the original NetCDF file, create a corresponding slice file variable for each variable, and set compression parameters for the data variables and non-coordinate variables;

[0124] Step D10: Copy all attributes of the variable to the slice file to ensure data integrity;

[0125] Step D11: Write the slice data, specifically including:

[0126] For latitude variables: write the latitude value into the slice;

[0127] For longitude variables: write the longitude value into the slice;

[0128] For data variables: apply a spatial mask based on the dimensional position to extract slice data;

[0129] Step D12: Identify the location of latitude and longitude dimensions in the data variables, apply a mask to the spatial dimensions, retain all other dimensions to construct a multidimensional slice index, perform the actual data slicing operation, and extract subset data;

[0130] Step D13: Monitor for anomalies during the data slicing process. If slicing fails, implement a degradation solution while writing complete data, and record detailed error information for troubleshooting.

[0131] Step D14: After completing all slices, close the original NetCDF file, ensure that all file handles are properly released, and record the slice generation completion status.

[0132] In this embodiment, for ocean data in multiple NetCDF files, the user further performs data slicing processing based on the following additional procedure through the slicing module:

[0133] Step E1: Prepare the file list, recursively scan the input directory to find all NetCDF and NetCDF4 files and build a complete list of paths to the files to be processed;

[0134] Step E2: Configure parallel processing parameters, set the thread pool size based on the number of CPU cores and the number of files, and prepare the task parameter tuple: file path, output directory, and slice parameters;

[0135] Step E3: Perform parallel slice processing, submit all file processing tasks using a thread pool, monitor task execution status and progress, and collect processing results and error information;

[0136] Step E4: Results Statistics and Summary. Count the number of successfully processed files, record the failed files and the reasons for the errors, and generate a processing report.

[0137] In the format conversion module, based on a single file format conversion unit or multiple file format conversion units, you can select between single file format conversion function and multiple file format conversion function;

[0138] Under the single file format conversion function, the user configures the input NetCDF file path, TIFF file output directory, variable name, EPSG coordinate system code, and time index. After the configuration is completed, the system calls the TIFF file calculation function to parse the NetCDF file according to the single file format conversion function, extracts the metadata of the ocean data in the NetCDF file, performs geographic parameter calculation, performs depth layer loop processing and 2D data extraction, and finally uses the GDAL library to write the NetCDF file containing ocean data into a TIFF file and saves the converted TIFF file in the TIFF file output directory.

[0139] With multiple file format conversion functions, the user configures the input NetCDF file directory, TIFF file output directory, variable names, EPSG coordinate system code, and time index. After configuration, the system uses a parallel approach, employing the same process as a single file format conversion function, to write multiple NetCDF files into TIFF files and saves the converted TIFF files in the TIFF file output directory.

[0140] In this embodiment, for ocean data in a single NetCDF file, the user performs data slicing processing based on the following procedure using the format conversion module:

[0141] Step F1: Set the spatial reference system to the WGS84 coordinate system, initialize the spatial reference system according to the EPSG code specified by the user, and if the EPSG code is invalid, automatically fall back to the corresponding EPSG code EPSG:4326 in WGS84, and record the coordinate system usage.

[0142] Step F2: Register all GDAL drivers and set the GDAL filename UTF8 encoding option to initialize and prepare the GDAL runtime environment;

[0143] Step F3: Use the xarray library to open the specified NetCDF file and check if the specified variable information exists in the file to verify that the file can be read normally. The xarray library is an open-source Python library designed specifically for processing labeled multidimensional scientific data.

[0144] Step F4: Read the specified variable information, obtain the latitude and longitude coordinates, and verify the completeness and validity of the latitude and longitude coordinates;

[0145] Step F5: Calculate the height and width of the latitude and longitude grid, determine the latitude and longitude range, and calculate the spatial resolution in the latitude and longitude directions;

[0146] Step F6: Process invalid values ​​in ocean data, identify and obtain imputed value attributes or missing value attributes, set invalid value parameters (default value is 100000000.0), and prepare invalid value processing parameters;

[0147] Step F7: Prepare for depth layer processing, generate a depth layer identifier array (0000, 0002, ..., 0098), ensure the TIFF file output directory exists, and initialize the depth layer loop;

[0148] Step F8: Multi-depth layer loop processing, traversing all depth layers and skipping layers that exceed the actual depth range of the data, and performing the transformation process for each depth layer;

[0149] Step F9: Extract 2D data of the current depth according to the data dimension structure. In this embodiment, it can be extracted from the following four dimensions: time, depth, latitude, and longitude, or it can be extracted from the following three dimensions: depth, latitude, and longitude.

[0150] Step F10: Define geographic transformation parameters and set a six-parameter affine transformation. The six geographic transformation parameters in the six-parameter affine transformation include: top left longitude, longitude resolution, rotation parameter, top left latitude, rotation parameter, and latitude resolution (negative value).

[0151] Step F11: Create a temporary TIFF file, generate a standardized output filename, create a temporary TIFF file of Float64 data type, and set the georeference system and geotransformation parameters;

[0152] Step F12: Write the 2D data array to the TIFF band, set the NoData value flag, and calculate the data statistics;

[0153] Step F13: Use the gdal warp function to resample the raster data to a resolution of 0.08 degrees, and apply the cubic convolution resampling method to maintain the consistency of invalid values;

[0154] Step F14: Clean up temporary resources, delete temporary TIFF files, and close the GDAL dataset to release memory resources;

[0155] Step F15: Complete file processing, close the NetCDF file, record the processing completion status, and return the processing result.

[0156] In this embodiment, for ocean data in multiple NetCDF files, the user also performs TIFF file conversion based on the following additional process within the single file format conversion function of the format conversion module:

[0157] Step G1: Prepare the file list by recursively scanning the input directory to find all .NetCDF and .NetCDF4 files and to build a complete list of paths to the files to be processed;

[0158] Step G2: Configure parallel processing parameters, set the thread pool size based on the number of CPU cores and the number of files, and prepare task parameter tuples: file path, output directory, latitude and longitude tile range size;

[0159] Step G3: Execute parallel processing tasks, submit all file processing tasks using a thread pool, monitor task execution status and progress, and collect processing results and error information;

[0160] Step G4: Results statistics and summary. Count the number of successfully processed files, record the failed files and the reasons for the errors, and generate a processing report.

[0161] This invention addresses the lack of efficient, batch processing tools for marine data in NetCDF files, and the absence of integrated multi-format file conversion and visualization capabilities in existing tools. It provides an integrated processing system and method for marine data stored in NetCDF files, integrating and implementing functions for slicing, rasterization, and contour generation of marine data stored in NetCDF files. This reduces the burden of overall marine environmental data format conversion and data dissemination. Furthermore, it enables parallel processing of marine data in NetCDF files. Parallel computation is used in the following processes—contour generation, slicing, and format conversion—when processing multiple NetCDF files, thereby improving data processing efficiency.

[0162] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any indirect modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An integrated processing system for ocean data stored in a NetCDF file, characterized in that, The application relates to a marine data processing system and method. The file opening module is used for receiving user instructions to open and parse a NetCDF file to be processed and storing marine data, and extracting the following marine data information: variable information, dimension information and attribute information; The data visualization control module is used for receiving user configurations of variable names, coordinate scale intervals and contour interval values of the NetCDF file to be processed, and realizing data visualization display, contour visualization display and local saving according to the extracted marine data information and the matplotlib library; The contour generation module is used for receiving user configurations of variable names, contour interval values, time indexes and depth indexes of the NetCDF file to be processed, generating vector contour data after rasterizing the marine data, visualizing variable contour lines and saving the variable contour lines in a Shapefile format; The slicing module is used for receiving user configurations of longitude and latitude slicing sizes, compression levels and output directories of the NetCDF file to be processed, performing data slicing processing on marine data in the NetCDF file, and saving valid data as a new NetCDF file after verification; The format conversion module is used for receiving user configurations of variable names, EPSG coordinate system codes, time indexes, depth indexes and TIFF file output directories of the NetCDF file to be processed, extracting 2D marine data of specified variables, time slices and depth layers from the NetCDF file, and converting the 2D marine data into a TIFF file which retains an original longitude and latitude grid structure and geographical coordinate association based on the GDAL library; The view control module is used for managing processing logs; The help module is used for providing operation instructions.

2. The integrated processing system for ocean data oriented to storage into a NetCDF file according to claim 1, wherein, The variable information of the marine data includes physical meaning description, data unit, length unit and effective value range of the variable, and the dimension information of the marine data includes time, longitude, latitude and depth.

3. The integrated processing system for ocean data oriented to storage into a NetCDF file according to claim 2, wherein, The contour generation module comprises a single NetCDF file processing unit and multiple NetCDF file processing units; The single NetCDF file processing unit is based on user configurations of a single NetCDF file path, a contour output directory, a variable name, a contour interval value, a time index and a depth index, calls a contour calculation function to generate raster data of marine data in the NetCDF file through the GDAL library, and saves the raster data as SHP format based on the raster data; The multiple NetCDF file processing units are based on user configurations of a directory of multiple NetCDF files, a contour output directory, a variable name, a contour interval value, a time index and a depth index, call a contour calculation function through a parallel computing mode, synchronously generate raster data of marine data in all the NetCDF files through the GDAL library, and save the raster data as SHP format based on the raster data.

4. The integrated processing system for ocean data oriented to storage into a NetCDF file according to claim 3, wherein, The processing process of the single NetCDF file processing unit is as follows: Based on the user-configured time index and depth index, the specified time layer and depth layer marine data in the NetCDF file are filtered and the target variable is read. The invalid data and values outside the valid range are identified and marked as missing values. Then, the GDAL library is used to read the preprocessed variable and latitude and longitude data to generate a GeoTIFF file with EPSG: 4326 geographic coordinate reference. The ContourGenerate function of the GDAL library is called to generate vector contour lines by comparing the grid values with the user-set contour level. Finally, the elevation field of the Shapefile is created to store the marine data values corresponding to the contour lines, and the contour lines are written as LineString geometric objects with the elevation field attribute values in SHP format.

5. The integrated processing system for ocean data oriented to storage into a NetCDF file according to claim 2, wherein, The slicing module includes a single NetCDF file slicing unit and multiple NetCDF file slicing units; The single NetCDF file slicing unit is based on the user-configured single NetCDF file path, slicing output directory, latitude and longitude slicing size, and compression level. The slicing calculation function is called to parse the metadata of marine data in the single NetCDF file, divide the latitude and longitude grid, and loop slicing. The sliced NetCDF file is saved to the slicing output directory. The multiple NetCDF file slicing units are based on the user-configured multiple NetCDF file directory, slicing output directory, latitude and longitude slicing size, and compression level. The slicing calculation function is called through parallel computing to synchronously parse the metadata of marine data in each NetCDF file, divide the latitude and longitude grid, and loop slicing. The sliced NetCDF file is saved to the slicing output directory.

6. The integrated processing system for ocean data oriented to storage into a NetCDF file according to claim 5, wherein, The single NetCDF file slicing unit uses a serial loop method to process each grid unit in order, and the multiple NetCDF file slicing units use a multi-level parallel computing architecture. The multiprocessing.Pool is used to realize multi-process parallel computing, automatically distribute different NetCDF files to multiple same processes, and simultaneously process based on the processing process of the single NetCDF file slicing unit.

7. The integrated processing system for ocean data oriented to storage into a NetCDF file according to claim 2, wherein, The format conversion module includes a single file format conversion unit and multiple file format conversion units; The single file format conversion unit is based on the user-configured single NetCDF file path, TIFF file output directory, variable name, EPSG coordinate system code, and time index. The TIFF file calculation function is called to parse the metadata of marine data in the NetCDF file, calculate the geographic parameters, loop process the depth layer data, and extract 2D data. The GDAL library is used to write the 2D data into the TIFF file and save it to the output directory. The multiple file format conversion units call a TIFF file calculation function in parallel computing mode based on a directory where multiple NetCDF files configured by a user are located, a TIFF file output directory, a variable name, an EPSG coordinate system code, and a time index, synchronously analyze metadata of marine data in the NetCDF files, calculate geographic parameters, cyclically process depth layer data and extract 2D data, and write the 2D data into a TIFF file and save it to the output directory through a GDAL library.

8. The integrated processing system for ocean data oriented to storage into a NetCDF file according to claim 7, wherein, When the formats of multiple NetCDF files are converted, all the NetCDF files are recursively searched and a thread pool is created for task allocation, and multiple worker threads perform format conversion of a single NetCDF file in parallel based on a processing procedure of a single file format conversion unit.

9. A method for integrated processing of marine data stored in a NetCDF file, characterized in that, The application is applied to the integrated processing system for marine data in a NetCDF file as claimed in claim 1, and includes the following steps: A file opening module opens and analyzes a NetCDF file to be processed according to a user instruction, and extracts the following marine data information: variable information, dimension information, and attribute information. A data visualization control module receives a variable name, a coordinate scale interval, and an isopleth interval value configured by a user for a NetCDF file to be processed, and realizes data visualization display, isopleth visualization display, and local saving according to the extracted marine data information and a matplotlib library. An isopleth generation module receives a variable name, an isopleth interval value, a time index, and a depth index configured by a user for a NetCDF file to be processed, generates vector isopleth data after rasterizing marine data, visualizes variable isopleths, and saves them in a Shapefile format. A slicing module receives a latitude and longitude slicing size, a compression level, and an output directory configured by a user for a NetCDF file to be processed, performs data slicing on marine data in the NetCDF file, and saves valid data as a new NetCDF file after verification. A format conversion module receives a variable name, an EPSG coordinate system code, a time index, a depth index, and a TIFF file output directory configured by a user for a NetCDF file to be processed, extracts 2D marine data of a specified variable, a time slice, and a depth layer from the NetCDF file, and converts it into a TIFF file that retains an original latitude and longitude grid structure and is associated with geographic coordinates based on a GDAL library. A view control module manages a processing log. A help module provides operation instructions.

Images