A Data Visualization-Based Parallel I / O Optimization Method and System Based on NetCDF

Abstract

This invention discloses a parallel I / O optimization method and system for data visualization based on NetCDF. The method includes: S1, performing fine-grained data block division on the input NetCDF file, discretizing the small data blocks obtained from the block division into multiple data parts, merging the small data blocks, and saving them as a block-optimized NetCDF file; S2, reading the block-optimized NetCDF file in parallel; S3, constructing virtual cells to represent the cell data structure using point data so that the point sequence of the cell forms a completely ordered sequence; S4, performing visualization calculations using the cell data structure represented by point data; S5, after the visualization calculation is completed, discarding the virtual cell calculation results and outputting the visualization calculation results. This invention aims to balance the computational load of each node process, improve the I / O efficiency of the visualization program, and improve the memory utilization efficiency and visualization calculation efficiency of the visualization program.

Description

Technical Field

[0001] This invention relates to the field of visualization technology for scientific computing, and specifically to a data visualization parallel I / O optimization method and system based on NetCDF. Background Technology

[0002] Scientific computing visualization targets data with geometric properties or structural characteristics used in scientific and engineering measurements. It utilizes computer graphics, image processing, and other technologies to simulate and display the real-world scenarios of the data in three dimensions or dynamically. The main application areas of scientific computing visualization are in the natural sciences, such as physics, chemistry, earth sciences, astronomy, medicine, and biology. It enables scientists to identify patterns, characteristics, relationships, and anomalies by interpreting, manipulating, and processing scientific and technological data and models. With the rapid development of computer hardware and software and the widespread adoption of networks, the amount of data in scientific computing has exploded, making scientific big data visualization a crucial research area. High-performance computing systems are typically used to accelerate data processing in scientific big data visualization. As the data volume increases, the runtime of scientific big data visualization is increasingly limited by I / O rates. Currently, most scientific data is stored in the NetCDF format. NetCDF (Network Common Data Form) is a common network data format that allows for efficient storage, management, retrieval, and distribution of grid data.

[0003] The common schemes of the current NetCDF-based I / O optimization method are: parallel read and write optimization of NetCDF files; tuning of the parameters of the underlying parallel file system to optimize the I / O efficiency of large-scale visualization datasets; and optimization of data caching strategies. For example, Jared et al. [1] (Jared Brzenski, Christopher Paolini, José E. Castillo, Improving the I / O of large geophysical models using PnetCDF and BeeGFS. Parallel Comput. 2021: 104-105) used PnetCDF and the parallel file system BeeGFS to optimize the I / O of geophysical modeling applications, and combined with the application, analyzed and used the best parameter settings of BeeGFS. Behzad et al. [2] (Behzad B, Luu HVT, Huchette J, Byna S, Prabhat, Aydt RA, Koziol Q, Snir M. Taming parallel I / O complexity with auto-tuning, SC 2013, ACM, Denver, CO, USA, 2013: 17–22) developed an automatic tuning framework that uses a genetic algorithm to attempt to automatically provide the optimal I / O settings relative to the entire I / O stack. Li Linlin et al. [3] (Li Linlin, Wu Wanguo, Sun Liangxu. Performance optimization of fine-grained I / O of parallel file system Lustre [J]. Computer Engineering and Applications, 2012: 88-92) proposed a fine-grained priority LRU algorithm, which maximizes the retention of fine-grained I / O pages in the page cache of object storage server and client, reduces the page sinking speed caused by fine-grained I / O, prolongs the time of fine-grained I / O pages in main memory, and thus reduces the number of disk accesses and reduces disk access overhead. Wong et al. [4] ([4]Wong DC, Yang CE, Fu JS,Wong K, and Gao Y. An approach to enhance pnetCDF performance in environment modeling applications, Geosci. Model Dev, 2015: 1033–1046) proposed an application-level data aggregation method, which enables processes to perform data aggregation in row or column dimensions and uses PnetCDF to read and write aggregated data in parallel.While current NetCDF-based I / O optimization methods have improved the I / O efficiency of NetCDF files to some extent, these optimization schemes do not specifically address the characteristics of data reading and processing in visualization applications. Scientific big data visualization computation exhibits locality of reference; in a parallel computing environment, using traditional uniform data partitioning methods will result in uneven computation, with computation concentrated in a few processes while most processes remain idle. NetCDF contains dimensions, variables, and attributes, with each part's values ​​stored sequentially in the NetCDF file. After fine-grained partitioning, processes involve a large number of fine-grained discrete I / O requests, which are inefficient for parallel file systems. NetCDF data is stored in points, while visualization programs perform parallel computations in cells. Visualization programs cannot directly perform calculations on data read from files; a data transformation from point data structures to cell data structures is required. Cell data structures require more memory than point data structures and increase memory copying overhead. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a parallel I / O optimization method and system for data visualization based on NetCDF, which aims to balance the computational load of each node process, improve the I / O efficiency of the visualization program, and enhance the memory usage efficiency and visualization computation efficiency of the visualization program.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A data visualization-based parallel I / O optimization method based on NetCDF includes: S1, perform fine-grained data block division on the input NetCDF file, discretize the small data blocks obtained from the block division into multiple data parts, merge the small data blocks of each data part, and finally save it as the block-optimized NetCDF file; S2, reads the block-optimized NetCDF file in parallel; S3, construct virtual cells to represent cell data structures using point data so that the point sequence of cells forms a completely ordered sequence; S4 performs visual calculations using a cell data structure represented by point data; S5 discards the virtual cell calculation results and outputs the visual calculation results after the visualization calculation is completed.

[0006] Optionally, step S1 includes: S1.1, Read the input NetCDF file; S1.2, set parameters, including: data block size, number of file data hash buckets; S1.3, the input NetCDF file is divided into fine-grained data blocks based on parameters. The small data blocks obtained from the block division are discretized into coordinate data and scalar value data, and the coordinate data and scalar value data are distributed and stored in various hash buckets. The coordinate data and scalar value data in the NetCDF file are merged according to the hash bucket number to which the discrete data belongs, so as to ensure that the data in the hash bucket is stored continuously to convert the discrete fine-grained I / O into sequential coarse-grained I / O. S1.4 outputs the block-optimized NetCDF file.

[0007] Optionally, step S1.3 includes: S1.3.1 First, the NetCDF file is divided into first-level blocks according to the number of processes used for parallel processing, so that the number of first-level blocks is the same as the number of processes used for parallel processing. S1.3.2, then divide each first-level block into second-level blocks according to the number of processes used for parallel processing, so that each first-level block contains the same number of second-level blocks used for parallel processing, and each process used for parallel processing takes turns processing the second-level blocks in order. S1.3.3, and then the second-level blocks in the hash bucket are reorganized so that multiple second-level blocks processed by each process are stored contiguously in the same hash bucket to convert discrete fine-grained I / O into sequential coarse-grained I / O.

[0008] Optionally, step S3 includes: S3.1, retrieve coordinate data and scalar value data from the NetCDF file. Each process determines the data to be processed based on the process ID and hash bucket number. Each process processes data from one or more hash buckets. S3.2, Based on the coordinate data and scalar value data, add virtual cell data at the end of each column of data so that the point sequence of the cells forms a completely ordered sequence; S3.3 uses cell data represented by point data as input for visualization calculations, so that after adding virtual cells, cell data is represented by continuous point data, and no data conversion is required when performing visualization calculations.

[0009] Optionally, the coordinate data and scalar value data are numbered in a bottom-up, left-to-right order. Alternatively, the coordinate data and scalar value data are numbered in a bottom-up, left-to-right order.

[0010] Optionally, before step S1, it may also include determining whether the input NetCDF file is a block-optimized NetCDF file. If it is a block-optimized NetCDF file, then proceed to step S2; otherwise, proceed to step S1.

[0011] Furthermore, this invention also provides a data visualization parallel I / O optimization system based on NetCDF, including... The NetCDF file optimization processing unit is used to perform fine-grained data block division on the input NetCDF file, discretize the small data blocks obtained by block division into multiple data parts, merge the small data blocks of each data part, and finally save them as the block-optimized NetCDF file. NetCDF file parallel reading program unit, used for parallel reading of block-optimized NetCDF files; The cell data structure construction program unit is used to construct virtual cells to represent cell data structures using point data so that the point sequence of cells forms a completely ordered sequence; A visualization calculation unit for performing visualization calculations using a cell data structure represented by point data; The calculation result output program unit is used to discard the virtual cell calculation results and output the visual calculation results after the visualization calculation is completed.

[0012] Furthermore, the present invention also provides a NetCDF-based data visualization parallel I / O optimization system, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the NetCDF-based data visualization parallel I / O optimization method.

[0013] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program for being programmed or configured by a microprocessor to execute the NetCDF-based data visualization parallel I / O optimization method.

[0014] Compared with traditional methods, this invention has the following main advantages: Based on the I / O patterns, data characteristics, and data processing flow of analytical scientific big data visualization applications, this invention optimizes the data storage format of NetCDF files and balances the computational load of each node process by discretizing the data files. Furthermore, this invention improves the application's I / O efficiency by merging small data blocks discretized into the same part. In addition, this invention also includes optimizing the NetCDF memory data structure, introducing virtual cells, and using point data to represent coordinate data, thereby improving memory utilization efficiency and visualization computation efficiency. Attached Figure Description

[0015] Figure 1This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram illustrating the process of optimizing NetCDF files in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of data segmentation in an embodiment of the present invention.

[0018] Figure 4 This is a schematic diagram of data recombination in an embodiment of the present invention.

[0019] Figure 5 This is a flowchart illustrating the optimization process of the NetCDF memory data structure in an embodiment of the present invention.

[0020] Figure 6 This is a schematic diagram of NetCDF cell numbering in an embodiment of the present invention.

[0021] Figure 7 This is a schematic diagram of a virtual cell in an embodiment of the present invention. Detailed Implementation The present invention provides a parallel I / O optimization method for data visualization based on NetCDF, which aims to improve the I / O efficiency of ultra-large-scale visualization datasets. The invention will be further described in detail below with reference to the accompanying drawings.

[0022] like Figure 1 As shown, the data visualization parallel I / O optimization method based on NetCDF in this embodiment includes: S1, perform fine-grained data block division on the input NetCDF file, discretize the small data blocks obtained from the block division into multiple data parts, merge the small data blocks of each data part, and finally save it as the block-optimized NetCDF file; S2, reads the block-optimized NetCDF file in parallel; S3, construct virtual cells to represent cell data structures using point data so that the point sequence of cells forms a completely ordered sequence; S4 performs visual calculations using a cell data structure represented by point data; S5 discards the virtual cell calculation results and outputs the visual calculation results after the visualization calculation is completed.

[0023] like Figure 2 As shown, step S1 in this embodiment includes: S1.1, Read the input NetCDF file; S1.2, set parameters, including: data block size, number of file data hash buckets; S1.3, the input NetCDF file is divided into fine-grained data blocks based on parameters. The small data blocks obtained from the block division are discretized into coordinate data and scalar value data, and the coordinate data and scalar value data are distributed and stored in various hash buckets. The coordinate data and scalar value data in the NetCDF file are merged according to the hash bucket number to which the discrete data belongs, so as to ensure that the data in the hash bucket is stored continuously to convert the discrete fine-grained I / O into sequential coarse-grained I / O. S1.4 outputs the block-optimized NetCDF file.

[0024] In this embodiment, step S1.3 includes: S1.3.1 First, the NetCDF file is divided into first-level blocks according to the number of processes used for parallel processing, so that the number of first-level blocks is the same as the number of processes used for parallel processing (similar to the traditional uniform data partitioning method). S1.3.2, then divide each first-level block into second-level blocks according to the number of processes used for parallel processing, so that each first-level block contains the same number of second-level blocks used for parallel processing, and each process used for parallel processing takes turns processing the second-level blocks in order. S1.3.3, and then the second-level blocks in the hash bucket are reorganized so that multiple second-level blocks processed by each process are stored contiguously in the same hash bucket to convert discrete fine-grained I / O into sequential coarse-grained I / O.

[0025] Scientific big data visualization computation exhibits locality of computation. In a parallel computing environment, if traditional uniform data partitioning methods are used, computational imbalance will occur, with computation concentrated in a few processes while most processes remain idle. To balance the computational load across nodes, this embodiment performs fine-grained data partitioning. This embodiment converts discrete fine-grained I / O into sequential coarse-grained I / O, and the block size of the second-level partition is set to be greater than or equal to the size of the parallel file system block. Each process processes the partitions alternately. Figure 3 Figure (a) shows a traditional method of uniformly dividing data, in which processes P0-P3 each process a contiguous block of data. For visualization processing programs, this method of division will lead to uneven computation between processes. Figure 3As shown in (b) of this embodiment, the improved data block partitioning method involves processes P0-P3 alternating to process discrete data blocks. While this method balances the computational load across processes, during data reading, the data blocks processed by a single process are discontinuous, requiring each block to be read individually. This results in the underlying file system containing many discrete, small-volume read / write requests. Parallel file systems perform well on coarse-grained I / O requests but poorly on fine-grained I / O requests. To improve the read / write efficiency of scientific big data files, this embodiment optimizes the NetCDF storage format. NetCDF contains dimensions, variables, and attributes, with each value stored sequentially in the NetCDF file. After fine-grained partitioning, the process contains a large number of fine-grained discrete I / O requests. To improve the efficiency of process data read / write, this embodiment reorganizes the data, such as... Figure 4 As shown, process P0 processes data blocks 0, 4, 8, and 12. Before data reassembly, these data blocks are discretely distributed across the file. After data reassembly, these data blocks maintain physical spatial continuity, transforming discrete fine-grained I / O into sequential coarse-grained I / O, thus improving data I / O efficiency.

[0026] Step S3 is used for NetCDF memory data structure optimization. For example... Figure 5 As shown, step S3 in this embodiment includes: S3.1, retrieve coordinate data and scalar value data from the NetCDF file. Each process determines the data to be processed based on the process ID and hash bucket number. Each process processes data from one or more hash buckets. S3.2, Based on the coordinate data and scalar value data, add virtual cell data at the end of each column of data so that the point sequence of the cells forms a completely ordered sequence; S3.3 uses cell data represented by point data as input for visualization calculations, so that after adding virtual cells, cell data is represented by continuous point data, and no data conversion is required when performing visualization calculations.

[0027] See Figure 5 As can be seen, after constructing virtual cells to represent cell data structures using point data so that the point sequences of cells form a completely ordered sequence, visual calculations can be performed using the point data-represented cell data structure. After the visual calculation is completed, the virtual cell calculation results are discarded and the visual calculation results are output. Since virtual cell data is invalid data, the virtual cell calculation results need to be discarded after obtaining the visual calculation results.

[0028] NetCDF data is stored in units of coordinate points. Generally, each point coordinate has a corresponding location scalar. Cells are numbered from bottom to top and left to right, or from left to right and bottom to top. Points within cells are numbered counter-clockwise. For example, if cells are numbered from bottom to top and left to right... Figure 6 As shown in (a), cell 0 contains the points (p0, p9, p10, p1). When cells are numbered from left to right and from bottom to top, as shown in (a), the cells are arranged in a specific order. Figure 6 As shown in (b), cell 0 contains points (p0, p1, p10, p9). In the following text, cells are numbered from bottom to top and from left to right by default. NetCDF data is stored in units of points, while the visualization program performs parallel calculations in units of cells. The visualization program cannot directly perform calculations on data read from the file; a data conversion is required, from a point data structure to a cell data structure. The cell data structure requires more memory space than the point data structure and increases the overhead of memory copying. In the point-based data structure, the distribution of points in the cells follows certain patterns. For example... Figure 6 As shown in (a), the sequence of the first points in cells 0-15 is (p0, p1, p2, p3, p4, p5, p6, p7, p9, p10, p11, p12, p13, p14, p15, p16). Except for the last point in each column, the point sequence is continuous. Similarly, other points in the cells also exhibit this pattern. However, because the point sequence in the cells has breaks and is not a complete sequential sequence, the cell data cannot be directly represented using point data.

[0029] To address this issue, this embodiment introduces virtual cells within the original cells, such as... Figure 7 As shown, by adding virtual cells, such as Figure 7 Cells numbered 8 and 17 have a completely ordered sequence of points. For example, the sequence of the first point in cells 0-17 is (p0, p1, p2, p3, p4, p5, p6, p7, p8, p9, p10, p11, p12, p13, p14, p15, p16, p17). Similarly, the sequences of other points in the cells are also completely ordered (first ordering each subsequence, then ordering the subsequences between segments), satisfying the requirements for parallel cell computation. After parallel processing of cells is completed, the calculated values ​​of the virtual cells are discarded, and the calculation results of the actual cells are obtained.

[0030] Step S4 uses a cell data structure represented by point data to perform visual calculations, which is an existing method. This method does not depend on a specific visual calculation program or method, so its implementation details will not be elaborated here. See also... Figure 1 To improve the robustness of the method in this embodiment, before step S1, this embodiment further includes determining whether the input NetCDF file is a block-optimized NetCDF file. If it is a block-optimized NetCDF file, then jump to step S2; otherwise, jump to step S1.

[0031] In summary, current NetCDF-based I / O optimization methods do not consider the characteristics of data reading and processing in visualization applications. This embodiment of the NetCDF-based parallel I / O optimization method for data visualization analyzes the I / O patterns, data characteristics, and data processing flows of scientific big data visualization applications. It optimizes the data storage format, memory data structure, and data processing flow of NetCDF files, including optimizing the NetCDF format and using the parallel implementation of PnetCDF to complete parallel data reading and writing. This accelerates the I / O efficiency of large-scale visualization data. The main innovations of this NetCDF-based parallel I / O optimization method include: 1. NetCDF storage format optimization: Fine-grained data partitioning is performed, distributing data discretely across multiple parts. Each process is responsible for processing one or several parts to balance the computational load of each node process. The partitioned data of each data part is merged, replacing multiple small, non-contiguous I / O requests with a single large, contiguous I / O request, improving the application's I / O efficiency; 2. NetCDF memory data structure optimization. By introducing virtual cells and using point data to represent coordinate data, memory usage is saved, memory copying is reduced, and visualization computation efficiency is improved. Compared with traditional methods, this embodiment's NetCDF-based parallel I / O optimization method for data visualization optimizes the data storage format of NetCDF files by analyzing the I / O patterns, data characteristics, and data processing flows of scientific big data visualization applications. It balances the computational load of each node process by discretizing the data file. By merging small data blocks discretized to the same part, the application's I / O efficiency is improved. This embodiment's NetCDF-based parallel I / O optimization method for data visualization also optimizes the NetCDF memory data structure, introduces virtual cells, and uses point data to represent coordinate data, improving memory usage efficiency and visualization computation efficiency.

[0032] Furthermore, this embodiment also provides a data visualization parallel I / O optimization system based on NetCDF, including... The NetCDF file optimization processing unit is used to perform fine-grained data block division on the input NetCDF file, discretize the small data blocks obtained by block division into multiple data parts, merge the small data blocks of each data part, and finally save them as the block-optimized NetCDF file. NetCDF file parallel reading program unit, used for parallel reading of block-optimized NetCDF files; The cell data structure construction program unit is used to construct virtual cells to represent cell data structures using point data so that the point sequence of cells forms a completely ordered sequence; A visualization calculation unit for performing visualization calculations using a cell data structure represented by point data; The calculation result output program unit is used to discard the virtual cell calculation results and output the visual calculation results after the visualization calculation is completed.

[0033] Furthermore, this embodiment also provides a NetCDF-based data visualization parallel I / O optimization system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the NetCDF-based data visualization parallel I / O optimization method. Additionally, this embodiment also provides a computer-readable storage medium storing a computer program for being programmed or configured by the microprocessor to execute the NetCDF-based data visualization parallel I / O optimization method.

[0034] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0035] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for NetCDF-based data visualization and parallel IO optimization, characterized in that, include: S1, perform fine-grained data block division on the input NetCDF file, discretize the small data blocks obtained from the block division into multiple data parts, merge the small data blocks of each data part, and finally save it as the block-optimized NetCDF file; S2, reads the block-optimized NetCDF file in parallel; S3, construct virtual cells to represent cell data structures using point data so that the point sequence of cells forms a completely ordered sequence; S4 performs visual calculations using a cell data structure represented by point data; S5: After the visual calculation is completed, discard the virtual cell calculation results and output the visual calculation results; Step S1 includes: S1.1, Read the input NetCDF file; S1.2, set parameters, including: data block size, number of file data hash buckets; S1.3, the input NetCDF file is divided into fine-grained data blocks based on parameters. The small data blocks obtained from the block division are discretized into coordinate data and scalar value data, and the coordinate data and scalar value data are distributed and stored in various hash buckets. The coordinate data and scalar value data in the NetCDF file are merged according to the hash bucket number to which the discrete data belongs, so as to ensure that the data in the hash bucket is stored continuously to convert the discrete fine-grained I / O into sequential coarse-grained I / O. S1.4, Output the block-optimized NetCDF file; Step S3 includes: S3.1, retrieve coordinate data and scalar value data from the NetCDF file. Each process determines the data to be processed based on the process ID and hash bucket number. Each process processes data from one or more hash buckets. S3.2, Based on the coordinate data and scalar value data, add virtual cell data at the end of each column of data so that the point sequence of the cells forms a completely ordered sequence; S3.3 uses cell data represented by point data as input for visualization calculations, so that after adding virtual cells, cell data is represented by continuous point data, and no data conversion is required when performing visualization calculations.

2. The NetCDF-based data visualization parallel IO optimization method according to claim 1, wherein, Step S1.3 includes: S1.3.1 First, the NetCDF file is divided into first-level blocks according to the number of processes used for parallel processing, so that the number of first-level blocks is the same as the number of processes used for parallel processing. S1.3.2, then divide each first-level block into second-level blocks according to the number of processes used for parallel processing, so that each first-level block contains the same number of second-level blocks used for parallel processing, and each process used for parallel processing takes turns processing the second-level blocks in order. S1.3.3, and then the second-level blocks in the hash bucket are reorganized so that multiple second-level blocks processed by each process are stored contiguously in the same hash bucket to convert discrete fine-grained I / O into sequential coarse-grained I / O. 3.The NetCDF-based data visualization parallel IO optimization method according to claim 1, wherein, The coordinate data and scalar value data are numbered in order from bottom to top and from left to right.

4. The data visualization parallel I / O optimization method based on NetCDF according to claim 1, characterized in that, The coordinate data and scalar value data are numbered in order from bottom to top and from left to right.

5. The data visualization parallel I / O optimization method based on NetCDF according to claim 1, characterized in that, Before step S1, it also includes determining whether the input NetCDF file is a block-optimized NetCDF file. If it is a block-optimized NetCDF file, then proceed to step S2. Otherwise, proceed to step S1.

6. A data visualization parallel I / O optimization system for applying the NetCDF-based data visualization parallel I / O optimization method according to any one of claims 1 to 5, characterized in that, include: The NetCDF file optimization processing unit is used to perform fine-grained data block division on the input NetCDF file, discretize the small data blocks obtained by block division into multiple data parts, merge the small data blocks of each data part, and finally save them as the block-optimized NetCDF file. NetCDF file parallel reading program unit, used for parallel reading of block-optimized NetCDF files; The cell data structure construction program unit is used to construct virtual cells to represent cell data structures using point data so that the point sequence of cells forms a completely ordered sequence; A visualization calculation unit for performing visualization calculations using a cell data structure represented by point data; The calculation result output program unit is used to discard the virtual cell calculation results and output the visual calculation results after the visualization calculation is completed.

7. A data visualization parallel I / O optimization system based on NetCDF, comprising interconnected microprocessors and memory, characterized in that, The microprocessor is programmed or configured to execute the NetCDF-based data visualization parallel I / O optimization method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by a microprocessor to execute the NetCDF-based data visualization parallel I / O optimization method according to any one of claims 1 to 5.

Images