Parallel read-in method of ten-billion and hundred-gigabit GB-magnitude grid data file

A grid data and file technology, applied in the field of data processing, can solve the problems of unacceptable CFD calculation efficiency and the decrease of file reading and writing speed, so as to achieve the effect of improving calculation efficiency and economic benefits, and saving reading time.

Pending Publication Date: 2021-03-09
AERODYNAMICS NAT KEY LAB
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, when the amount of grids reaches tens of billions, the grid files and flow field files will reach hundreds of gigabytes. If the data is still stored in a single file and a single process is used for serial reading and writing, the file reading and writing speed It is bound to drop sharply, making the CFD calculation efficiency unacceptable

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Parallel read-in method of ten-billion and hundred-gigabit GB-magnitude grid data file
  • Parallel read-in method of ten-billion and hundred-gigabit GB-magnitude grid data file
  • Parallel read-in method of ten-billion and hundred-gigabit GB-magnitude grid data file

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] The present invention will be further described below in conjunction with the accompanying drawings.

[0023] The invention proposes a method for parallel reading in tens of billions of gigabytes of grid data files. Super-large-scale grid data files generated for multiple objects are stored in groups. Each group includes multiple files, and each file Contains multiple data partitions; when reading, multiple file processes are used to read files and send them to corresponding non-file processes for data load balancing.

[0024] In CFD calculation, the main large-scale files are grid files and flow field files. These files contain the topology and flow field values ​​of each discrete unit, which require a large amount of storage space, such as an unstructured grid with tens of millions of units. , under the framework of the second-order finite volume method, its occupied storage is about 1.5GB, and its flow field file size is about 5GB; but for a billion-level unstructure...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a parallel read-in method of a ten-billion and hundred-gigabit GB-magnitude grid data file, which is characterized in that a super-large-scale grid data file generated by a plurality of objects is stored by adopting a grouped file, each group comprises a plurality of files, and each file comprises a plurality of data partitions; and during reading, a plurality of file processes are used to read the files, and thus sending the files to the corresponding non-file processes to perform data load balancing allocation. By adopting the method provided by the invention, the IO efficiency of the grid file can be greatly improved, and the time consumption of the technical scheme provided by the invention is only 1 / 50-1 / 10 of the time consumption of the prior art when the samesuper-large-scale grid data is read. According to the invention, the grid data read-in time can be greatly saved, and the calculation efficiency and the economic benefit are improved.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a method for parallel reading in grid data files of tens of billions of gigabytes. Background technique [0002] The present invention relates to that with the rapid development of computer technology and numerical methods, computational fluid dynamics (Computational Fluid Dynamics, CFD) numerical simulations have been more and more widely used in aerospace and other fields. After decades of development, the conventional state aerodynamic force / moment prediction based on the Reynolds Averaged Navier-Stokes (RANS) equation has not been too difficult, but when encountering vortex, separation, transition, turbulent noise, turbulent combustion When the unsteady and nonlinear flow is obvious, solving the RANS equation on tens of millions of grids can no longer obtain a sufficiently accurate numerical solution. At this time, it is necessary to use a larger-scale grid and a higher-fidelit...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50G06F16/16
CPCG06F9/505G06F9/5072G06F16/16
Inventor 王年华常兴华赵钟张来平
Owner AERODYNAMICS NAT KEY LAB
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap