Statistics and analysis method and statistical analysis system for geophysical data

Abstract

The invention belongs to the technical field of geophysical data processing, and particularly relates to a statistical analysis method and statistical analysis system for geophysical data, which comprises an initialization module, a data calculation module, a data statistical module and a data analysis module. Scanning the storage space to obtain a storage path of the geophysical data; recording the storage path into a result file; cutting the result file to form N sub-files, and creating corresponding N sub-tasks; distributing each subtask to each computing node for parallel computing to obtain feature information of the geophysical data; and performing statistics according to the feature information of the geophysical data. According to the method, the use condition of the user storage space of 1000TB data can be counted within half an hour, the efficiency is greatly improved, the compression potential of the data can be analyzed, the data repetition rate is judged, the data capacity growth rate is mastered, whether the data can be filed or not is determined, and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of geophysical data processing, and in particular relates to a geophysical data statistics and analysis method and a statistical analysis system. Background Art

[0002] The oil and gas industry is an industry related to national energy security. Geophysical technology is the core technology of oil and gas exploration and development. It is highly dependent on information technology, especially the support of professional software and hardware. At present, the application of ultra-large-area "two wides and one high" (wide azimuth, wide bandwidth, high density) high-precision three-dimensional seismic acquisition technology in the northwest region provides processing personnel with seismic data with higher lateral resolution from the source, which brings convenience to the detailed characterization of fracture-cavity reservoirs, but also puts forward higher requirements on computer storage space and IO rate.

[0003] In the research and production of geophysics, a huge amount of geophysical data will be generated. The characteristics of geophysical data include: 1) There are many data formats, including SEG-Y, SEG-D, DIO and other formats; 2) The data volume is large, and a single data can reach tens of GB to hundreds of GB; 3) These data belong to non-relational databases and are stored in storage spaces of different types and manufacturers. It will take too long to count and analyze all the data at once, and the amount of data is also expanding. The total amount of data has exceeded 40PB, and an average of 800TB to 1000TB of data is added every month. Summary of the invention

[0004] The purpose of the present invention is to solve the above-mentioned problems existing in the prior art and to provide a statistical method for geophysical data.

[0005] The present invention is implemented by the following technical solution: A statistical method for geophysical data comprises the following steps:

[0006] Scan the storage space to obtain the storage path of geophysical data;

[0007] Record the storage path into the result file;

[0008] The result file is cut into N sub-files, and corresponding N sub-tasks are created;

[0009] Distribute each subtask to each computing node for parallel computing to obtain characteristic information of geophysical data;

[0010] Statistics are performed based on the characteristic information of geophysical data.

[0011] Furthermore, the steps of computing the subtask by the computing node include:

[0012] Extract the storage path in the subtask;

[0013] Scan the geophysical data in the folder corresponding to the storage path and record the characteristic information accordingly.

[0014] Furthermore, the computing nodes to which the subtasks are distributed are locked and unlocked after the computing is completed.

[0015] Furthermore, the characteristic information includes data name, data format, data size, occupied storage space, creation time, modification time, project characteristics and the user to whom it belongs.

[0016] Furthermore, the step of regularizing the characteristic information includes: arranging the original data according to a standard format; and cleaning the original data that does not need to be counted.

[0017] Furthermore, the statistical content includes: statistics on the storage space occupied by each project and / or multi-dimensional statistics including data format, data size, and time dimension according to the needs of massive data management.

[0018] The present invention also provides a geophysical data analysis method, which uses the geophysical data statistical method of the present invention to obtain statistical results and perform analysis based on the statistical results.

[0019] Furthermore, it includes at least one of the following: project operation status analysis, archiving analysis and data attribute analysis.

[0020] Furthermore, the data attribute analysis includes compression potential analysis, repeatability analysis and data capacity growth rate analysis, and the corresponding indicators are calculated according to the following formulas:

[0021] Compression ratio:

[0022] In the formula, P represents the compression ratio; N represents the total number of data with a size less than 1M; p n Indicates the data size of a single data that is less than 1M;

[0023] Repetition rate:

[0024] In the formula, U represents the repetition rate; F represents the size of the total amount of data in the data set; E represents the size of the repeated data in the data set, and the data that appears at least twice in the data set is repeated data;

[0025] growth rate:

[0026] In the formula, K represents the data capacity growth rate, B represents the data set capacity in the current statistical period, and A represents the data set capacity in the previous statistical period.

[0027] Furthermore, the operation status of the project is tracked and analyzed based on the daily storage space occupied by the physical earth data of the same project; archiving analysis is performed based on the time interval from the creation time to the modification time of the physical earth data of the same project, and if the time interval exceeds the threshold, archiving suggestions are given.

[0028] The present invention also provides a geophysical data statistical analysis system, comprising an initialization module, a data calculation module, a data statistics module and a data analysis module;

[0029] An initialization module is used to scan the storage space, obtain the storage path of the geophysical data, record the storage path in the result file, cut the result file into N sub-files, and create corresponding N sub-tasks;

[0030] A data computing module is used to process subtasks in parallel through computing nodes, each computing node is used to extract the storage path in the subtask, scan the geophysical data in the folder corresponding to the storage path and record the characteristic information accordingly;

[0031] Data statistics module, which performs statistics based on the characteristic information of geophysical data;

[0032] The data analysis module is used to perform analysis based on statistical results.

[0033] Compared with the prior art, the beneficial effects of the present invention include:

[0034] 1. The present invention divides subtasks into parallel calculations and can count the user storage space usage of 1000TB of data within half an hour, greatly improving efficiency.

[0035] 2. Scan the original data according to the storage path. Due to parallel computing, data in different storage spaces can be scanned simultaneously without the need to store the data in a centralized manner, thus reducing the data center's requirements for computer storage space.

[0036] 3. Be able to evaluate the number of files you have, the distribution of your data, understand the compression potential of your data, determine the data duplication rate, understand the data capacity growth rate, determine whether the data can be archived, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the architecture diagram of the geophysical data statistical analysis system;

[0038] Figure 2 It is a statistical chart of the number of files and data volume in each path;

[0039] Figure 3 This is a data overview diagram;

[0040] Figure 4It is a statistical chart of data format;

[0041] Figure 5 It is a statistical chart of data size;

[0042] Figure 6 It is a statistical chart of the time dimension of the data. DETAILED DESCRIPTION

[0043] The following is a further detailed description with reference to the accompanying drawings.

[0044] In the scientific research and production activities in the field of geophysics, a large amount of seismic acquisition, processing and interpretation data will be generated. System administrators need to conduct detailed statistical analysis on these data, monitor the resource usage of the data center through statistical analysis, and provide decision makers with resource usage reports.

[0045] As seismic data continues to increase, the amount of three-dimensional seismic data has increased from GB to TB, showing an order of magnitude increase. Storage space has also expanded from TB to PB, which has added difficulties for system administrators in data management and statistical analysis without effective means and tools.

[0046] In order to process massive geophysical data, a statistical method for geophysical data is proposed, which includes the following steps:

[0047] Scan the storage space to obtain the storage path of geophysical data;

[0048] Record the storage path into the result file;

[0049] The result file is cut into N sub-files, and corresponding N sub-tasks are created;

[0050] Distribute each subtask to each computing node for parallel computing.

[0051] Preferably, the step of computing the subtask by the computing node includes:

[0052] Extract the storage path in the subtask;

[0053] Scan the geophysical data in the folder corresponding to the storage path and record the characteristic information accordingly.

[0054] Preferably, the computing node to which the subtask is distributed is locked, and unlocked after the computing is completed.

[0055] Preferably, the characteristic information includes data name, data format, data size, occupied storage space, creation time, modification time, project characteristics and the user to whom it belongs.

[0056] Preferably, the step of regularizing the characteristic information includes: arranging the original data according to a standard format; and cleaning the original data that does not need to be counted.

[0057] Preferably, the statistical content includes: statistics on the storage space occupied by each project and / or multi-dimensional statistics including time, space, and users according to the needs of mass data management.

[0058] Based on the statistical method of geophysical data, a geophysical data analysis method is proposed, the statistical result is obtained by using the statistical method of geophysical data, and analysis is performed according to the statistical result.

[0059] Preferably, it includes at least one of project operation status analysis, archiving analysis and data attribute analysis.

[0060] Preferably, the data attribute analysis includes compression potential analysis, repeatability analysis and data capacity growth rate analysis, and the corresponding indicators are calculated according to the following formulas:

[0061] Compression ratio:

[0062] In the formula, P represents the compression ratio; N represents the total number of data with a size less than 1M; p n Indicates the data size of a single data that is less than 1M;

[0063] Repetition rate:

[0064] In the formula, U represents the repetition rate; F represents the total amount of data in the data set; E represents the size of repeated data in the data set, and data that appears at least twice in the data set is repeated data;

[0065] growth rate:

[0066] In the formula, K represents the data capacity growth rate, B represents the data set capacity in the current statistical period, and A represents the data set capacity in the previous statistical period.

[0067] Preferably, the operation status of the project is tracked and analyzed based on the storage space occupied by the physical earth data of the same project; archiving analysis is performed based on the time interval from the creation time to the modification time of the physical earth data of the same project, and if the time interval exceeds a threshold, archiving suggestions are given.

[0068] refer to Figure 1 As shown, in order to realize the statistical and analytical methods of geophysical data, a geophysical data statistical analysis system is proposed, which includes four modules: program initialization, data calculation, data statistics, and data analysis;

[0069] An initialization module is used to scan the storage space, obtain the storage path of the geophysical data, record the storage path in the result file, cut the result file into N sub-files, and create corresponding N sub-tasks;

[0070] A data computing module is used to process subtasks in parallel through computing nodes, each computing node is used to extract the storage path in the subtask, scan the geophysical data in the folder corresponding to the storage path and record the characteristic information accordingly;

[0071] Data statistics module, which performs statistics based on the characteristic information of geophysical data;

[0072] The data analysis module is used to perform analysis based on statistical results.

[0073] Each module will be described in more detail below in conjunction with each embodiment.

[0074] Example 1

[0075] Initialize the module:

[0076] (1) Clean up cache files and junk files in the program;

[0077] (2) Perform a health check on the operating environment, including the programs that need to be called, the system environment, and the server load.

[0078] This specific implementation is mainly based on a program written in a shell environment under a Linux system environment.

[0079] In the environment of a high-performance computing cluster of the Linux system, a parallel computing statistical analysis method is implemented using shell scripts to count the detailed information of the data in each storage space, analyze the storage resource usage of each user, and provide effective suggestions such as data compression, deduplication, and archiving.

[0080] Shell script: Shell is a program with special functions. It is an interface between the user and the core program (kernel) of the UNIX / Linux operating system. Shell scripts are most commonly used for system management or to combine existing programs to complete small, specific tasks. Once you find a way to complete the work, you can string the commands together and put them into a separate program or script. In the future, you can just execute the program directly to complete the work.

[0081] (3) By calling the find command to scan, the required information is retrieved, sorted and recorded according to the path characteristics of the geophysical data, and the acquisition rules of the storage path information can be customized according to different needs. The more specific steps are as follows:

[0082] ① Scan the storage space, retrieve all files and directories in the third level of the directory tree, and record them in the result file (i.e., find file). The scan results are referenced in Figure 2 shown.

[0083] ② Split the result file into N sub-files. The number of sub-files N can be the number of storage paths or the number of computing nodes.

[0084] ③Schedule the sub-file to the computing node for calculation.

[0085] Example 2

[0086] Data calculation module:

[0087] (1) Lock the computing node to which the subtask is distributed and unlock it after the computation is completed.

[0088] (2) Call the open source program ncdu, extract the storage path provided in the subtask, scan the original data in the folder corresponding to the storage path and record the characteristic information; the characteristic information includes data name, data format, data size, occupied storage space, creation time, modification time, project characteristics and the user to whom it belongs.

[0089] (3) The recorded characteristic information is documented.

[0090] Preferably, the characteristic information is processed in a regularized manner, and the regularized characteristic information is arranged in a standardized format for easier statistical analysis.

[0091] Example 3

[0092] Data statistics module:

[0093] Statistics are collected based on the characteristic information of geophysical data. The statistical content includes: statistics on the storage space occupied by each project and / or multi-dimensional statistics based on the needs of massive data management.

[0094] (1) Count the storage space of each project according to the project characteristics of geophysical data.

[0095] Geophysical data has its own data format. By analyzing the data name or suffix, a label is added to each data. (For example, the data belongs to the original data, intermediate process data, achievement data, etc. of the project)

[0096] (2) Statistics are collected in multiple dimensions according to the needs of massive data management, including data format, data size, time dimension, etc. The statistical results are respectively referred to Figure 4 , Figure 5 , Figure 6 shown.

[0097] Example 4

[0098] Data Analysis Module:

[0099] Conduct analysis based on statistical results, including project operation status analysis, archiving analysis and data attribute analysis.

[0100] (1) Track the operation of the project through comprehensive analysis of the project data and combined with the operation characteristics of the geophysical project. Track the operation of the project based on the daily storage space occupied by the physical geodata of the same project;

[0101] (2) By analyzing the attributes such as the creation time and the last modification time of the data, combined with the requirements of the data archiving management regulations, archiving suggestions are given for the data. The archiving analysis is performed based on the time interval from the creation time to the modification time of the physical earth data of the same project. If the time interval exceeds the threshold, an archiving suggestion is given. For example, the time interval from the creation time to the last modification time of the project is one and a half years, and the project only needs one year of operation. The project with a policy time interval of more than half a year needs to be archived.

[0102] (3) Data attribute analysis includes compression potential analysis, repeatability analysis and data volume growth rate analysis. The corresponding indicators are calculated according to the following formulas:

[0103] Compression ratio:

[0104] In the formula, P represents the compression ratio; N represents the total number of data with a size less than 1M (if the data before compression is less than 1M, the storage will also be calculated based on the space occupied by 1M); p n Indicates the data size of a single data that is less than 1M;

[0105] Repetition rate:

[0106] In the formula, U represents the repetition rate; F represents the total amount of data in the data set; E represents the size of repeated data in the data set, and data that appears at least twice in the data set is repeated data;

[0107] growth rate:

[0108] In the formula, K represents the data capacity growth rate, B represents the data set capacity in the current statistical period, and A represents the data set capacity in the previous statistical period. For example, B is the data set capacity of the second day or the second month; A is the data set capacity of the first day or the first month.

[0109] The above technical scheme is only a specific implementation method of the present invention. For those skilled in the art, it is easy to make various types of improvements or modifications based on the principles disclosed in the present invention, and it is not limited to the technical scheme described in the above specific embodiments of the present invention. Therefore, the above description is only preferred and does not have a restrictive meaning.

Claims

1. A statistical method for geophysical data, characterized in that: The following steps are involved: Scan the storage space to obtain the storage path of geophysical data; Record the storage path into the result file; The result file is cut into N sub-files, and corresponding N sub-tasks are created; Distribute each subtask to each computing node for parallel computing to obtain characteristic information of geophysical data; Statistics are performed based on the characteristic information of geophysical data.

2. The geophysical data statistical method according to claim 1, characterized in that: Features: The steps for computing subtasks by computing nodes include: Extract the storage path in the subtask; Scan the geophysical data in the folder corresponding to the storage path and record the characteristic information accordingly.

3. The geophysical data statistical method according to claim 2, characterized in that: The computing nodes to which the subtasks are distributed are locked and unlocked after the computation is completed.

4. The geophysical data statistical method according to claim 2, characterized in that: The characteristic information includes data name, data format, data size, occupied storage space, creation time, modification time, project characteristics and the user to whom it belongs.

5. The geophysical data statistical method according to claim 1, characterized in that: The statistical content includes: statistics on the storage space occupied by each project and / or multi-dimensional statistics including data format, data size, and time dimension based on the needs of massive data management.

6. A geophysical data analysis method, characterized in that: The statistical results are obtained by using the geophysical data statistical method as described in any one of claims 1 to 5, and analysis is performed based on the statistical results.

7. The geophysical data analysis method according to claim 6, characterized in that: It shall include at least one of the following: project operation status analysis, archiving analysis and data attribute analysis.

8. The geophysical data analysis method according to claim 7, characterized in that: The data attribute analysis includes compression potential analysis, repeatability analysis and data capacity growth rate analysis, and the corresponding indicators are calculated according to the following formulas: Compression ratio: In the formula, P represents the compression ratio; N represents the total number of data with a size less than 1M; p n Indicates the data size of a single data that is less than 1M; Repetition rate: In the formula, U represents the repetition rate; F represents the total amount of data in the data set; E represents the size of repeated data in the data set, and data that appears at least twice in the data set is repeated data; growth rate: In the formula, K represents the data capacity growth rate, B represents the data set capacity in the current statistical period, and A represents the data set capacity in the previous statistical period.

9. The geophysical data analysis method according to claim 8, characterized in that: Track the operation of the project based on the daily storage space occupied by the physical earth data of the same project; conduct archiving analysis based on the time interval from the creation time to the modification time of the physical earth data of the same project. If the time interval exceeds the threshold, give archiving suggestions.

10. A geophysical data statistical analysis system, characterized in that: It includes initialization module, data calculation module, data statistics module and data analysis module; An initialization module is used to scan the storage space, obtain the storage path of the geophysical data, record the storage path in the result file, cut the result file into N sub-files, and create corresponding N sub-tasks; The data computing module is used to process subtasks in parallel through computing nodes, each computing node is used to extract the storage path in the subtask, scan the geophysical data in the folder corresponding to the storage path and record the characteristic information accordingly; Data statistics module, which performs statistics based on the characteristic information of geophysical data; The data analysis module is used to perform analysis based on statistical results.

Images