Method for fast loading and using of seismic mass data

Abstract

This invention provides a method for rapidly loading and using massive seismic data. The method includes: Step 1, acquiring cluster environment parameters and data file information; Step 2, calculating the positional offsets of data blocks; Step 3, starting an MPI job and assigning the start and end offsets of each block to each process; Step 4, each process scanning the data blocks to obtain relevant trace header and gather information and establishing a primary index; Step 5, scanning the primary index of each block and establishing a secondary index; Step 6, establishing the association between the data file and the index file; Step 7, reading the data according to the index file. This method for rapidly loading and using massive seismic data employs block-based parallel technology, which can significantly improve the efficiency of loading and using massive seismic data. Based on MPI technology, it can meet the data requirements of many existing MPI seismic exploration algorithms.

Description

Technical Field

[0001] This invention relates to the field of seismic exploration technology, and in particular to a method for rapidly loading and using massive amounts of seismic data. Background Technology

[0002] To obtain detailed descriptions of complex underground structures, seismic exploration technology has gradually developed towards smaller area cells and higher density. This trend has led to increasingly larger volumes of seismic data, making the scanning, loading, and processing of massive amounts of data increasingly inefficient, thus affecting the efficiency of seismic exploration production.

[0003] In the field of seismic exploration, computer clusters containing multiple computing nodes are commonly used to achieve parallel operation of seismic exploration algorithms. The main technologies employed are MPI (Message Passing Interface) multi-process parallelism and OpenMP multi-threaded parallelism. These technologies improve operational efficiency by dividing the seismic exploration algorithm into multiple tasks and assigning these tasks to multiple processes (or threads) to achieve parallel processing. However, seismic algorithms based on these parallel technologies only achieve multi-task parallelism at the algorithm level and do not parallelize the input and output of data. All tasks read data from a single node or a single large data file, which often leads to parallel tasks queuing up to retrieve data, becoming a bottleneck for efficiency improvement.

[0004] One solution to this problem is to use next-generation distributed parallel frameworks such as Hadoop or Spark. These parallel technologies employ distributed storage at the underlying level, improving data read and write efficiency. However, solutions based on these parallel frameworks require customized development to address the characteristics of seismic exploration data and significant modifications to seismic exploration algorithms before they can be migrated to this framework.

[0005] Existing solutions based on MPI parallel technology suffer from efficiency bottlenecks in the data input and output stages, which are particularly evident when processing massive amounts of seismic exploration data. Solutions based on Hadoop or Spark parallel frameworks have two drawbacks. First, their support for seismic exploration data is still immature. Second, a large number of existing MPI-based algorithms need to be rewritten within the new frameworks to migrate to them, resulting in a huge workload and making it difficult to achieve significant improvements in seismic exploration data processing efficiency in the short term.

[0006] Chinese patent application CN201910950908.1 discloses a method and system for calculating seismic data based on MPI. The method may include: determining multiple minimum computational data units of the seismic data to be calculated, obtaining a seismic data file processing record table; assigning the multiple minimum computational data units to multiple computation nodes for calculation, obtaining calculated data; comparing the calculated data with the seismic data file processing record table; if they match, the seismic data to be calculated is considered complete; otherwise, the computation nodes recalculate until the calculated data matches the seismic data file processing record table.

[0007] Chinese patent application CN201310489545.9 discloses a method for extracting common imaging point gathers in the reverse-time migration offset domain, belonging to the field of seismic data imaging processing. This method includes: each computer node performing reverse-time migration processing on shot data to obtain a reverse-time migrated shot domain imaging data volume, and storing the reverse-time migrated shot domain imaging data volume on the local disk of that computer node; and extracting common imaging point gathers based on the multi-node reverse-time migrated shot domain imaging data.

[0008] Chinese patent application CN201510320303.6 discloses a custom data block partitioning method for a Hadoop file system, comprising: sorting input data; partitioning the sorted input data into blocks according to pre-set data partitioning parameters to obtain data blocks, wherein partitioning the sorted input data into blocks includes: recording the start and end positions of each data block in the sorted input data in partitioning information corresponding to each data block; and reading the corresponding data blocks from the sorted input data based on the partitioning information for parallel processing.

[0009] The existing technologies described above are significantly different from our invention and have failed to solve the technical problem we want to address. Therefore, we have invented a new method for rapidly loading and using massive amounts of earthquake data. Summary of the Invention

[0010] The purpose of this invention is to provide a method for rapidly loading and using massive seismic data based on MPI parallel technology, thereby improving processing efficiency.

[0011] The objective of this invention can be achieved through the following technical measures: a method for rapidly loading and using massive seismic data, comprising:

[0012] Step 1: Obtain cluster environment parameters and data file information;

[0013] Step 2: Calculate the position offset of the data block;

[0014] Step 3: Start the MPI job and assign the start and end offsets of each block to each process;

[0015] Step 4: Each process scans the data blocks, obtains relevant track header and track set information, and establishes a first-level index;

[0016] Step 5: Scan the primary index of each block and build the secondary index;

[0017] Step 6: Establish the association between the data file and the index file;

[0018] Step 7: Read the data from the index file.

[0019] The objective of this invention can also be achieved through the following technical measures:

[0020] In step 1, in a cluster environment with MPI configured, the cluster environment parameters are obtained, and the size of the seismic data file and the size of a single seismic trace are obtained according to the file name.

[0021] Step 1 also includes setting the number of processes used to load the data, as well as the size of each data block.

[0022] In step 2, based on the parameter settings, the position offset of the start and end tracks of each data block in the entire data file is calculated and saved.

[0023] In step 2, the data is divided into blocks based on a single seismic trace, and it is not allowed to decompose a single seismic trace into two blocks; therefore, the actual size of the data block will not be strictly equal to the set block size.

[0024] In step 4, each process scans the data between the start offset and the end offset to obtain the relevant trace head and trace set information. The information of each trace set is treated as a record, a first-level index is built, and the data is saved to the database or a file. The data of each block is output to the disk, and each block is output as a file.

[0025] In step 5, the first-level index of each block is scanned, and the relevant information of each first-level index is used as a record to create a second-level index, which is then saved to the database or a file.

[0026] In step 6, the association between the data file and the index file is established and saved to the database or a file.

[0027] In step 7, when using data, the associated information is queried based on the data file name to obtain the index file name; the secondary index of the data is retrieved based on the gather number or channel number of the required data to obtain the data block number where the required data is located; then the primary index corresponding to the block is retrieved to find the position offset of the required data in the file; based on the retrieved data block number and position offset, the data is read from the corresponding position in the corresponding data block file.

[0028] In step 7, the process of reading data from the index file runs in a multi-process environment, with each process executing independently and acquiring the required data in parallel.

[0029] The method for rapidly loading and using massive amounts of seismic data in this invention has the following advantages compared with existing technologies:

[0030] I. Compared with traditional methods of loading and using seismic data, this invention adopts block parallel technology, which can make full use of cluster resources and greatly improve the efficiency of loading and using massive seismic data.

[0031] Second, a large number of seismic exploration methods and algorithms based on MPI technology have been accumulated in the field of seismic exploration. Based on this invention, only minor interface modifications are needed to achieve efficient parallel acquisition of massive amounts of data in these methods and algorithms, thereby improving the processing efficiency of seismic exploration methods and algorithms. Attached Figure Description

[0032] Figure 1 This is a flowchart of a specific embodiment of the rapid loading process for massive earthquake data according to the present invention;

[0033] Figure 2 This is a flowchart of a specific embodiment of the rapid use of massive earthquake data according to the present invention. Detailed Implementation

[0034] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0035] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, and / or combinations thereof.

[0036] like Figure 1 As shown, Figure 1This is a flowchart of the method for rapidly loading and using massive seismic data according to the present invention. The method for rapidly loading and using massive seismic data includes:

[0037] (1) In a cluster environment with MPI configured, obtain the cluster environment parameters and obtain the size of the seismic data file and the size of a single seismic trace based on the file name.

[0038] (2) Set the number of processes used to load data, and the size of each data block.

[0039] (3) Based on the parameter settings, calculate and save the position offsets of the start and end traces of each data block within the entire data file. Data blocks are based on a single seismic trace data as the basic unit, and it is not allowed to decompose a single seismic trace into two blocks. Therefore, the actual size of the data block will not be strictly equal to the set block size.

[0040] (4) Start the MPI job and allocate the start and end offsets of each block to each process;

[0041] (5) Each process scans the data between the start offset and the end offset, obtains the relevant trace header and trace set information, treats the information of each trace set as a record, builds a first-level index, and saves it to the database or a file. Furthermore, each block of data is output to disk, with each block output as a separate file.

[0042] (6) Scan the first-level index of each block, take the relevant information of each first-level index as a record, create a second-level index, and save it to the database or file.

[0043] (7) Establish the association between the data file and the index file, and save them to the database or file.

[0044] (8) When using data, query the associated information based on the data file name to obtain the index file name. Retrieve the secondary index of the data based on the gather number or channel number of the required data to obtain the data block number where the required data is located. Then, retrieve the primary index corresponding to that block to find the position offset of the required data in the file. Based on the retrieved data block number and position offset, read the data from the corresponding position in the corresponding data block file. The above process can run in a multi-process environment, with each process executing independently and acquiring the required data in parallel.

[0045] Example 1:

[0046] In a specific embodiment 1 of the present invention, such as Figure 1 The diagram shows a flowchart of the rapid loading process for massive seismic data according to the present invention; the rapid loading process for massive seismic data includes the following steps:

[0047] (1) In a cluster environment with MPI configured, obtain the cluster environment parameters.

[0048] (2) Based on the file name, obtain file information such as the size of the earthquake data file and the size of a single seismic trace.

[0049] (3) Calculate and save the position offset of the start and end tracks of each data block in the entire data file according to the set parameters.

[0050] (4) Start the MPI job.

[0051] (5) Assign the start and end offsets of each block to each process.

[0052] (6) Each process scans the data between the starting offset and the ending offset to obtain relevant track head and track set information.

[0053] (7) Treat the information of each collection as a record, create a first-level index, and save it to a database or file. Also, output each block of data to disk, with each block output as a file.

[0054] (8) Scan the first-level index of each block and extract relevant information from the first-level index.

[0055] (9) Create a secondary index by treating the relevant information of each primary index as a record, and save it to the database or file.

[0056] (10) Establish the association between the data file and the index file, and save them to the database or file.

[0057] Example 2:

[0058] In a specific embodiment 2 of the present invention, such as Figure 2 The diagram shows a flowchart of the rapid use process for massive seismic data according to the present invention; this rapid use process for massive seismic data includes the following steps:

[0059] (1) Obtain information on the data requirements.

[0060] (2) Based on the data file name, query the related information and obtain the index file name.

[0061] (3) Based on the gather number or channel number of the demand data, retrieve the secondary index of the data and query the data block number where the demand data is located.

[0062] (4) Based on the number, retrieve the first-level index corresponding to the block and find the offset of the required data in the file.

[0063] (5) Read the data at the corresponding position in the corresponding data block file according to the data block number and position offset obtained from the query.

[0064] Example 3:

[0065] In a specific embodiment 3 of the present invention, 1.2TB of seismic data was selected and loaded using both a traditional serial method and the technology of this patent. Loading this data using the traditional serial method took approximately 7.6 hours. Using the technology of this patent, with 3 processes, the time was approximately 3.3 hours; with 6 processes, the time was approximately 1.4 hours. The comparison shows that the technology of this patent significantly improves the loading efficiency of massive amounts of data.

[0066] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0067] Except for the technical features described in the specification, all other technologies are known to those skilled in the art.

Claims

1. A method for fast loading and using seismic mass data, characterized in that, The methods for rapidly loading and using massive amounts of earthquake data include: Step 1: Obtain cluster environment parameters and data file information; Step 2: Calculate the position offset of the data block; Step 3: Start the MPI job and assign the start and end offsets of each block to each process; Step 4: Each process scans the data blocks, obtains relevant track header and track set information, and establishes a first-level index; Step 5: Scan the primary index of each block and build the secondary index; Step 6: Establish the association between the data file and the index file; Step 7: Read data from the index file; Step 1 also includes setting the number of processes used to load the data, and the size of each data block; In step 2, based on the parameter settings, the position offset of the start and end tracks of each data block in the entire data file is calculated and saved; In step 4, each process scans the data between the start offset and the end offset to obtain the relevant trace head and trace set information. The information of each trace set is treated as a record, a first-level index is built, and the data is saved to the database or a file. The data of each block is output to the disk, and each block is output as a file. In step 5, the first-level index of each block is scanned, and the relevant information of each first-level index is used as a record to create a second-level index, which is then saved to the database or a file.

2. The method for fast loading and using of seismic mass data according to claim 1, wherein, In step 1, in a cluster environment with MPI configured, the cluster environment parameters are obtained, and the size of the seismic data file and the size of a single seismic trace are obtained according to the file name.

3. The method for fast loading and using of seismic mass data according to claim 1, wherein, In step 2, the data is divided into blocks based on a single seismic trace, and it is not allowed to decompose a single seismic trace into two blocks; therefore, the actual size of the data block will not be strictly equal to the set block size.

4. The method for fast loading and using of seismic mass data according to claim 1, wherein, In step 6, the association between the data file and the index file is established and saved to the database or a file.

5. The method for fast loading and using of seismic big data according to claim 1, wherein, In step 7, when using data, the associated information is queried based on the data file name to obtain the index file name; the secondary index of the data is retrieved based on the gather number or channel number of the required data to obtain the data block number where the required data is located; then the primary index corresponding to the block is retrieved to find the position offset of the required data in the file; based on the retrieved data block number and position offset, the data is read from the corresponding position in the corresponding data block file.

6. The method for rapidly loading and using massive earthquake data according to claim 5, characterized in that, In step 7, the process of reading data from the index file runs in a multi-process environment, with each process executing independently and acquiring the required data in parallel.

Images