Index construction method, index library and query method for massive seismic data

Abstract

The present invention belongs to the technical field of big data processing, and discloses an index construction method, an index library and a query method for massive seismic data, wherein the disclosed index construction method for massive seismic data is to first create a data index analysis file of the first keyword of the index based on MapReduce, and to perform keyword-node allocation by reading the file, so that in the subsequent Map partitioning stage when creating the index file based on the MapReduce parallel programming framework, the data allocated to each Reducer process is orderly, and finally each Reducer process outputs the index file in the form of a multi-level data table respectively, and obtains an overall orderly sorting result. The biggest innovation of the index construction method for massive seismic data is that the target data is executed twice MapReduce operation, and it is chosen not to concentrate all the output results on the main node for reading, sorting and merging in the final stage, which solves the problem that the merging time increases significantly with the increase of the amount of seismic data, and effectively improves the efficiency of creating indexes for PB-level massive seismic data.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and relates to the processing of seismic data in the field of geophysical exploration, and specifically to an index construction method, an index library and a query method for massive seismic data. Background Art

[0002] Seismic data processing is an important technology in the oil exploration industry. Its function is to process and calculate seismic data collected in the field according to specific processing algorithms, so as to obtain images of underground geological structures for guiding subsequent drilling, oil extraction and other work.

[0003] With the continuous application of new exploration technologies and high-precision acquisition technologies in oil exploration, the amount of raw seismic data collected from the field has increased rapidly. At present, the amount of raw seismic data collected in a single work area has exceeded the PB level, and the number of seismic channels can reach hundreds of billions. The seismic channel consists of two parts: the channel header and the channel body. The channel header stores the attribute information related to the seismic channel, including: shot point coordinates, detection point coordinates, sampling point number, shot number, channel number and other information. Each attribute is called a channel header keyword; the channel body is a floating point array, and each floating point number is called a sampling point. At the same time, the seismic data body is a high-dimensional structured data. Each seismic channel has hundreds of attribute information, which are stored in different channel header keywords.

[0004] However, when interactive seismic applications access seismic data volumes, they are usually only interested in a portion of the data set of the seismic data volume. Therefore, in order to improve the diversity and response time of data gather information obtained by interactive seismic applications when accessing seismic data volumes, seismic data processing is usually required for PB-level massive seismic data.

[0005] At present, in the process of seismic data processing, the index structure used to access seismic data index keyword information is mainly implemented as a multi-level association table data structure, which mainly includes: index file, two-level index file and multi-level keyword information file. The data table at each level is stored independently on the disk.

[0006] There are two ways to create a keyword information table:

[0007] The first is a single-node multi-threaded method. After the program reads all the trace headers, it extracts the keywords that need to be indexed and generates index records to cache in memory. Then, it uses the multi-threading of the TBB library to sort these records in parallel and output them to the storage device in the form of a multi-level data table. Since this method can only use up to 10 threads, it is not suitable for use when the amount of seismic data is relatively large;

[0008] Another way is to generate indexes and sort them by segments based on the MapReduce parallel programming framework, generate multiple ordered index files, and finally physically merge all the output results on the master node to obtain the index data table. However, in order to ensure the physical order of the index data table, this method requires that all the output results generated in the Reduce stage be concentrated on the master node for reading, sorting and merging in the final stage, which greatly increases the time consumption of merging as the amount of seismic data increases. Summary of the invention

[0009] To solve the above deficiencies in the prior art, the present invention provides an index construction method, an index library and a query method for massive seismic data, wherein the disclosed index construction method for massive seismic data aims to improve the construction efficiency of massive seismic data indexes.

[0010] To achieve the above object, the technical solution adopted by the present invention is as follows: a method for indexing massive seismic data, which is implemented based on the MapReduce parallel programming framework and specifically includes the following steps:

[0011] S1. Analyze job parameters: Set the job parameters required for subsequent MapReduce job execution;

[0012] S2. Constructing a data index analysis file: Based on the job parameters obtained in step S1, start the MapReduce job and control the job flow, count the values ​​of the first keyword of the index from small to large and the corresponding number of channels, and generate a data index analysis file;

[0013] S3. Build data index file: Also based on the job parameters obtained in step S1, start the MapReduce job and control the job flow. In the Map stage, count all index keywords and the corresponding channel number information. In the Map partition stage, arrange all index keywords in order, and read the data index analysis file generated in step S2 to distribute all index keywords and the corresponding channel number information to the Reducer process in order. In the Reduce stage, each Reducer process sorts the received data and directly outputs the generated index file in the form of a multi-level association table.

[0014] As a limitation of the present invention, step S1 includes: obtaining seismic data information provided by a user; obtaining keyword information required for creating an index; obtaining a Hosts file; obtaining the number of Reduce processes; and obtaining a temporary file storage path.

[0015] As a further limitation of the present invention, step S2 comprises:

[0016] S2.1, Initial stage: Split the trace header of seismic data into multiple Map tasks and dynamically assign them to each Mapper process;

[0017] S2.2, Map stage: Each Mapper process reads the data block information obtained in the Map task, reads the header data of each track, extracts the first keyword of the index in the track header to form a Key, reads the corresponding data track from the storage to form a Value, and stores it in the Key / Value pair;

[0018] S2.3, Map partitioning stage: Call the partitioning function GetPartition to get the Reducer process number corresponding to the Key value, so as to evenly partition the Key / Value pair and assign it to the corresponding Reduce task. In each partition, the comparison function class will be called to sort the Key values ​​from small to large, and finally output them to the local storage medium of the node where each Mapper process is located;

[0019] S2.4, Reduce stage: After the Reducer process receives all the Map outputs mapped to the Reducer process, it calls the Key comparison function class to sort all the data mapped to the Reducer process and stores all the Values ​​corresponding to the same Key together; in the final Reduce task, all the Values ​​with the same Key in the Reducer process are accumulated to obtain the number of records corresponding to the first keyword of each index;

[0020] S2.5, output stage: read the output results of each Reducer process, sort them from small to large according to the first keyword of the index, then aggregate all files into a data index analysis file, and finally output it to the storage device.

[0021] As a further limitation of the present invention, in step S2.1, each Map task processes 4096MB of data, divides the header of the seismic data into blocks, and calculates the range of seismic data that needs to be read for each data block.

[0022] As a further limitation of the present invention, step S3 includes:

[0023] S3.1, Initial stage: Split the header of seismic data into multiple Map tasks and dynamically assign them to each Mapper process;

[0024] S3.2, Map stage: Each Mapper process reads the acquired data block information, reads the header data one by one, and extracts all the index keywords in the header of the track in turn to form a Key, reads the corresponding data track from the storage to form a Value, and stores it in the Key / Value pair;

[0025] S3.3, Map partitioning stage: first rewrite the comparison function class, and use the rewritten comparison function class to arrange all index keywords in order; then rewrite the partitioning function GetPartition, read the data index analysis file generated in step S2, and use the rewritten partitioning function GetPartition to distribute all index keywords and corresponding channel number information to the Reducer process in order;

[0026] S3.4, Reduce stage: After the Reducer process receives all Map outputs mapped to the Reducer process, it sorts the data and directly outputs the index file in the form of a multi-level association table on the storage device.

[0027] As a further limitation of the present invention, in step S3.3, after rewriting the comparison function class, all index keywords need to be parsed out in sequence for comparison, specifically:

[0028] First, sort the first keywords of the index from small to large. If the first keywords of the index are equal, continue to compare the next index keyword, and so on. If the index keywords of two tracks are relative, compare the track position information until all index keywords are arranged in order.

[0029] As a further limitation of the present invention, in step S3.3, the specific method of partitioning using the rewritten partitioning function GetPartition is:

[0030] Get the first keyword Key1 of the index from Key, divide Key1 into numPartitions groups, and the number of groups is the same as the number of Reducer processes;

[0031] Read the data index analysis file generated in step S2, accumulate the number of tracks of the first keyword of each index to obtain the total number of tracks; divide the total number of tracks by the number of Reducer processes according to the average distribution principle to obtain the number of tracks that each Reducer process needs to process;

[0032] According to the number of records that each Reduce task needs to process, the number of records corresponding to Key1 in the numPartition0 grouping file is accumulated line by line. When it is less than the number of records that the Reduce0 task needs to process, all Values ​​corresponding to Key1 are divided into the Reduce0 task. When it is greater than the number of records that the Reduce0 task needs to process, the corresponding redundant Values ​​are divided into the Reduce1 task. Then, the number of records of the redundant Values ​​is accumulated line by line with the number of records corresponding to Key1 in the numPartition1 grouping file. When it is less than the number of records that the Reduce1 task needs to process, all Values ​​corresponding to Key1 are divided into the Reduce1 task. When it is greater than the number of records that the Reduce1 task needs to process, the corresponding redundant Values ​​are divided into the Reduce2 task. This process is repeated until the Reduce process number corresponding to the value of the first keyword of each index is obtained.

[0033] Finally, the data is output to the local storage medium of each node where the Mapper process is located.

[0034] The present invention also discloses a massive seismic data index library, which is created by the above-mentioned massive seismic data index construction method, and comprises:

[0035] Index file, which stores the number of seismic data channels processed by the current Reducer process, all index keywords and types;

[0036] The first keyword index file stores the value of the first keyword, the number of corresponding seismic channels, and the position in other keyword index files;

[0037] Other keyword index files, which store the values ​​of other index keywords and the corresponding seismic channel numbers;

[0038] A multi-level keyword information file, the number of files is the number of index keywords minus one, which stores the value of the current index keyword, the minimum value of the next-level index keyword, the maximum value of the next-level index keyword, the number of next-level index keywords, and the position of the index keyword in the next-level index keyword information file.

[0039] As a limitation of the present invention, it also includes a total index file, which stores the total number of seismic data channels, all index keywords and types, a flag for creating an index library using an index construction method for massive seismic data, and the number of Reducer processes.

[0040] The present invention also discloses a method for querying massive seismic data. Based on the massive seismic data index library, the method first obtains the number of Reduce processes by reading the total index file, and then searches the corresponding ReduceNum index files according to the index keyword information of each level to be queried.

[0041] The massive seismic data index library is created by the index construction method for massive seismic data as described above.

[0042] As a limitation of the present invention, when it is necessary to query seismic data in the range of index keywords Key1 to KeyN, the query is performed in the first keyword index file and other keyword index files, including the following steps:

[0043] A1. According to the obtained number of Reduce processes, find the index items that meet the Key1 condition by querying the first keyword index file;

[0044] A2. According to the index table position corresponding to the queried Key1, locate other keyword index files and read the index data corresponding to the position;

[0045] A3. Continue to read other keyword index files in the order of Reduce process numbers until all index items that meet the conditions of Key2 to KeyN are screened out in turn;

[0046] A4. Read the corresponding seismic channel data according to the seismic channel number stored in the index item.

[0047] As another limitation of the present invention, when it is necessary to query index keyword information, the query is performed in the multi-level keyword information file, specifically:

[0048] First, search for qualified keyword information in all ReduceNum first-level keyword information files in turn, and then read from the second-level keyword information file according to the corresponding position in the next-level keyword information file, and so on, until the required index keyword information is obtained.

[0049] Due to the adoption of the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0050] The index construction method for massive seismic data disclosed in the present invention relies on the MapReduce parallel programming framework. Its greatest innovation lies in executing two MapReduce jobs on the massive seismic data, and choosing not to concentrate all output results on the main node for reading, sorting and merging in the final stage. This solves the problem that the merging time increases significantly with the increase in the amount of seismic data, and effectively improves the efficiency of creating indexes for PB-level massive seismic data.

[0051] Specifically, the first MapReduce job is to create a data index analysis file, which only counts the values ​​of the first keyword of the index from small to large and the corresponding number of channels. In the Map partitioning phase of the subsequent second MapReduce job, the keyword-node allocation is performed by reading this file, so that the data allocated to each Reducer process is orderly.

[0052] The second MapReduce job is to create a data index file (i.e., an index created for massive seismic data). This file counts all index keywords and the corresponding channel number information. Since the data assigned to each Reducer process is ordered, each Reducer process outputs an index file in the form of a multi-level data table, and the overall ordered sorting result is obtained. Therefore, there is no need to concentrate all the results on the master node for reading, sorting and merging, and they can be directly output to the storage device. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] Figure 1 This is a flowchart of building a data index analysis file based on MapReduce in an embodiment of the present invention;

[0055] Figure 2 This is a flow chart of building a data index file based on MapReduce in an embodiment of the present invention;

[0056] Figure 3 The following is a query flow chart of index keyword information in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and understand the present invention, and are not used to limit the present invention.

[0058] It should be noted in advance that GPP (GeoEast Parallel Programming Framework) is a parallel programming framework independently developed by PetroChina Oriental Geophysical Company, which has the characteristics of high performance, high reliability and high scalability. MapReduce developed on this basis provides programmers with the corresponding interface for writing distributed programs, and can provide functions such as task allocation, load balancing, runtime fault tolerance and slow node processing. The main idea of ​​MapReduce is to decompose tasks into Map (mapping) and Reduce (reduction). After the data is divided, the Map function is used to map the data into different blocks, which are assigned to computer clusters for processing to achieve the effect of distributed computing, and then the Reduce function is used to aggregate the results to output the final required results.

[0059] This embodiment discloses a method for indexing massive seismic data. The implementation of this method mainly relies on the MapReduce parallel programming framework in the above-mentioned GPP. First, a data index analysis file for the first keyword of the index is created based on MapReduce. The keyword-node allocation is performed by reading the file, so that in the subsequent Map partitioning stage when the index file is created based on the MapReduce parallel programming framework, the data allocated to each Reducer process is ordered. Finally, each Reducer process outputs index files in the form of multi-level data tables to obtain an overall ordered sorting result.

[0060] The index construction method for massive seismic data disclosed in this embodiment mainly includes three parts: parsing operation parameters, constructing data index analysis files, and constructing data index files.

[0061] S1. Analyze job parameters

[0062] The purpose of this step is to set the job parameters required for the subsequent MapReduce job running process, including:

[0063] S1.1. Obtain the seismic data information provided by the user, including database, project, work area, survey line and seismic data name, etc.;

[0064] S1.2. Obtain the keyword information required to create an index. 1 to 5 keywords are supported. The type of each keyword can be integer or floating point.

[0065] S1.3. Get the Hosts file. Hosts is a list of nodes where MapReduce jobs are executed. The number of processes started on each node can be configured.

[0066] S1.4. Get the number of Reducer processes, which is the number of processes started when the MapReduce job is executed in step S1.3 / 2.0*0.8;

[0067] S1.5. Get the temporary file storage path, which is used to store the temporary results generated by the Mapper process when the MapReduce job is running. The file will be automatically deleted after the job is completed.

[0068] S2. Build data index analysis file

[0069] Based on the job parameters obtained in step S1, the MapReduce job is started and the job flow is controlled, and finally a data index analysis file is generated. The file contains statistics of the values ​​of the first keyword of the index from small to large and the corresponding number of channels, which is mainly used for keyword-node allocation when designing the global sorting algorithm in step S3;

[0070] Among them, the type names needed in the running process of MapReduce jobs mainly include Mapper, Reducer, InputFormat, OutputFormat, Partitioner for job division, and Comparator;

[0071] The specific process is as follows Figure 1 As shown, it mainly includes the following steps:

[0072] S2.1, Initial stage: InputFormat provides a way to split the input data (i.e., seismic data). The header of the seismic data is split into multiple Map tasks according to a fixed data block size (4096MB), and the seismic data range to be read for each data block is calculated (i.e., the starting offset of the seismic track and the number of tracks to be read for each block), and then these Map tasks are dynamically allocated to each Mapper process;

[0073] S2.2, Map stage: This stage only counts the first keyword of the index and the corresponding number of tracks. Specifically, after each Mapper process receives a Map task, it first completes the initialization of the job parameters and the creation of the buffer, and then reads the acquired data block information according to the RecordReader provided by InputFormat, reading the track header data one track at a time, and extracting the first keyword of the index in the track header to form the Key, reading the corresponding data track from the storage to form the Value, and storing it in the Key / Value pair;

[0074] S2.3, Map partitioning stage: MapReduce will call the partitioning function GetPartition to get the Reducer process number corresponding to the Key value, so as to evenly distribute the Key / Value pairs to the corresponding Reduce tasks. The number of partitions is the same as the number of Reducer processes. In each partition, the comparison function class will be called to sort the Key values ​​from small to large, and finally output to the local storage medium of the node where each Mapper process is located. The result files generated by each subsequent Reduce will be ordered internally, but not sorted between files;

[0075] By default, MapReduce provides a comparison algorithm between single keys and an algorithm for allocating data to Reduce.

[0076] S2.4, Reduce stage: When the Reducer process receives all Map outputs mapped to the Reducer process, it calls the Key comparison function class to sort all data mapped to the Reducer process and stores all Values ​​corresponding to the same Key together; in the final Reduce task, all Values ​​with the same Key in the Reducer process are accumulated through the Value iterator to obtain the number of records corresponding to the first keyword of each index; then, a RecordWriter object is generated according to the job parameter information obtained in the OutputFormat, which is responsible for outputting the sorted first keyword information of the index and the counted record information generated by the Reduce task to the local storage where the Reducer process is located in rows;

[0077] S2.5, Output stage: All results are aggregated and output. The output results of each Reducer process are read, sorted from small to large by the first keyword of the index, and then all files are aggregated into a data index analysis file, which is finally output to the storage device.

[0078] S3, build data index file

[0079] Based on the job parameters obtained in step S1, the MapReduce job is started and the job process is controlled, and finally a data index file is generated. The specific process is as follows: Figure 2 As shown, it mainly includes the following steps:

[0080] S3.1, Initial stage: InputFormat provides a way to split the input file (i.e., seismic data). The header of the seismic data is split into multiple Map tasks according to a fixed data block size (4096MB), and the seismic data range to be read for each data block is calculated (i.e., the starting offset of the seismic track and the number of tracks to be read for each block), and then these Map tasks are dynamically allocated to each Mapper process;

[0081] S3.2, Map phase: This phase counts all index keywords and the corresponding number of tracks. Specifically, after each Mapper process receives a Map task, it first completes the initialization of job parameters and the creation of buffers, and then reads the acquired data block information according to the RecordReader provided by InputFormat, reading a track header data each time, and extracting all index keywords in the track header in turn to form a Key, reading the corresponding data track from the storage to form a Value, and storing it in a Key / Value pair;

[0082] S3.3, Map partitioning stage: First, rewrite the comparison function class, and use the rewritten comparison function class to arrange all index keywords in order; then rewrite the partitioning function GetPartition, read the data index analysis file generated in step S2, and use the rewritten partitioning function GetPartition to distribute all index keywords and corresponding channel number information to the Reducer process in order;

[0083] S3.4, Reduce stage: When the Reducer process receives all Map outputs mapped to the Reducer process, it calls the Comparator function to sort all data. Then, it generates a RecordWriter object based on the job parameter information obtained in the OutputFormat. This object is responsible for outputting all the sorted index keyword information and statistical channel information generated by the Reduce task directly to the storage device in the form of an index structure of a multi-level association table to generate an index file.

[0084] It should be added that in the above step S3.3, Map partitioning phase, the process of rewriting the comparison function to arrange all index keywords in order is as follows:

[0085] Since MapReduce can only sort a single Key value by default, and each Key value here contains multiple index keywords, all index keywords need to be parsed out in turn for comparison. First, sort the first index keyword from small to large. If the first index keyword is equal, continue to compare the next index keyword, and so on; if the index keywords of two paths are relative, compare the path position information until all index keywords are arranged in order.

[0086] In the above step S3.3, Map partitioning phase, in order to achieve global sorting between the files generated by the subsequent Reducer processes, it is necessary to rewrite the partitioning function GetPartition, and then use the rewritten partitioning function to perform partitioning:

[0087] Get the first keyword Key1 of the index from Key, divide Key1 into numPartitions groups, and the number of groups is the same as the number of Reducer processes;

[0088] Read the data index analysis file generated in step S2, accumulate the number of tracks of the first keyword of each index to obtain the total number of tracks; divide the total number of tracks by the number of Reducer processes according to the average distribution principle to obtain the number of tracks that each Reducer process needs to process;

[0089] According to the number of records that each Reduce task needs to process, the number of records corresponding to Key1 in the numPartition0 grouping file is accumulated line by line. When it is less than the number of records that the Reduce0 task needs to process, all Values ​​corresponding to Key1 are divided into the Reduce0 task. When it is greater than the number of records that the Reduce0 task needs to process, the corresponding redundant Values ​​are divided into the Reduce1 task. Then, the number of records of the redundant Values ​​is accumulated line by line with the number of records corresponding to Key1 in the numPartition1 grouping file. When it is less than the number of records that the Reduce1 task needs to process, all Values ​​corresponding to Key1 are divided into the Reduce1 task. When it is greater than the number of records that the Reduce1 task needs to process, the corresponding redundant Values ​​are divided into the Reduce2 task. This process is repeated until the Reduce process number corresponding to the value of the first keyword of each index is obtained.

[0090] Finally, the data is output to the local storage medium of each node where the Mapper process is located.

[0091] This embodiment also discloses a massive seismic data index library, which is created by the method described above and includes:

[0092] Index file (file name Index-id.index.partR), which stores the number of seismic data channels processed by the current Reducer process, all index keywords and types;

[0093] The first keyword index file (Index-first-id.index.partR) stores the value of the first keyword of the index, the number of corresponding seismic channels, and the position in other keyword index files;

[0094] Other keyword index file (Index-other-id.index.partR), which stores the values ​​of other index keywords and the corresponding seismic channel numbers;

[0095] Multi-level keyword information file (Index-keyN-id.index.partR), the number of files is the number of index keywords minus one, which stores the value of the current index keyword, the minimum value of the next-level index keyword, the maximum value of the next-level index keyword, the number of next-level index keywords, and the position of the index keyword in the next-level index keyword information file.

[0096] In each output file name, id is automatically assigned by the system when the data is indexed, and each partR corresponds to the current Reduce process number.

[0097] In order to ensure that the interface used by users when querying indexes is not affected by the scale of seismic data, a massive seismic data index library disclosed in this embodiment also includes a general index file (Index-id.index) created on the master node. The content of this file includes the total number of seismic data channels, all index keywords and types, a flag for creating an index using the above method, and the number of Reducer processes.

[0098] It should be noted that the difference between the index structure created in the above embodiment and the index structure created using the original GeoEast method is that the index files of each level in the index structure created by the original GeoEast method are stored in a separate file, while the index files of each level in the index structure created in the above embodiment are split into ReduceNum parts according to the number of Reduce processes.

[0099] This embodiment also discloses a method for querying massive seismic data, which is applicable to the massive seismic data index library as described above. The method is specifically as follows:

[0100] First, the number of Reduce processes is obtained by reading the total index file, and then the corresponding ReduceNum index files are searched according to the index keyword information of each level to be queried.

[0101] According to the index information to be searched, two-level index files (first keyword index file, other keyword index files) or multi-level keyword information files are selected to query related keywords. The query principle is the same.

[0102] When it is necessary to query the earthquake data in the index keyword range of Key1 to KeyN, the first keyword index file and other keyword index files are searched. The specific process is as follows: Figure 3 As shown, it mainly includes the following steps:

[0103] A1. According to the number of Reduce processes obtained, find the index items that meet the Key1 condition by querying the first keyword index file. When querying, start from Index-first-id.index.part0 and compare one by one. If it does not exist in the file, enter Index-first-id.index.part1 in order to search until an index item that meets the Key1 range is found in the first keyword index file of a certain part;

[0104] A2. According to the index table position corresponding to the queried Key1, locate other keyword index files and read the index data corresponding to the position;

[0105] A3. Continue to read other keyword index files in the order of Reduce process numbers until all index items that meet the conditions of Key2 to KeyN are screened out in turn;

[0106] A4. Read the corresponding seismic channel data according to the seismic channel number stored in the index item, thereby completing the search of the index keyword range.

[0107] When you need to query index keyword information, query in the multi-level keyword information file, specifically:

[0108] First, search for qualified keyword information in all ReduceNum first-level keyword information files in turn, and then read from the second-level keyword information file according to the corresponding position in the next-level keyword information file, and so on, until the required index keyword information is obtained.

[0109] It should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art can still modify the technical solutions described in the above embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for constructing an index for massive seismic data, characterized by: This method is implemented based on the MapReduce parallel programming framework and specifically includes the following steps: S1. Analyze job parameters: Set the job parameters required for subsequent MapReduce job execution; S2. Constructing a data index analysis file: Based on the job parameters obtained in step S1, start the MapReduce job and control the job flow, count the values ​​of the first keyword of the index from small to large and the corresponding number of channels, and generate a data index analysis file; S3. Build data index file: Also based on the job parameters obtained in step S1, start the MapReduce job and control the job flow. In the Map stage, count all index keywords and the corresponding number of channels. In the Map partition stage, arrange all index keywords in order. And by reading the data index analysis file generated in step S2, all index keywords and the corresponding number of channels are allocated to the Reducer process in order. In the Reduce stage, each Reducer process sorts the received data and directly outputs the generated index file in the form of a multi-level association table.

2. The index construction method for massive seismic data according to claim 1, characterized in that: Step S1 includes: obtaining seismic data information provided by the user; obtaining keyword information required for creating an index; obtaining the Hosts file; obtaining the number of Reduce processes; and obtaining a temporary file storage path.

3. The index construction method for massive seismic data according to claim 2, characterized in that: Step S2 includes: S2.1, Initial stage: Split the trace header of seismic data into multiple Map tasks and dynamically assign them to each Mapper process; S2.2, Map stage: Each Mapper process reads the data block information obtained in the Map task, reads the header data of each track, extracts the first keyword of the index in the track header to form a Key, reads the corresponding data track from the storage to form a Value, and stores it in the Key / Value pair; S2.3, Map partitioning stage: Call the partitioning function GetPartition to get the Reducer process number corresponding to the Key value, so as to evenly partition the Key / Value pair and assign it to the corresponding Reduce task. In each partition, the comparison function class will be called to sort the Key values ​​from small to large, and finally output them to the local storage medium of the node where each Mapper process is located; S2.4, Reduce stage: After the Reducer process receives all the Map outputs mapped to the Reducer process, it calls the comparison function class to sort all the data mapped to the Reducer process and stores all the Values ​​corresponding to the same Key together; in the final Reduce task, all the Values ​​with the same Key in the Reducer process are accumulated to obtain the number of records corresponding to the first keyword of each index; S2.5, output stage: read the output results of each Reducer process, sort them from small to large according to the first keyword of the index, then aggregate all files into a data index analysis file, and finally output it to the storage device.

4. The index construction method for massive seismic data according to claim 3 is characterized by: In step S2.1, each Map task processes 4096 MB of data, divides the header of the seismic data into blocks, and calculates the range of seismic data that needs to be read for each data block.

5. The method for constructing an index for massive seismic data according to claim 3 or 4, characterized in that: Step S3 includes: S3.1, Initial stage: Split the header of seismic data into multiple Map tasks and dynamically assign them to each Mapper process; S3.2, Map stage: Each Mapper process reads the acquired data block information, reads the header data one by one, and extracts all the index keywords in the header of the track in turn to form a Key, reads the corresponding data track from the storage to form a Value, and stores it in the Key / Value pair; S3.3, Map partitioning stage: first rewrite the comparison function class, and use the rewritten comparison function class to arrange all index keywords in order; then rewrite the partitioning function GetPartition, read the data index analysis file generated in step S2, and use the rewritten partitioning function GetPartition to distribute all index keywords and corresponding channel number information to the Reducer process in order; S3.4, Reduce stage: After the Reducer process receives all Map outputs mapped to the Reducer process, it sorts the data and directly outputs the index file in the form of a multi-level association table on the storage device.

6. The index construction method for massive seismic data according to claim 5, characterized in that: In step S3.3, after rewriting the comparison function class, all index keywords need to be parsed out in sequence for comparison, specifically: First, sort the first keywords of the index from small to large. If the first keywords of the index are equal, continue to compare the next index keyword, and so on. If the index keywords of two tracks are relative, compare the track position information until all index keywords are arranged in order.

7. The method for constructing an index for massive seismic data according to claim 6, characterized in that: In step S3.3, the specific method of partitioning using the rewritten partitioning function GetPartition is: Get the first keyword Key1 of the index from Key, divide Key1 into numPartitions groups, and the number of groups is the same as the number of Reducer processes; Read the data index analysis file generated in step S2, accumulate the number of tracks of the first keyword of each index to obtain the total number of tracks; divide the total number of tracks by the number of Reducer processes according to the average distribution principle to obtain the number of tracks that each Reducer process needs to process; According to the number of records that each Reduce task needs to process, the number of records corresponding to Key1 in the numPartition0 grouping file is accumulated line by line. When it is less than the number of records that the Reduce0 task needs to process, all Values ​​corresponding to Key1 are divided into the Reduce0 task. When it is greater than the number of records that the Reduce0 task needs to process, the corresponding redundant Values ​​are divided into the Reduce1 task. Then, the number of records of the redundant Values ​​is accumulated line by line with the number of records corresponding to Key1 in the numPartition1 grouping file. When it is less than the number of records that the Reduce1 task needs to process, all Values ​​corresponding to Key1 are divided into the Reduce1 task. When it is greater than the number of records that the Reduce1 task needs to process, the corresponding redundant Values ​​are divided into the Reduce2 task. This process is repeated until the Reduce process number corresponding to the value of the first keyword of each index is obtained. Finally, the data is output to the local storage medium of each node where the Mapper process is located.

8. A massive seismic data index library, characterized by: Created by the index construction method of massive seismic data as claimed in claim 1, comprising: Index file, which stores the number of seismic data channels processed by the current Reducer process, all index keywords and types; The first keyword index file stores the value of the first keyword, the number of corresponding seismic channels, and the position in other keyword index files; Other keyword index files, which store the values ​​of other index keywords and the corresponding seismic channel numbers; A multi-level keyword information file, the number of files is the number of index keywords minus one, which stores the value of the current index keyword, the minimum value of the next-level index keyword, the maximum value of the next-level index keyword, the number of next-level index keywords, and the position of the index keyword in the next-level index keyword information file.

9. The massive seismic data index library according to claim 8, characterized in that: It also includes a total index file, which stores the total number of seismic data channels, all index keywords and types, a flag for creating an index library using the index construction method for massive seismic data as described in claim 1, and the number of Reducer processes.

10. A method for querying massive seismic data, characterized by: Based on the massive seismic data index library, the number of Reduce processes is first obtained by reading the total index file, and then the corresponding ReduceNum index files are searched according to the index keyword information of each level to be queried; The massive seismic data index library is created by the massive seismic data index construction method as described in claim 1.

11. The method for querying massive seismic data according to claim 10, characterized in that: When it is necessary to query the seismic data in the index keyword range of Key1 to KeyN, the query is performed in the first keyword index file and other keyword index files, including the following steps: A1. According to the obtained number of Reduce processes, find the index items that meet the Key1 condition by querying the first keyword index file; A2. According to the index table position corresponding to the queried Key1, locate other keyword index files and read the index data corresponding to the position; A3. Continue to read other keyword index files in the order of Reduce process numbers until all index items that meet the conditions of Key2 to KeyN are screened out in turn; A4. Read the corresponding seismic channel data according to the seismic channel number stored in the index item.

12. The method for querying massive seismic data according to claim 10, characterized in that: When you need to query index keyword information, query in the multi-level keyword information file, specifically: First, search for qualified keyword information in all ReduceNum first-level keyword information files in turn, and then read from the second-level keyword information file according to the corresponding position in the next-level keyword information file, and so on, until the required index keyword information is obtained.

Images