Mesh division indexing method, device and equipment based on seismic data and storage medium

Abstract

The invention belongs to the technical field of computers and big data processing, and discloses a seismic data-based grid division indexing method, which comprises index construction and index query, and the index construction comprises hotspot keyword determination, grid sampling, grid sorting and index structure construction; the index query comprises determining a non-empty grid address and scanning to obtain an index result; the invention further discloses a seismic data-based grid division indexing device, equipment and a storage medium. The method is suitable for constructing an index structure based on grid division for mass seismic trace header data, and can realize quick construction, quick search and quick reading of the seismic trace header data index.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer and big data processing, and relates to an index method, device and equipment based on seismic data grid division and a storage medium. BACKGROUND

[0002] Seismic data processing is one of the important technologies in the oil exploration industry. Seismic data obtained in the field is processed and calculated through specific processing algorithms to obtain an image of the underground geological structure, which is used to guide subsequent drilling and oil extraction. With the gradual advancement of seismic exploration equipment and instruments, and the progress of seismic exploration technology, the amount of seismic data collected has grown rapidly over the past decade. The amount of raw seismic data collected in the field has grown by more than 200 times. The amount of data collected in a single seismic exploration has reached the PB level, with more than 10 billion seismic traces.

[0003] The object of a seismic application program is usually such a huge seismic data body, which is logically similar to a data table in a relational database and is organized in row order. Each row record is called a seismic trace. A seismic trace consists of two parts, a trace header and a trace body. The trace header stores attribute information related to the seismic trace, including shotpoint coordinates, geophone coordinates, sample point number, shot number, trace number, and other information. Each attribute is called a trace header key. The trace body is a floating-point array, and each floating-point number is called a sample point. Since the seismic data body is a high-dimensional structured data, each seismic trace has hundreds of attribute information stored in different trace header keys.

[0004] Because the seismic data body is huge, it is impossible to rely on sequential access to each data to select the required data in the tens of billions to hundreds of billions of data entries. Instead, an index needs to be built on the data set to meet the access requirements. SUMMARY

[0005] The purpose of the present application is to provide an index method based on seismic data grid division to improve the efficiency of index construction and query.

[0006] Another purpose of the present application is to provide an index device, equipment and storage medium based on seismic data grid division.

[0007] To achieve the above purposes, the technical solutions adopted by the present application are as follows:

[0008] An index method based on seismic data grid division, including index construction and index query.

[0009] The index construction includes the following steps:

[0010] A1, determining hot keywords, counting the query statements of users and grouping the query statements into a query set, determining hot query statements in the query set according to query frequency, and taking the keywords involved in the hot query statements as hot keywords;

[0011] A2, grid sampling, sampling all seismic header data sets based on the number of Reduce nodes in the server cluster, determining the number of data blocks after sampling of the seismic header data sets, dividing each data block into parts ≥2, randomly selecting a seismic header data in each part for reading, obtaining the hot keywords in the seismic header data and the offset of the hot keywords in the seismic header data, and obtaining rows columns of multi-dimensional seismic header data from the read seismic header data according to the number of hot keywords;

[0012] dividing the seismic header data in the first column into parts, wherein 1≤ ≤ -1, determining boundary values as boundaries, and dividing the data in the first -1 columns according to the quantile points of the cumulative distribution function, and taking the first column as a sorting column for internal sorting of the grid; generating a grid division file according to the quantile points of the cumulative distribution function of the data in the first -1 columns;

[0013] dividing each row of the rows columns of multi-dimensional seismic header data into a grid, taking the first column as a sorting column for internal sorting of the grid, obtaining the number of rows -1 columns of multi-dimensional grids, mapping the number of rows -1 columns of multi-dimensional grids to row-column grid numbers , and sorting the grid numbers , and generating a sorting partition file according to the quantile points of the cumulative distribution function;

[0014] A3, grid sorting, generating rows columns of multi-dimensional seismic header data from the seismic header data set and the number of hot keywords;

[0015] row Each row of the multi-dimensional seismic header data is mapped to a 1-dimensional grid number, each row of data is mapped to a 1-dimensional grid number according to the sorting partition file obtained in step A2 to determine the Reduce node number of the row data, and row The multi-dimensional seismic header data is sorted on the Reduce node to generate a sorted data file.

[0016] Record The number of each grid in the 1-dimensional grid number, the position of each grid in the sorted data file, and the number of seismic header data in each grid, constitute a grid information file only recording non-empty grid information.

[0017] A4, index structure construction, read out the grid information file generated by each Reduce node, and respectively construct the corresponding roaring bitmap structure, and finally summarize to a Reduce node to construct the final roaring bitmap file.

[0018] The index query includes the following steps:

[0019] B1, determine the non-empty grid address, change the query condition into the form of a 1-dimensional grid, obtain the grid intersected by the seismic header data set and the query condition, compare the grid intersected by the seismic header data set and the query condition with the roaring bitmap file, obtain the non-empty grid involved in the query, and obtain the position of the non-empty grid in the sorted data file by querying the grid information file.

[0020] B2, scanning, after obtaining the position of the non-empty grid in the sorted data file, read all the data in the grid in the sorted data file, and further compare each piece of data with the query condition to obtain the index result.

[0021] As a limitation, in step A2, the first column of data is divided after determining the boundary value according to the quantile of the cumulative distribution function, and each row of column data is divided into a grid , is defined as follows:

[0022]

[0023] wherein, is the first The data in the column, For the first The data in the column is sorted from smallest to largest according to the cumulative distribution function. quantiles, For the first The data in the column is sorted from smallest to largest according to the cumulative distribution function. 1 quantile.

[0024] As a further limitation, in step A2, OK -1 column of multidimensional grid numbering mapping is Grid numbering in one row and one column hour, OK -1 The numbering of the multidimensional grid in column 1 maps each row to a column of grid numbers. The mapping rules are as follows:

[0025] .

[0026] As a further limitation, in step A2, the grid numbering... The sorting method is as follows:

[0027] according to The values ​​are sorted in ascending order.

[0028] As a second limitation, in step A3, OK The sorting rules for the seismic trace head data in Ledov on the Reduce node are as follows:

[0029] If earthquake track data Earthquake track data Seismic trace data are located in different grids. Earthquake track data Data from the earthquake trackhead Grid number ( (less than earthquake head data) Grid number ( It is valid under the following conditions;

[0030] If earthquake track data Earthquake track data Within the same grid, seismic trace head data Earthquake track data Is It is valid under the following conditions, among which Earthquake track head data The Column data, Earthquake track head data The Column data.

[0031] The present invention also provides an indexing device based on seismic data grid partitioning, including an index building module and an index query module;

[0032] The index building module includes:

[0033] The hot keyword identification module is used to count user query statements and form query sets. Hot query statements are identified in the query sets according to their query frequency, and the keywords involved in the hot query statements are identified as hot keywords.

[0034] The grid sampling module is used to sample the entire seismic trace head dataset based on the number of Reduce nodes in the server cluster, determine the number of data blocks after sampling, and divide each data block into sections using the overall sampling method. share, ≥2, randomly select one seismic trace head data point from each sample for reading, obtain the hotspot keywords and their offsets within the seismic trace head data, and obtain the results based on the read seismic trace head data. OK Ledovii's seismic track data, The number of trending keywords;

[0035] The first The seismic trace head data of the column is divided into Parts, of which 1≤ ≤ -1, confirm Using these values ​​as boundaries, the preceding... The data in column -1 is divided after determining the boundary values ​​based on the quantiles of the cumulative distribution function. Columns are used as sorting columns within the grid; based on the previous... The quantiles of the cumulative distribution function of the data in column -1 are used to generate a grid partitioning file;

[0036] Will OK Each row of seismic trace head data in the LEDOV dataset is divided into a grid, and the first row is the first row of seismic trace head data. Columns are used as sorting columns for internal grid sorting, resulting in... OK -1 column of multidimensional grid numbering, will OK -1 column of multidimensional grid numbering mapping is Grid numbering in one row and one column and number the grid. Sort the data and generate a sorted partition file based on the quantiles of the cumulative distribution function;

[0037] The grid sorting module is used to generate [a sorting algorithm] based on the number of seismic trace head datasets and hotspot keywords. OK Ledovii's seismic track data, This represents the number of seismic head data points in the seismic head dataset.

[0038] OK Each row of seismic trace head data in the multidimensional seismic trace head data is mapped according to the grid partitioning file obtained in step A2. -1 dimension grid numbering, each row of data mapped Based on the -1 dimension grid numbering and the sorted partition file obtained in step A2, determine the Reduce node number for that row of data. OK The seismic trace head data of Liedodimensional are sorted on the Reduce node to generate sorted data files;

[0039] Record The grid information file, which records only non-empty grid information, consists of the grid number of each grid in the -1 dimension, the position of each grid in the sorted data file, and the number of seismic trace head data in each grid.

[0040] The index structure construction module is used to read the grid information file generated by each Reduce node, construct the corresponding roaring bitmap structure, and finally aggregate them into a Reduce node to construct the final roaring bitmap file.

[0041] The index query module includes:

[0042] The non-empty grid address determination module is used to convert the query conditions into a one-dimensional grid form, obtain the grids where the seismic trace dataset intersects with the query conditions, compare the grids where the seismic trace dataset intersects with the query conditions with the ridingbitmap file, obtain the non-empty grids involved in the query, and obtain the position of the non-empty grids in the sorted data file by querying the grid information file.

[0043] The scanning module is used to obtain the position of the non-empty grid in the sorted data file, read all the data in that grid in the sorted data file, and further compare each piece of data with the query conditions to obtain the index result.

[0044] The present invention also provides a computer device, the computer device including a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded and executed by the processor using a seismic data grid partitioning indexing method.

[0045] The present invention also provides a storage medium for storing at least one computer program for executing a seismic data grid partitioning indexing method.

[0046] The present invention, by adopting the above-described technical solution, achieves the following technical advancements compared to existing technologies:

[0047] (1) This invention determines hot query statements in the query set and uses the keywords in the hot query statements as hot keywords in the index. No matter how the order or number of hot keywords is changed, the index structure does not need to be rebuilt.

[0048] (2) This invention makes full use of the MapReduce framework to perform distributed computing in parallel to improve the efficiency of index construction, and meets the high scalability during construction so that the index can be built on PB-level seismic data;

[0049] (3) This invention can meet the fast query of earthquake track head dataset, making the query selection strategy more extensive. Different hot keywords can be queried through an index with any rules. There is no need to build repeatedly, reducing the average waiting time of users.

[0050] In summary, this invention is applicable to constructing a grid-based index structure for massive seismic trace head data, and uses a novel index structure to achieve rapid construction, retrieval, and reading of seismic trace head data indexes. Attached Figure Description

[0051] Figure 1 The diagram shown is a flowchart of the method in Embodiment 1 of the present invention;

[0052] Figure 2 The diagram shows the index construction process in Embodiment 1 of the present invention;

[0053] Figure 3 The image shown is an illustration of the two-dimensional mesh division method in Embodiment 1 of the present invention.

[0054] Figure 4 The diagram shown is a schematic of the routing bitmap in Embodiment 1 of the present invention.

[0055] Figure 5 The diagram shown is a query flowchart based on an index structure in Embodiment 1 of the present invention;

[0056] Figure 6 The diagram shown is a flowchart of a grid query based on an index structure in Embodiment 1 of the present invention;

[0057] Figure 7 The diagram shown is a structural block diagram of the index construction module in Embodiment 2 of the present invention;

[0058] Figure 8 The diagram shown is a structural block diagram of the index query module in Embodiment 2 of the present invention;

[0059] Figure 9 The diagram shown is a structural schematic of the computer device in Embodiment 2 of the present invention;

[0060] Figure 10 The diagram shown is a schematic diagram of the structure of the computer storage medium in Embodiment 2 of the present invention. Detailed Implementation

[0061] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Example 1: A seismic data grid partitioning and indexing method

[0063] like Figure 1 As shown, this embodiment is a method for indexing seismic data grid partitioning, including index construction and index query;

[0064] like Figure 2 As shown, index building includes the following steps:

[0065] A1. Identify hot keywords, analyze user query statements and form query sets, identify hot query statements in the query sets according to query frequency, and use the keywords involved in the hot query statements as hot keywords.

[0066] In this step, since the seismic trace data may contain hundreds of keywords, the main access method in the actual access process is to use a few keywords in the seismic trace data as filtering conditions for range queries. First, it is necessary to count the user's query statements and form these query statements into a query set. Select the frequently accessed query statements as its hot query statements, and define the keywords involved in these hot query statements as hot keywords.

[0067] A2. Grid sampling: Based on the number of Reduce nodes in the server cluster, sample all seismic tracehead datasets to determine the number of data blocks after sampling. Divide each data block into n parts (n≥2) using a whole-sample method. Randomly select one seismic tracehead data point from each part for reading, obtain the hot keywords and their offsets within the seismic tracehead data, and obtain the results based on the read seismic tracehead data. OK Ledovii's seismic track data, The number of trending keywords;

[0068] In this step, the index construction of GIndex is based on grid sampling. During sampling, because the data space is very large, consisting of billions to tens of billions of seismic trace head data, the number of data blocks after sampling must be sampled as a whole.

[0069] The first The seismic trace head data of the column is divided into Parts, of which 1≤ ≤ -1, confirm Using these values ​​as boundaries, the preceding... The data in column -1 is divided after determining the boundary values ​​based on the quantiles of the cumulative distribution function. Columns are used as sorting columns within the grid; based on the previous... The quantiles of the cumulative distribution function of the data in column -1 are used to generate a grid partitioning file;

[0070] In this step, the front The data in column -1 is divided according to the boundary values ​​determined by the quantiles of the cumulative distribution function. Then, each row is... Column data Divide into grid )middle, The definition is as follows:

[0071]

[0072] in, For each row, the first The data in the column, For the first The data in the column is sorted from smallest to largest according to the cumulative distribution function. quantiles, For the first The data in the column is sorted from smallest to largest according to the cumulative distribution function. 1 quantile;

[0073] The cumulative distribution function, also known as the distribution function, is the integral of the probability density function and can completely describe a real random variable. The probability distribution of all real random variables; The cumulative distribution function represents Less than or equal to The sum of the probabilities of the occurrence of a value is defined as follows:

[0074]

[0075] The quantiles of the cumulative distribution function represent dividing the probability distribution range of a random variable into several continuous intervals with the same probability.

[0076] Will OK Each row of seismic trace head data in the LEDOV dataset is divided into a grid, and the first row is the first row of seismic trace head data. Columns are used as sorting columns for internal grid sorting, resulting in... OK -1 column of multidimensional grid numbering, will OK -1 column of multidimensional grid numbering mapping is Grid numbering in one row and one column and number the grid. Sort the data and generate a sorted partition file based on the quantiles of the cumulative distribution function;

[0077] In this step, unlike with single-dimensional data, we cannot... The grid coordinates in column -1 are sorted naturally to meet the needs of range lookups. If we intuitively... Sorting by concatenating column data is not conducive to range searches, therefore it is necessary to... OK -1 column of multidimensional grid numbering mapping is Grid numbering in one row and one column Among them, OK -1 column of multidimensional grid numbers, each row mapped to a column of grid numbers. The mapping rules are as follows:

[0078] ;

[0079] As can be seen from the mapping rules, the grid number Only from the front -1 column determines; the first Columns are used as sorting columns to sort the seismic trace head data within the grid; assuming a four-column dataset, the first three columns are divided into... So, for a piece of data that falls within a grid with grid coordinates (1,1,1), its grid number is... ;

[0080] In this step, the grid numbers are... The sorting method is as follows:

[0081] according to The values ​​are sorted in ascending order.

[0082] Specifically, this step involves sampling the obtained seismic trace head dataset based on the number of Reduce nodes in the server cluster, using the MapReduce framework:

[0083] First, in the Map phase of the MapReduce framework, the TraceScanInputFormat class is called to sample data blocks. The number of sampled data blocks is determined based on the number of Reduce nodes in the server cluster and then stored in the InputSplit class.

[0084] After receiving the task, the Map Worker uses the TraceScanRecordReader class to divide each data block into segments using a whole-data-sampling method. Each set contains a randomly selected seismic trace data point, from which the selected data point is extracted. Each hot keyword and its offset in the seismic trace data are stored in a key / value pair;

[0085] Map Worker uses the partitioning function GetPartition(const std::string &key, const std::string &value, int numPartitions) of the Partitioner class to send each key / value pair to a unique Reduce task;

[0086] After receiving all key / value pairs from the Map Worker, the Reduce Worker uses the Compare(const std::string &key1, const std::string &key2) function of the TraceScanComparator class to sample the received data, i.e., the entire data block is sampled. The sample data was obtained and sorted; based on the previous... The quantiles of the cumulative distribution function of the data in column -1 are used to generate a grid partitioning file. Simultaneously, the grid numbers for all sampled data are determined. It also obtains the cumulative distribution function to get the quantiles and generates a sorted partition file;

[0087] Use TraceScanRecordWriter to store the gridded file and sorted partition file into non-volatile memory;

[0088] like Figure 3 The image shows the effect of a two-dimensional grid partitioning method, demonstrating the sorting order of a seismic trace dataset with two attributes. The arrows in the image indicate the direction of data from smallest to largest, and the horizontal axis is used to determine the grid number. The ordinate was chosen as the sorting column. After obtaining the grid number... Afterwards, it is necessary to address the data skew problem. To ensure that the data distributed to each reducer node is as evenly distributed as possible, all data points are sorted during the sampling phase and then sorted based on grid number. The grid number is determined using a cumulative distribution function. The quantile of the cumulative distribution function is used to ensure that the amount of data allocated to each Reduce node is as even as possible during the sorting process of all seismic trace head data, thereby reducing the average time consumption of this stage.

[0089] A3. Grid sorting: Based on the number of seismic trace head datasets and hot keywords, generate... OK Ledovii's seismic track data, This represents the number of seismic head data points in the seismic head dataset.

[0090] OK Each row of seismic trace head data in the multidimensional seismic trace head data is mapped according to the grid partitioning file obtained in step A2. -1 dimension grid numbering, each row of data mapped -1D grid numbering maps to 1D grid numbering By comparing with the sorted partition file obtained in step A2, the Reduce node number of that row of data is determined, and then... OK The seismic trace head data of Liedodimensional are sorted on the Reduce node to generate sorted data files;

[0091] Among them, OK The sorting rules for the seismic trace head data in Ledov on the Reduce node are as follows:

[0092] If earthquake track data Earthquake track data Seismic trace data are located in different grids. Earthquake track data Data from the earthquake trackhead Grid number ( (less than earthquake head data) Grid number ( It is valid under the following conditions;

[0093] If earthquake track data Earthquake track data Within the same grid, seismic trace head data Earthquake track data Is It is valid under the following conditions, among which Earthquake track head data The Column data, Earthquake track head data The Column data;

[0094] Record The grid information file, which records only non-empty grid information, consists of the grid number of each grid in the -1 dimension, the position of each grid in the sorted data file, and the number of seismic trace head data in each grid.

[0095] Specifically, the process of sorting the grid based on the MapReduce framework in this step is as follows:

[0096] During the Map phase, the TraceScanInputFormat class is called to sample data blocks. The number of sampled data blocks is determined based on the number of Reduce nodes in the server cluster and then stored in the InputSplit class.

[0097] After receiving the task, the Map Worker uses the TraceSortRecordReader class to read the seismic head data for each data block and extract the selected data from each seismic head data. Each hot keyword and its offset in the seismic trace data are stored together in a key / value pair;

[0098] The Map Worker reads the sorted partition file using the `GetPartition(const std::string &key, const std::string &value, int numPartitions)` function of the `TraceSortPartitioner` class, mapping each row of data to... Inputting a -1 dimension grid number yields the corresponding Reduce node number, which is then used to send each key / value pair to the corresponding Reduce task.

[0099] After receiving the key / value pairs from the Map Worker, the Reduce Worker uses the Compare(const std::string &key1, const std::string &key2) function of the TraceSortComparator class to compare the received key / value pairs. OK The seismic trace head data of Liedodimensional are sorted on the Reduce node according to the sorting rules described above;

[0100] The TraceSortRecordWriter class is used to write the sorted data file and grid information file, which consist of the sorted data, to non-volatile storage.

[0101] A4. Index structure construction: Read out the grid information file generated by each Reduce node, construct the corresponding roaring bitmap structure, and finally aggregate them into a Reduce node to construct the final roaring bitmap file.

[0102] In this step, the index structure is built upon multiple hot keywords. However, queries may not involve all hot keywords. The fewer hot keywords used in a query, the more grids theoretically relevant to that query might be. This may include a large number of empty grids, meaning these grids do not contain seismic trace head data, and their corresponding information is not stored in the grid information file. Therefore, these query conditions have a very high empty search rate. As the number of hot keywords used during construction increases, the number of non-empty grids also increases, resulting in more grids intersecting with the query conditions, but the number of non-empty grids involved in the query remains small. Based on the above, this paper uses a riding bitmap to quickly filter data. The riding bitmap processing, based on 64-bit grid numbers, is as follows: Figure 4 As shown in the principle of roaring bitmap, all non-empty grids are numbered. After being read into memory, it is divided into high 48 bits and low 16 bits. The high 48 bits represent... Each container stores 16 bits of memory. The non-empty grid number is stored within each container. First, the corresponding container is found using the high 48 bits of the grid number. Then, the low 16 bits are stored in the corresponding container (a new container is created if the grid number is not found during storage). There are two ways to store the low 16 bits: one is to store them using an array (each element is a short). In this case, the container can store 4096 elements because of the grid number. Since the elements are ordered, they will also be ordered after being stored in the array. However, when the number of elements in the array exceeds 4096 (the container size is 8KB), all elements will be converted to be stored using a bitmap. This method allows for less space usage when elements are sparse, and ensures that the space usage remains a fixed 8KB even as the number of elements in the container increases. Whenever a query is received, the non-empty grid number to be searched and its position in the grid information file are obtained by querying the routing bitmap file. Then, the position of the data to be scanned in the sorted data file can be obtained by querying the grid information file.

[0103] In this step, the process of constructing the index structure based on the MapReduce framework is as follows:

[0104] During the Map phase, the TraceScanInputFormat class is called to sample data blocks. The number of sampled data blocks is determined based on the number of Reduce nodes in the server cluster and then stored in the InputSplit class.

[0105] After receiving the task, the Map Worker uses the TraceIndexRecordReader class to read the grid information file for each data block and extract the grid number. Store key / value pairs;

[0106] Map Worker uses TraceIndexCombiners to number non-empty grid cells. Each `roaringbitmap` structure is constructed and converted into a string as a new key / value pair.

[0107] Map Worker uses the partitioning function GetPartition(const std::string &key, const std::string &value, int numPartitions) of the Partitioner class to send each key / value pair to a unique Reduce task;

[0108] After receiving all the key / value pairs from the Map Worker, the Reduce Worker converts the key into a riding bitmap structure and merges multiple riding bitmap structures into a single riding bitmap file.

[0109] Use the TraceIndexRecordWriter class to convert the roaring bitmap file into a binary format file and write it to non-volatile storage;

[0110] A typical multi-dimensional query SQL input consists of filtering conditions composed of ranges of one or more attributes, connected by AND operations. The intersection of these ranges defines a hyperrectangle; for example, in a query condition... In this context, z1 and z2 are hot keywords in two seismic trace data sets, and t, m, e, and f are the actual values. For example, if z1 represents "height" and z2 represents "weight," and the query condition is to find individuals with a height of 170 to 180 cm and a weight of 70 to 80 kg, then t=170, m=180, e=70, and f=80. Figure 5 In and The goal of the query is to accurately locate and process the points within the hyperrectangle formed by the hyperrectangle.

[0111] In this embodiment, the index query includes the following steps:

[0112] B1. Determine the addresses of non-empty grids, change the query conditions to a one-dimensional grid form, obtain the grids where the seismic trace head dataset intersects with the query conditions, compare the grids where the seismic trace head data intersects with the query conditions with the riding bitmap file, obtain the non-empty grids involved in the query, and obtain the position of the non-empty grids in the sorted data file by querying the grid information file.

[0113] In this step, to determine which seismic trace data matches the query criteria, the query criteria first need to be converted into a one-dimensional grid. Taking a two-dimensional query criterion as an example: assuming the query starting point is... The endpoint of the query is First, they need to be converted into and ( and If we can easily find the cells in the entire earthquake trace dataset that intersect with the query conditions, then it's easy to obtain the cells where the query conditions are met. ,like Figure 5 Cells 1-4 in the text.

[0114] In this step, for hot keywords not mentioned in the query conditions but present in the index file (obtained by sampling and sorting hot keywords), the query conditions can be changed to search for the range of that hot keyword as -∞ and +∞ to ensure the accuracy of the results. In other words, if the query condition is based on three hot keywords, the input query statement will be transformed into... This also explains why the more hot keywords are used in the query, the more grids the same query might involve. For dimensions that exist in the query but not in the index file, the query conditions can be changed to ranges from 0 to 0. Non-empty grids can be obtained through the roaring bitmap file. Figure 5 The addresses of cell1, cell3, and cell4 in the grid information file;

[0115] like Figure 6 As shown, the query conditions are {10 ≤ KeyA ≤ 65, 30 ≤ KeyB ≤ 70}, so KeyA is used to determine the mapped grid. By querying the grid partitioning file, we can obtain cell(10) = 0 and cell(65) = 3. This is converted into a one-dimensional grid form, i.e. (10) = 0, (65) = 3, where (10) signifies the start of grid numbering. (65) represents the end of the grid number. Therefore, the list of grids that intersect the seismic track head dataset with the query conditions can be determined as {0, 1, 2, 3}. The grids that intersect the seismic track head dataset with the query conditions need to be compared with the riding bitmap file. Since their container numbers are all 0, we know that the non-empty grids are {0, 2, 3}. After obtaining the non-empty grids, we can query the grid information file to get the position of the non-empty grids in the sorted data file (P1, P2, P3 in the figure). If the query conditions use a sorting sequence (KeyB), it will help reduce the amount of data that needs to be scanned. That is, the amount of data that needs to be scanned is reduced from all the data in the grid (the solid-lined grids Cell0, Cell2, Cell3 in the figure) to only the dashed-lined grids. Here, the binary search method is used to find the data in the grids cell0, cell2, and cell3 that match the query range with KeyB, so that the amount of data that needs to be scanned is reduced from the original 12 to 5.

[0116] B2. After scanning to obtain the position of the non-empty grid in the sorted data file, read all the data in that grid in the sorted data file and further compare each piece of data with the query conditions to obtain the index result;

[0117] In this step, after obtaining the position of the non-empty grid in the sorted data file, it is necessary to read the position of the seismic trace head data in the sorted data file to confirm the physical address of the data. Figure 5 (The box with sequence number 1). If a sorted sequence is involved, it will help reduce the amount of data that needs to be scanned. For example, let's say the box with sequence number 1... The column is used as the sorting column, and the first column in the query... The query range of the column is It only requires by examining the first... Using binary search on a column, find the first value greater than... The position of the value , and the first one greater than The position of the value Therefore, the range of data that needs to be read from the disk for this grid is [ , ]( Figure 5 (See box number 2 in the middle). This will reduce the amount of data read from the disk to some extent. There are two types of non-empty matching grids: the first is a fully matched grid, where the data range of each column in the grid is completely contained within the query range. For this type of grid, its start and end addresses in disk space can be directly obtained, and data can be read in batches without further range comparison. The second is a partially matched grid, where the data in the grid intersects with the query range or is completely contained within the query conditions in at least one column. Therefore, data that meets the conditions in one column may not meet the conditions in another column. Within this type of grid, there may be data that meets the query conditions, or there may be no data that meets the conditions. It is necessary to match the seismic trace head data in the grid item by item.

[0118] Example 2: A seismic data grid partitioning indexing device

[0119] This invention is a seismic data grid partitioning indexing device, which includes an index building module and an index query module.

[0120] like Figure 7 As shown, the index building module includes:

[0121] The hot keyword identification module is used to count user query statements and form query sets. Within the query sets, hot query statements are identified based on their query frequency, and the keywords involved in these hot query statements are designated as hot keywords.

[0122] The grid sampling module is used to sample the entire seismic trace head dataset based on the number of Reduce nodes in the server cluster, determine the number of data blocks after sampling, and divide each data block into sections using the overall sampling method. share, ≥2, randomly select one seismic trace head data point from each sample for reading, obtain the hotspot keywords and their offsets within the seismic trace head data, and obtain the results based on the read seismic trace head data. OK Ledovii's seismic track data, The number of trending keywords;

[0123] The first The seismic trace head data of the column is divided into Parts, of which 1≤ ≤ -1, confirm Using these values ​​as boundaries, the preceding... The data in column -1 is divided after determining the boundary values ​​based on the quantiles of the cumulative distribution function. Columns are used as sorting columns within the grid; based on the previous... The quantiles of the cumulative distribution function of the data in column -1 are used to generate a grid partitioning file;

[0124] Will OK Each row of seismic trace head data in the LEDOV dataset is divided into a grid, and the first row is the first row of seismic trace head data. Columns are used as sorting columns for internal grid sorting, resulting in... OK -1 column of multidimensional grid numbering, will OK -1 column of multidimensional grid numbering mapping is Grid numbering in one row and one column and number the grid. The data is sorted, and a sorted partition file is generated based on the quantiles of the cumulative distribution function.

[0125] The grid sorting module is used to generate [a sorting algorithm] based on the number of seismic trace head datasets and hotspot keywords. OK Ledovii's seismic track data, This represents the number of seismic head data points in the seismic head dataset.

[0126] OK Each row of seismic trace head data in the multidimensional seismic trace head data is mapped according to the grid partitioning file obtained in step A2. -1 dimension grid numbering, each row of data mapped Based on the -1 dimension grid numbering and the sorted partition file obtained in step A2, determine the Reduce node number for that row of data. OK The seismic trace head data of Liedodimensional are sorted on the Reduce node to generate sorted data files;

[0127] Record The grid information file, which records only non-empty grid information, consists of the grid number of each grid in the -1 dimension, the position of each grid in the sorted data file, and the number of seismic trace head data in each grid.

[0128] The index structure construction module is used to read the grid information file generated by each Reduce node, construct the corresponding roaring bitmap structure, and finally aggregate them into a Reduce node to construct the final roaring bitmap file.

[0129] like Figure 8 As shown, the index query module includes:

[0130] The non-empty grid address determination module is used to convert the query conditions into a one-dimensional grid form, obtain the grids that intersect the seismic trace head dataset with the query conditions, compare the grids that intersect the seismic trace head dataset with the query conditions with the ridingbitmap file, obtain the non-empty grids involved in the query, and obtain the position of the non-empty grids in the sorted data file by querying the grid information file.

[0131] The scanning module is used to obtain the position of the non-empty grid in the sorted data file, read all the data in that grid in the sorted data file, and further compare each piece of data with the query conditions to obtain the index result.

[0132] In this embodiment, the seismic data grid partitioning indexing device is used to illustrate the division of the above-mentioned functional modules when performing data processing. In actual applications, the above functions can be assigned to different functional modules as needed.

[0133] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 9 As shown, this embodiment also provides a computer device, including: at least a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded by the processor and executed by the processor according to the seismic data grid partitioning indexing method of Embodiment 1.

[0134] Based on the same inventive concept, such as Figure 10 As shown, this embodiment also provides a computer-readable storage medium for storing at least one computer program for executing a seismic data grid partitioning indexing method.

[0135] It should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art can still modify the technical solutions described in the above embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A seismic data grid partitioning indexing method, characterized in that, This includes index building and index querying; The index construction includes the following steps: A1. Identify hot keywords, analyze user query statements and form query sets, identify hot query statements in the query sets according to query frequency, and use the keywords involved in the hot query statements as hot keywords. A2. Grid sampling: Based on the number of Reduce nodes in the server cluster, the entire seismic trace head dataset is sampled to determine the number of data blocks after sampling. Each data block is then divided into... share, ≥2, randomly select one seismic trace head data point from each sample for reading, obtain the hotspot keywords and their offsets within the seismic trace head data, and obtain the results based on the read seismic trace head data. OK Ledovii's seismic track data, The number of trending keywords; The first The seismic trace head data of the column is divided into Parts, of which 1≤ ≤ -1, confirm Using these values ​​as boundaries, the preceding... The data in column -1 is divided after determining the boundary values ​​based on the quantiles of the cumulative distribution function. Columns are used as sorting columns within the grid; based on the previous... The quantiles of the cumulative distribution function of the data in column -1 are used to generate a grid partitioning file; Will OK Each row of seismic trace head data in the LEDOV dataset is divided into a grid, and the first row is the first row of seismic trace head data. Columns are used as sorting columns for internal grid sorting, resulting in... OK -1 column of multidimensional grid numbering, will OK -1 column of multidimensional grid numbering mapping is Grid numbering in one row and one column and number the grid. Sort the data and generate a sorted partition file based on the quantiles of the cumulative distribution function; A3. Grid sorting: Based on the number of seismic trace head datasets and hot keywords, generate... OK Ledovii's seismic track data, This represents the number of seismic head data points in the seismic head dataset. OK Each row of seismic trace head data in the multidimensional seismic trace head data is mapped according to the grid partitioning file obtained in step A2. -1 dimension grid numbering, each row of data mapped Based on the -1 dimension grid numbering and the sorted partition file obtained in step A2, determine the Reduce node number for that row of data. OK The seismic trace head data of Liedodimensional are sorted on the Reduce node to generate sorted data files; Record The grid information file, which records only non-empty grid information, consists of the grid number of each grid in the -1 dimension, the position of each grid in the sorted data file, and the number of seismic trace head data in each grid. A4. Index structure construction: Read out the grid information file generated by each Reduce node, construct the corresponding roaring bitmap structure, and finally aggregate them into a Reduce node to construct the final roaring bitmap file. The index query includes the following steps: B1. Determine the addresses of non-empty grids, change the query conditions to a one-dimensional grid form, obtain the grids where the seismic trace dataset intersects with the query conditions, compare the grids where the seismic trace dataset intersects with the query conditions with the riding bitmap file, obtain the non-empty grids involved in the query, and obtain the position of the non-empty grids in the sorted data file by querying the grid information file. B2. After scanning to obtain the position of the non-empty grid in the sorted data file, read all the data in that grid in the sorted data file and further compare each piece of data with the query conditions to obtain the index result.

2. The seismic data grid partitioning indexing method according to claim 1, characterized in that, In step A2, the first The data in column -1 is divided according to the boundary values ​​determined by the quantiles of the cumulative distribution function. Then, each row is... Column data Divide into grid )middle, The definition is as follows: in, For each row, the first The data in the column, For the first The data in the column is sorted from smallest to largest according to the cumulative distribution function. quantiles, For the first The data in the column is sorted from smallest to largest according to the cumulative distribution function. 1 quantile.

3. The seismic data grid partitioning indexing method according to claim 2, characterized in that, In step A2, OK -1 column of multidimensional grid numbering mapping is Grid numbering in one row and one column hour, OK -1 The numbering of the multidimensional grid in column 1 maps each row to a column of grid numbers. The mapping rules are as follows: 。 4. The seismic data grid partitioning indexing method according to claim 3, characterized in that, In step A2, the grid numbers are... The sorting method is as follows: according to The values ​​are sorted in ascending order.

5. The seismic data grid partitioning indexing method according to claim 1, characterized in that, In step A3, OK The sorting rules for the seismic trace head data in Ledov on the Reduce node are as follows: If earthquake track data Earthquake track data Seismic trace data are located in different grids. Earthquake track data Data from the earthquake trackhead Grid number ( (less than earthquake head data) Grid number ( It is valid under the following conditions; If earthquake track data Earthquake track data Within the same grid, seismic trace head data Earthquake track data Is It is valid under the following conditions, among which Earthquake track head data The Column data, Earthquake track head data The Column data.

6. A seismic data grid partitioning indexing device, characterized in that, Includes an index building module and an index query module; The index building module includes: The hot keyword identification module is used to count user query statements and form query sets. Hot query statements are identified in the query sets according to their query frequency, and the keywords involved in the hot query statements are identified as hot keywords. The grid sampling module is used to sample the entire seismic trace head dataset based on the number of Reduce nodes in the server cluster, determine the number of data blocks after sampling, and divide each data block into sections using the overall sampling method. share, ≥2, randomly select one seismic trace head data point from each sample for reading, obtain the hotspot keywords and their offsets within the seismic trace head data, and obtain the results based on the read seismic trace head data. OK Ledovii's seismic track data, The number of trending keywords; The first The seismic trace head data of the column is divided into Parts, of which 1≤ ≤ -1, confirm Using these values ​​as boundaries, the preceding... The data in column -1 is divided after determining the boundary values ​​based on the quantiles of the cumulative distribution function. Columns are used as sorting columns within the grid; based on the previous... The quantiles of the cumulative distribution function of the data in column -1 are used to generate a grid partitioning file; Will OK Each row of seismic trace head data in the LEDOV dataset is divided into a grid, and the first row is the first row of seismic trace head data. Columns are used as sorting columns for internal grid sorting, resulting in... OK -1 column of multidimensional grid numbering, will OK -1 column of multidimensional grid numbering mapping is Grid numbering in one row and one column and number the grid. Sort the data and generate a sorted partition file based on the quantiles of the cumulative distribution function; The grid sorting module is used to generate [a sorting algorithm] based on the number of seismic trace head datasets and hotspot keywords. OK Ledovii's seismic track data, This represents the number of seismic head data points in the seismic head dataset. OK Each row of seismic trace head data in the multidimensional seismic trace head data is mapped according to the grid partitioning file obtained in step A2. -1 dimension grid numbering, each row of data mapped Based on the -1 dimension grid numbering and the sorted partition file obtained in step A2, determine the Reduce node number for that row of data. OK The seismic trace head data of Liedodimensional are sorted on the Reduce node to generate sorted data files; Record The grid information file, which records only non-empty grid information, consists of the grid number of each grid in the -1 dimension, the position of each grid in the sorted data file, and the number of seismic trace head data in each grid. The index structure construction module is used to read the grid information file generated by each Reduce node, construct the corresponding roaring bitmap structure, and finally aggregate them into a Reduce node to construct the final roaring bitmap file. The index query module includes: The non-empty grid address determination module is used to convert the query conditions into a one-dimensional grid form, obtain the grids where the seismic trace dataset intersects with the query conditions, compare the grids where the seismic trace dataset intersects with the query conditions with the riding bitmap file to obtain the non-empty grids involved in the query, and obtain the position of the non-empty grids in the sorted data file by querying the grid information file. The scanning module is used to obtain the position of the non-empty grid in the sorted data file, read all the data in that grid in the sorted data file, and further compare each piece of data with the query conditions to obtain the index result.

7. A computer device, characterized in that, The computer device includes a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded by the processor and executed as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is used to store at least one computer program, which is used to execute the seismic data grid partitioning indexing method according to any one of claims 1 to 5.

Images