Seismic data query method, device, electronic equipment and medium

Abstract

The present invention relates to the field of seismic exploration, and in particular to a seismic data query method, device, electronic device, and medium. The method comprises: performing data sampling and distributed data learning on seismic data based on a MapReduce framework and a machine learning model to generate a node allocation model; performing distributed segmented sorting on the seismic data based on the MapReduce framework and the node allocation model to generate an ordered index file; training a learning index including an underlying model and a non-underlying model based on the MapReduce framework and the ordered index file, and combining the learning index with the node allocation model to construct a learning index structure; obtaining a keyword to be queried and inputting it into the learning index structure to obtain a query result. In the solution of the present invention, the learning index can be reused to a certain extent, effectively reducing the index construction time and improving the efficiency of index construction.

Description

Technical Field

[0001] The present invention relates to the field of seismic exploration, and in particular to a seismic data query method, device, electronic equipment and medium. Background Art

[0002] Seismic data processing is a crucial technology in the oil exploration industry. It processes and calculates seismic data collected in the field using specific processing algorithms to generate images of subsurface geological structures, which are used to guide subsequent drilling and oil extraction operations. With the continuous application of new exploration and high-precision acquisition technologies in oil exploration, the volume of raw seismic data collected from the field has grown rapidly. Currently, the size of a single data volume exceeds petabytes, with the number of seismic traces reaching hundreds of billions. Seismic applications typically process such massive seismic data volumes. Logically, seismic data volumes are similar to tables in relational databases, organized in rows, with each row being a seismic trace. A seismic trace consists of two parts: a trace header and a trace body. The trace header stores attribute information related to the seismic trace, including shot point coordinates, receiver coordinates, number of sampling points, shot number, and trace number. Each attribute is called a trace header key. The trace body is an array of floating-point numbers, with each floating-point number being a sampling point. Since seismic data is high-dimensional structured data, each seismic trace has hundreds of attribute information stored in different trace header keywords.

[0003] However, when accessing seismic data volumes, many interactive seismic applications are typically only interested in a subset of the data sets within that volume. Therefore, a large number of seismic data accesses specify the value ranges of some attributes to filter out specific data sets, and may also specify the order of certain attributes to determine how the query results are sorted. Since multi-dimensional range queries are the most common data query pattern in seismic applications, their query speed is crucial to the performance and user experience of seismic applications, especially interactive applications. Efficient index queries are the basis for ensuring query efficiency and reducing seismic data query latency. As a balanced search tree designed for disks or other direct-access auxiliary storage devices, the B+ tree index can effectively reduce the number of disk I / O operations during queries. And because the B+ tree can support fast range scans along leaf nodes, it has relatively good range query performance.

[0004] However, the traditional B+ tree construction method is to insert each record into an empty tree, and the process involves many complex operations, such as node splitting and rotation. For hundreds of billions of unordered data, the overhead of this traditional construction method is enormous, and it is difficult to build in parallel on multiple machines. The speedup ratio in a multi-threaded environment is also relatively limited. At the same time, if every piece of data is stored in the B+ tree, the final tree size will also be huge, and the search will still face the problem of low efficiency. In addition, the B+ tree can only be used to search for a single keyword. When searching in a multi-keyword range, it is necessary to first query all data that meets the first keyword conditions, and then filter subsequent keywords, which will greatly reduce the search efficiency. Therefore, for massive seismic data, it is necessary to design a distributed index construction method that is different from the traditional B+ tree structure to improve index construction efficiency, speedup ratio, scalability and query efficiency. Summary of the Invention

[0005] In view of this, it is necessary to provide a seismic data query method, device, electronic device and medium to address the above technical problems.

[0006] According to a first aspect of the present invention, a seismic data query method is provided, the method comprising:

[0007] Based on the MapReduce framework and machine learning model, data sampling and data distributed learning are performed on seismic data to generate a node allocation model;

[0008] Based on the MapReduce framework and the node allocation model, distributed segment sorting of seismic data is performed to generate an ordered index file;

[0009] Based on the MapReduce framework, the ordered index file trains a learning index including an underlying model and a non-underlying model, and combines the learning index with the node allocation model to construct a learning index structure;

[0010] A keyword to be queried is obtained and input into the learning index structure to obtain a query result.

[0011] In some embodiments, the step of performing data sampling and data distributed learning on seismic data based on the MapReduce framework and the machine learning model to generate a node allocation model includes:

[0012] Set the sampling size and the number of reducers to 1, and use the MapReduce framework to evenly distribute the sampling size to each node;

[0013] In the Map phase, each node reads seismic traces at intervals according to the sampling scale, extracts the keyword information containing the selected keyword value in each seismic trace, and ensures that the number of seismic traces sampled and read meets the sampling scale;

[0014] In the Reduce phase, the read keyword information is integrated, and the sampling results are used as the overall data scan results to generate the allocation table of the subsequent sorting data range of each node;

[0015] Based on the allocation table, the keyword value is used as input and the node number specified for the value is used as output. After normalization, it is passed into the machine learning model according to the channel ratio of the keyword value for training. The cumulative distribution function of the keyword is fitted and saved to generate the node allocation model.

[0016] In some embodiments, the step of performing distributed segment sorting on the seismic data based on the MapReduce framework and the node allocation model to generate an ordered index file includes:

[0017] In the Map phase, the header data is evenly divided into multiple segments. Each Map task is responsible for reading a segment of data. These Map tasks are then evenly distributed to the Map Workers of each node in the cluster.

[0018] After receiving the Map task, the Map Worker starts processing the corresponding trace header data segment, reading all trace headers, sequentially extracting the selected keyword data and the trace number in each trace as index items, and storing them using key / value pairs;

[0019] The Map Worker obtains the node number corresponding to each index item by calling the node allocation model, and then sends each key / value to the corresponding Reduce Worker for sorting;

[0020] After the Reduce Worker receives all the key / value pairs sent by the Map Worker, it sorts the index data stored in the key / value pairs according to the keyword order, and finally stores the sorted index data in the file corresponding to the Reduce Worker, and writes the statistics of each keyword information into the corresponding keyword information file.

[0021] In some embodiments, the step of training a learned index including an underlying model and a non-underlying model based on the MapReduce framework and the ordered index file, and combining the learned index with the node allocation model to construct a learned index structure includes:

[0022] In the Map phase, the MakeMLIndexInputFormat class is called to divide the data into blocks. The number of data blocks is determined based on the number of index files and stored in the InputSplit class.

[0023] After receiving the task, the Map Worker uses the MakeMLIndexRecordReader class to read the index data for each data block. Every time a fixed number of index items are read, they are counted as a training set for an underlying model in the learning index. The MakeIndexLeafModel function is used to generate an underlying model in the learning index for training and storage.

[0024] After all leaf nodes are created, use the MakeIndexNodeModel function to complete the generation, training, and saving of the non-bottom-level model of the learning index from bottom to top;

[0025] The node allocation model is used as the root node model of the learning index to obtain a learning index structure, wherein the learning index structure starts from the root node model when querying, each layer model selects the next layer model, and the last layer model outputs the approximate position of the query value in the overall index data.

[0026] In some embodiments, the step of obtaining a keyword to be queried and inputting it into the learning index structure to obtain a query result includes:

[0027] Obtain the keyword to be queried input by the user, and input the keyword value into the root node model of the learning index structure, obtain the lower model number where the keyword is located, and so on downward to the bottom model;

[0028] The underlying model is used to predict the offset of the keyword in the index data file, and the index file is read based on the predicted offset.

[0029] If the index item value read is inaccurate, you can determine whether to search forward or backward item by item by comparing the current item with the search value until the correct value is matched.

[0030] In some embodiments, the method further comprises:

[0031] After the query, the query conditions and the actual offset are passed into the model again as training data for training to update the learning index structure.

[0032] In some embodiments, the machine learning model is a multi-layer perceptron model.

[0033] According to a second aspect of the present invention, a seismic data query device is provided, the device comprising:

[0034] A sampling learning module is configured to perform data sampling and data distributed learning on seismic data based on the MapReduce framework and the machine learning model to generate a node allocation model;

[0035] A sorting module configured to perform distributed segment sorting on the seismic data based on the MapReduce framework and the node allocation model to generate an ordered index file;

[0036] A construction module is configured to train a learning index including an underlying model and a non-underlying model based on a MapReduce framework and the ordered index file, and combine the learning index with the node allocation model to construct a learning index structure;

[0037] The query module is configured to obtain keywords to be queried and input them into the learning index structure to obtain query results.

[0038] According to a third aspect of the present invention, there is further provided an electronic device, comprising:

[0039] at least one processor; and

[0040] The memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned seismic data query method when executing the program.

[0041] According to a fourth aspect of the present invention, a computer-readable storage medium is further provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the aforementioned seismic data query method is executed.

[0042] The above-mentioned seismic data query method has the following beneficial technical effects: the learning index structure can learn and store the distribution pattern of data with a small amount of space, significantly reducing the storage space of the index structure. It treats the index structure as a black box and regards the index structure as a model simply from the perspective of input and output data. It predicts its position in the ordered data body by fitting the cumulative distribution function of the input data, and uses the machine learning model to learn and predict the distribution pattern of seismic data, thereby realizing the indexing function. The learning index can be reused to a certain extent, effectively reducing the index construction time and improving the efficiency of index construction.

[0043] In addition, the present invention also provides a seismic data query device, an electronic device and a computer-readable storage medium, which can also achieve the above-mentioned technical effects and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A flowchart of a seismic data query method provided by one embodiment of the present invention;

[0046] Figure 2 A schematic diagram of the overall structure of a large-scale seismic data indexing system for implementing the method of the present invention is provided as an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the implementation principle of the learning indexing function designed by the present invention;

[0048] Figure 4 This is a schematic diagram of the multi-layer perceptron model used in the learning index designed by the present invention;

[0049] Figure 5 Schematic diagram of the relationship between the first keyword value and the offset in PB-level seismic data using the indexing function of the method of the present invention;

[0050] Figure 6 This is a schematic diagram of the data sampling process based on the MapReduce framework and machine learning model in the method of the present invention;

[0051] Figure 7 This is a schematic diagram of the distributed segment sorting process based on the MapReduce framework in the method of the present invention;

[0052] Figure 8 This is a flow chart of a learning index construction process based on the MapReduce framework in the method of the present invention;

[0053] Figure 9 It is a schematic diagram of the learning index structure and query process in the method of the present invention;

[0054] Figure 10 A schematic structural diagram of a seismic data query device provided by another embodiment of the present invention;

[0055] Figure 11 is a diagram showing the internal structure of an electronic device according to another embodiment of the present invention;

[0056] Figure 12 This is a structural diagram of a computer-readable storage medium in another embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0058] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. Subsequent embodiments will not explain this one by one.

[0059] In one embodiment, please refer to Figure 1 As shown, the present invention provides a seismic data query method 100. Specifically, the method includes the following steps:

[0060] Step 101: performing data sampling and data distributed learning on seismic data based on a MapReduce framework and a machine learning model to generate a node allocation model;

[0061] Step 102: sorting the seismic data in a distributed manner based on the MapReduce framework and the node allocation model to generate an ordered index file;

[0062] Step 103: training a learning index including an underlying model and a non-underlying model based on a MapReduce framework and the ordered index file, and combining the learning index with the node allocation model to construct a learning index structure;

[0063] Step 104: Obtain keywords to be searched and input them into the learning index structure to obtain search results.

[0064] A seismic data query method of this embodiment has the following beneficial technical effects: the learning index structure can learn and store the distribution pattern of data with a small amount of space, significantly reducing the storage space of the index structure. It treats the index structure as a black box and regards the index structure as a model simply from the perspective of input and output data. It predicts its position in the ordered data body by fitting the cumulative distribution function of the input data, and uses a machine learning model to learn and predict the distribution pattern of seismic data, thereby realizing the indexing function. The learned index can be reused to a certain extent, effectively reducing the index construction time and improving the efficiency of index construction.

[0065] In some embodiments, the aforementioned step 101, performing data sampling and data distributed learning on seismic data based on the MapReduce framework and the machine learning model to generate a node allocation model, includes:

[0066] Set the sampling size and the number of reducers to 1, and use the MapReduce framework to evenly distribute the sampling size to each node;

[0067] In the Map phase, each node reads seismic traces at intervals according to the sampling scale, extracts the keyword information containing the selected keyword value in each seismic trace, and ensures that the number of seismic traces sampled and read meets the sampling scale;

[0068] In the Reduce phase, the read keyword information is integrated, and the sampling results are used as the overall data scan results to generate the allocation table of the subsequent sorting data range of each node;

[0069] Based on the allocation table, the keyword value is used as input and the node number specified for the value is used as output. After normalization, it is passed into the machine learning model according to the channel ratio of the keyword value for training. The cumulative distribution function of the keyword is fitted and saved to generate the node allocation model.

[0070] In some embodiments, the aforementioned step 102, performing distributed segment sorting on the seismic data based on the MapReduce framework and the node allocation model to generate an ordered index file, includes:

[0071] In the Map phase, the header data is evenly divided into multiple segments. Each Map task is responsible for reading a segment of data. These Map tasks are then evenly distributed to the Map Workers of each node in the cluster.

[0072] After receiving the Map task, the Map Worker starts processing the corresponding trace header data segment, reading all trace headers, sequentially extracting the selected keyword data and the trace number in each trace as index items, and storing them using key / value pairs;

[0073] The Map Worker obtains the node number corresponding to each index item by calling the node allocation model, and then sends each key / value to the corresponding Reduce Worker for sorting;

[0074] After the Reduce Worker receives all the key / value pairs sent by the Map Worker, it sorts the index data stored in the key / value pairs according to the keyword order, and finally stores the sorted index data in the file corresponding to the Reduce Worker, and writes the statistics of each keyword information into the corresponding keyword information file.

[0075] In some embodiments, the aforementioned step 103, training a learned index including an underlying model and a non-underlying model based on the MapReduce framework and the ordered index file, and combining the learned index with the node allocation model to construct a learned index structure, includes:

[0076] In the Map phase, the MakeMLIndexInputFormat class is called to divide the data into blocks. The number of data blocks is determined based on the number of index files and stored in the InputSplit class.

[0077] After receiving the task, the Map Worker uses the MakeMLIndexRecordReader class to read the index data for each data block. Every time a fixed number of index items are read, they are counted as a training set for an underlying model in the learning index. The MakeIndexLeafModel function is used to generate an underlying model in the learning index for training and storage.

[0078] After all leaf nodes are created, use the MakeIndexNodeModel function to complete the generation, training, and saving of the non-bottom-level model of the learning index from bottom to top;

[0079] The node allocation model is used as the root node model of the learning index to obtain a learning index structure, wherein the learning index structure starts from the root node model when querying, each layer model selects the next layer model, and the last layer model outputs the approximate position of the query value in the overall index data.

[0080] In some embodiments, the aforementioned step 104, obtaining a keyword to be searched and inputting it into the learning index structure to obtain a search result, includes:

[0081] Obtain the keyword to be queried input by the user, and input the keyword value into the root node model of the learning index structure, obtain the lower model number where the keyword is located, and so on downward to the bottom model;

[0082] The underlying model is used to predict the offset of the keyword in the index data file, and the index file is read based on the predicted offset.

[0083] If the index item value read is inaccurate, you can determine whether to search forward or backward item by item by comparing the current item with the search value until the correct value is matched.

[0084] In some embodiments, the method further comprises:

[0085] After the query, the query conditions and the actual offset are passed into the model again as training data for training to update the learning index structure.

[0086] In some embodiments, the machine learning model is a multi-layer perceptron model.

[0087] In another embodiment, in order to facilitate understanding of the solution of the present invention, the seismic data query method of the present invention is used on a high-performance cluster as an example. Figure 2 As shown, Figure 2 The large-scale seismic data indexing system is generally divided into two parts: index construction and index query. Index construction is further divided into two parts: data sorting and index structure construction. Similarly, the present invention summarizes the indexing system into two levels: the index data layer and the index structure layer, both of which are constructed in a distributed manner using the MapReduce framework. First, selected keyword values ​​and the sequence numbers of the seismic traces are extracted from all seismic traces as index items. After all index items are sorted, the ordered index files are stored in a centralized storage, completing the construction of the index data layer. Subsequently, the index structure is established for the ordered index item data set.

[0088] The cumulative distribution function, also known as the distribution function, is the integral of the probability density function and can fully describe the probability distribution of a real random variable X. For all real discrete variables X, the cumulative distribution function represents the sum of the probabilities of all values ​​less than or equal to x. Its definition is as follows:

[0089] F X (x) = P(X≤x)

[0090] Therefore, for an ordered data set, its cumulative distribution function can be expressed as the ratio of the position of a certain value x in the set to the position of the entire data set. After knowing the number of elements in the data set, the product of this ratio and the total number of elements is the specific position of the value x in the entire data set. Assume that the cumulative distribution function of an ordered data set with a total number of elements N is F X (x), then the position O(x) of a value x in the overall data can be calculated by the following formula:

[0091] O(x)=N·F X (x)

[0092] Therefore, the index of each value in a data set can be calculated by the above formula. To ensure that the formula is valid, three conditions must be met: the data set is ordered as a whole, the overall data size N is known, and the cumulative distribution function F is X(x) can be known. Among them, the overall order of the data set can be completed by sorting the data set as a whole; at the same time, the overall data size can also be obtained after sorting; finally, the cumulative distribution function of the data set can be obtained by scanning and collecting data samples. When the data scale is small, the accurate cumulative distribution can be obtained by complete scanning. When the data scale is large, the approximate cumulative distribution function can be obtained by partial sampling of the data body. From this, it can be concluded that by learning the accurate or approximate cumulative distribution function of the data body through the machine learning model (i.e., function fitting), a mapping relationship between keywords (values) and positions (offsets) can be established. At the same time, the position relationship of unknown values ​​can also be predicted, thereby achieving the effect of data indexing, such as Figure 3 As shown. Since the learning index directly calculates the index prediction value of the query content, the time complexity of querying it is O(1). Its model can be roughly understood as a hash function based on a machine learning algorithm. The mapping relationship is simply as follows Figure 4 The multi-layer perceptron model shown can be implemented.

[0093] According to the above basic principles, the learning index takes the keyword value as input and the offset of the value in the index file as output, and uses the index items in the generated ordered index file as the training set to input the model for training. The scatter plot of the first keyword value and the index file offset of the value is shown in the figure below. Figure 5 As shown, this value-offset relationship can be understood as the cumulative distribution function of the keyword. This graph is similar to the overall curve of the node allocation graph, so the same model structure can be used for learning and prediction. However, due to the large number of seismic traces, if all of them are fed into a single model for training, the learning time will be very long and the learning accuracy will be low. To address this situation, a divide-and-conquer approach can be adopted. The entire index data can be divided into multiple segments, and a query model is established for each segment to learn and save. This not only increases learning accuracy but also enables the learning process to be executed in parallel, reducing the learning time.

[0094] The design of the learning index structure has the following main features:

[0095] [1.1] Build a learning index based on a distributed training method. The construction process is divided into three parts: sampling prediction, data sorting, and learning index construction.

[0096] [1.2] Use machine learning models to predict data distribution during sampling, scanning, prediction and sorting, and perform segmented sorting using node numbers assigned by the model to achieve efficient large-scale data sorting.

[0097] [1.3] All parts of the index construction process are distributed based on the MapReduce programming model;

[0098] [1.4] The number of index files is the same as the number of cluster nodes, and the number of sub-model trees of the learning index is the same as the number of index files;

[0099] [1.5] When querying, start from the root model, each layer of model selects the next layer of model, and the last layer of model outputs the approximate position value of the query data;

[0100] [1.6] After the query, the data and position values ​​are re-entered into the model for training to improve the query efficiency of high-frequency queries.

[0101] [2] Sampling prediction based on MLP model.

[0102] In the MapReduce framework, the GetPartition function can be used during the Map phase to send values ​​with the same key value to the same Reducer for unified processing. Therefore, when sorting using the MapReduce framework, each index item can be sent to a designated Reducer. An ordered, contiguous, and mutually exclusive key range is set for each Reducer. Each Mapper sends the index item to the Reducer responsible for processing that index item. After sorting, the Reducer generates multiple ordered index files, each of which is also ordered. This maximizes the parallelism of the cluster during sorting.

[0103] However, in real-world seismic trace data, different keywords vary in type, value range, and distribution. To avoid the "bucket effect" caused by some nodes processing too much data, which can negatively impact overall efficiency, it is necessary to ensure that each node receives as evenly distributed data as possible. This requires scanning the entire data before sorting to determine all possible keyword values ​​and the number of traces in which each value appears. These values ​​are then distributed as evenly and orderly as possible to different nodes based on the number of traces for each value. This scanning of all trace headers can also be implemented using the MapReduce framework. First, in the Map phase, each node reads the header keywords in parallel, calculates the keyword information (i.e., the keyword value and the number of traces it contains), and sends each node's keyword information to a unique node. In the Reduce phase, the receiving node collects all keyword information and merges it into a table of total keyword information. Each node is then assigned as evenly as possible the number of traces according to the keyword value, ultimately generating a table of sorted data ranges for each node. Subsequently, each node reads this table during the sort phase and sends different values ​​to a designated node, achieving segmented sorting. Then, a search structure is constructed for each file. The orderliness between the structures allows them to be directly merged into an overall index search structure, and the result can be obtained by performing a single search.

[0104] Since scanning all the data is very inefficient, and given that there are many channels with the same keyword value in the actual data, it is not possible to evenly distribute the same channel number to each node for sorting, a method combining sampling and machine learning can be used to predict the distribution of data. The main process of this solution is as follows: Figure 6 The specific steps are as follows:

[0105] (Step 2.1) Set the sampling size and the number of reducers to 1, and use the MapReduce framework to evenly distribute the sampling size to each node;

[0106] (Step 2.2) In the Map phase, each node reads seismic traces at intervals according to the sampling scale, extracts the selected keyword value in each seismic trace, and ensures that the number of seismic traces sampled and read meets the sampling scale;

[0107] (Step 2.3) In the Reduce phase, the read keyword information is integrated and the sampling results are regarded as the overall data scan results to generate the allocation table of the sorted data range of each node;

[0108] (Step 2.4) Take the keyword value as input and the node number assigned to the value as output. After normalization, input it into the machine learning model according to the channel ratio of the keyword value for training, fit the cumulative distribution function of the keyword, and save it.

[0109] [3] Distributed sorting based on the MapReduce programming model.

[0110] Traditional distributed sorting algorithms are divided into two parts: internal sorting within each node and merge sorting. In the final stage of sorting, the ordered data of each node needs to be merged and sorted. However, the merge step will concentrate all data IO on a single node, resulting in a serious reduction in sorting efficiency and a significant increase in time consumption. To address this, a segmented sorting method can be used to avoid merging and improve sorting efficiency. First, the data distribution is determined by scanning the entire data. Then, data of different non-overlapping numerical ranges are evenly assigned to each node, so that the sorted files are ordered both internally and between files, resulting in an overall ordered sorting result.

[0111] As the second part of the learning index construction process, the data sorting part has two purposes: sorting and generating index data files. The specific sorting process is as follows: Figure 7 As shown, it includes the following steps:

[0112] (Step 3.1) First, the header data is evenly divided into multiple segments during the Map phase. Each Map task is responsible for reading a segment of data. These Map tasks are then evenly distributed to the Map Workers on each node in the cluster.

[0113] (Step 3.2) After receiving the Map task, the Map Worker begins processing the corresponding trace header data segment. It reads all trace headers, sequentially extracts the selected keyword data and the trace number in each trace as index items, and stores them using key / value pairs.

[0114] (Step 3.3) The Map Worker obtains the node number corresponding to each index item by calling the node allocation model, and then sends each key / value to the corresponding Reduce Worker for sorting, thereby achieving segmented sorting;

[0115] (Step 3.4) After receiving all key / value pairs from the Map Worker, the Reduce Worker sorts the index data stored in the key / value pairs according to keyword order and stores the sorted index data in the file corresponding to the Reduce Worker. Simultaneously, it collects information about each keyword (unique values, next keyword maximum value, next keyword minimum value, etc.) and writes it to the corresponding keyword information file. At this point, all trace header index data has been extracted and sorted into multiple index files.

[0116] [4] Distributed learning index construction based on MapReduce framework.

[0117] The present invention designs a distributed learning index construction scheme based on the MapReduce framework, and its flow chart is as follows: Figure 8 As shown, the main steps are as follows:

[0118] (Step 4.1) Set the model's maximum training time time_max, maximum loss loss_max, and prediction tolerance offset_tol;

[0119] (Step 4.2) After the data is sorted, read the index items and corresponding offsets of all different values ​​of the index data;

[0120] (Step 4.3) All index items and corresponding offsets are passed into the model as a training set;

[0121] (Step 4.4) Start model training until the training time is greater than time_max, or the loss value is less than loss_max and the training output tolerance is less than offset_tol;

[0122] (Step 4.5) Save the model.

[0123] Combine the index model generated in this step with the node allocation model generated in the sampling prediction step, and use the node allocation model as the root node model to achieve the following: Figure 9 The hierarchical index model shown in the figure. When querying, start from the root model, each layer of models selects the next layer of models, and the last layer of models outputs the approximate position of the query value in the overall index data.

[0124] [5], Query algorithm based on learned index structure.

[0125] The query process for the learning index structure is as follows Figure 9 The specific steps are as follows:

[0126] (Step 5.1) Input the keyword value into the root model (i.e., the node allocation model), obtain the number of the lower model where the keyword is located, and so on down to the bottom model;

[0127] (Step 5.2) Predict the offset of the keyword in the index data file through the underlying model, and read the index file according to the predicted offset;

[0128] (Step 5.3) If the index item value read is inaccurate, the current item can be compared with the search value to determine whether to search forward or backward until the correct value is matched;

[0129] (Step 5.4) After the query, the query conditions and the true offset are passed into the model again for training.

[0130] [6] Compared with traditional index structures such as B+ trees, the present invention has the following advantages:

[0131] (1) The learning index structure only needs to save the underlying multi-layer perceptron model, and its space occupation is significantly smaller than that of the traditional index structure.

[0132] (2) After the learning index query, the data and the real offset are passed into the model again for training, which can effectively improve the query efficiency of the index for the same conditions, that is, it has higher high-frequency query performance.

[0133] (3) A distributed index construction algorithm based on the MapReduce programming model was designed and implemented, which can quickly and concurrently build indexes using the computing resources of multiple nodes and has good scalability, thereby improving the efficiency of index construction and enhancing the user experience of interactive applications.

[0134] (4) GPU can be used to greatly improve the efficiency of index construction.

[0135] (5) After one construction, data with similar distribution patterns can be reused, reducing model training time.

[0136] The seismic data query method of this embodiment has the following beneficial technical effects:

[0137] (1) Check whether there is an index that meets the query conditions according to the specified query. If the index exists, it is used directly; otherwise, it is constructed.

[0138] (2) Taking full advantage of high-performance clusters, the rapid construction of learning indexes is achieved based on the parallelization of the MapReduce programming model, so that the index construction tasks are dispatched to each node as evenly as possible. In a large-scale cluster environment, good performance scalability is achieved, and a near-linear acceleration ratio is obtained.

[0139] (3) For the constructed learning index structure, the corresponding query method proposed by the present invention is used to quickly perform queries.

[0140] (4) The query performance of the learned index is basically the same under different data scales.

[0141] (5) Under similar distribution patterns of data, the learning index can be reused to a certain extent, effectively reducing the index construction time.

[0142] In another embodiment, the present invention also provides a seismic data query method, which specifically includes the construction of a learning index and the use of the learned index after construction. The implementation example of the present invention mainly divides the construction of the learned index into two parts: data sorting and index construction, wherein the data sorting part can be further divided into sampling scanning and segmented sorting. First, based on the MapReduce framework and the machine learning model, data sampling and distribution rules are learned to generate a node allocation model; then, based on the MapReduce framework, the node allocation model is used to generate an index and segmented sorting to generate multiple ordered index files; finally, the MapReduce framework is used to parallelize the construction of a learning index for each index file. When querying a learned index, start from the root node model, each layer of model selects the next layer of model, and the last layer of model outputs the approximate position of the query value in the overall index data. The specific implementation steps of each part are as follows:

[0143] Data sampling and data distribution learning based on MapReduce framework and machine learning model:

[0144] (Step 1) In the Map phase, the MLTraceScanInputFormat class is called to divide the data into blocks. The number of divided data blocks is determined based on the number of available nodes in the cluster, and the blocks are stored in the InputSplit class.

[0145] (Step 2) After receiving the task, the Map Worker uses the MLTraceScanRecordReader class to read the header data of each data block by randomly sampling at intervals. It reads the header data of one track at a time and extracts the selected keyword value in the track header and stores it in a key / value pair.

[0146] (Step 3) The Map Worker sends each key / value pair to a unique Reduce task through the partitioning function GetPartition(const std::string&key,const std::string&value,int numPartitions) of the Partitioner class.

[0147] (Step 4) After receiving all key / value pairs sent by Map Worker, Reduce Worker uses the Compare(const std::string&key1,const std::string&key2) function of the MLTraceScanComparator class to sort the received index item sample data.

[0148] (Step 5) After the Reduce part is completed, the number of different values ​​of each keyword in the ordered index samples is counted, and the index item samples are distributed to each node in an orderly manner as evenly as possible to generate a keyword-node distribution table.

[0149] (Step 6) Build an MLP model based on the Libtorch framework, and pass the keyword value and node number in the keyword-node assignment table as the input and output of the model for training.

[0150] (Step 7) Finally, the trained model and its parameter information are stored in non-volatile storage.

[0151] Data sorting based on the MapReduce programming model:

[0152] (Step 1) In the Map phase, the MLTraceSortInputFormat class is called to divide the data into blocks. The number of divided data blocks is determined based on the number of available nodes in the cluster, and the blocks are stored in the InputSplit class.

[0153] (Step 2) After receiving the task, the Map Worker uses the MLTraceSortRecordReader class to read the header data of each data block, extract the selected keyword value in each header, and store it together with the current track number in a key / value pair.

[0154] (Step 3) The Map Worker reads the node allocation model through the partitioning function GetPartition(const std::string&key,const std::string&value,int numPartitions) of the MLTraceSortPartitioner class, inputs the keyword value of each index item into the model, obtains the corresponding node number, and sends each key / value pair to the corresponding Reduce task.

[0155] (Step 4) After receiving the key / value pairs sent by the Map Worker, the Reduce Worker uses the Compare(const std::string&key1,const std::string&key2) function of the MLTraceSortComparator class to sort the received index data, and finally uses the MLTraceSortRecordWriter class to write the index data and keyword information to non-volatile storage.

[0156] Learning index construction based on MapReduce programming model:

[0157] (Step 1) In the Map phase, call the MakeMLIndexInputFormat class to divide the data into blocks. The number of data blocks is determined based on the number of index files and stored in the InputSplit class.

[0158] (Step 2) After receiving the task, the Map Worker uses the MakeMLIndexRecordReader class to read the index data of each data block. Every time a fixed number of index items are read, they are counted as a training set for the underlying model. <std::string>&input_index_data,intmodel_id,double time_max,std::string save_path) function generates an underlying model for training and saving. After all leaf nodes are created, use MakeIndexNodeModel(std::vector <std::vector <std::string>>&index_data_strs,double time_max,std::string save_path) function completes the generation, training and saving of non-lower-level models of learning indexes from the bottom up.

[0159] (Step 3) The Map Worker sends each key / value pair to a unique Reduce task (before the task starts, the number of Reduce tasks has been set to 1 using the SetNumReduceTasks(int num) function).

[0160] Learning index query:

[0161] (Step 1) Acquire information of the current index through a given index path, and read the first keyword information file and the top-level keyword-node allocation model of the learning index.

[0162] (Step 2) by search_index(vector <bakeyvalue>&vFromKey,vector <bakeyvalue>&vToKey,std::vector<IndexData_v> &keyDatas) function, starting from the top-level model, searches the given start and end ranges of each keyword and stores all qualified index items in the keyDatas array.

[0163] (Step 3) Summarize all index item results, continue to filter other attributes (such as grouping, tolerance, etc.), and store the index items that meet all conditions in the BATraceIndexs class.

[0164] In some embodiments, please refer to Figure 10 As shown, the present invention further provides a seismic data query device 200, the device comprising:

[0165] A sampling learning module 201 is configured to perform data sampling and data distributed learning on seismic data based on a MapReduce framework and a machine learning model to generate a node allocation model;

[0166] A sorting module 202 is configured to perform distributed segment sorting on the seismic data based on the MapReduce framework and the node allocation model to generate an ordered index file;

[0167] A construction module 203 is configured to train a learning index including an underlying model and a non-underlying model based on a MapReduce framework and the ordered index file, and combine the learning index with the node allocation model to construct a learning index structure;

[0168] The query module 204 is configured to obtain keywords to be queried and input them into the learning index structure to obtain query results.

[0169] A seismic data query device of this embodiment has the following beneficial technical effects: the learning index structure can learn and store the distribution pattern of data with a small amount of space, significantly reducing the storage space of the index structure. It regards the index structure as a black box and simply regards the index structure as a model from the perspective of input and output data. It predicts its position in the ordered data body by fitting the cumulative distribution function of the input data, and uses a machine learning model to learn and predict the distribution pattern of seismic data, thereby realizing the indexing function. The learning index can be reused to a certain extent, effectively reducing the index construction time and improving the efficiency of index construction.

[0170] It should be noted that the specific limitations of the seismic data query device can be found in the limitations of the seismic data query method above and will not be repeated here. The various modules in the above-mentioned seismic data query device can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in hardware form, or can be stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0171] According to another aspect of the present invention, an electronic device is provided. The electronic device may be a server. Figure 11 As shown. The electronic device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above-mentioned seismic data query method is implemented.

[0172] According to another aspect of the present invention, a computer readable storage medium is provided. Figure 12 As shown, a computer program is stored thereon, and when the computer program is executed by a processor, the above-mentioned seismic data query method is implemented.

[0173] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0174] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.< / bakeyvalue> < / bakeyvalue> < / std::string> < / std::string>

Claims

1. A seismic data query method, characterized in that: The method comprises: Based on the MapReduce framework and machine learning model, data sampling and data distributed learning are performed on seismic data to generate a node allocation model; Based on the MapReduce framework and the node allocation model, distributed segment sorting of seismic data is performed to generate an ordered index file; Based on the MapReduce framework, the ordered index file trains a learning index including an underlying model and a non-underlying model, and combines the learning index with the node allocation model to construct a learning index structure; Obtaining keywords to be queried and inputting them into the learning index structure to obtain query results; The steps of training a learning index including an underlying model and a non-underlying model based on the MapReduce framework and the ordered index file, and combining the learning index with the node allocation model to construct a learning index structure include: In the Map phase, the MakeMLIndexInputFormat class is called to divide the data into blocks. The number of data blocks is determined based on the number of index files and stored in the InputSplit class. After receiving the task, the Map Worker uses the MakeMLIndexRecordReader class to read the index data for each data block. Every time a fixed number of index items are read, they are counted as a training set for an underlying model in the learning index. The MakeIndexLeafModel function is used to generate an underlying model in the learning index for training and storage. After all leaf nodes are created, use the MakeIndexNodeModel function to complete the generation, training, and saving of the non-bottom-level model of the learning index from bottom to top; The node allocation model is used as the root node model of the learning index to obtain a learning index structure, wherein the learning index structure starts from the root node model when querying, each layer model selects the next layer model, and the last layer model outputs the approximate position of the query value in the overall index data.

2. The seismic data query method according to claim 1, characterized in that: The steps of performing data sampling and data distributed learning on seismic data based on the MapReduce framework and the machine learning model to generate a node allocation model include: Set the sampling size and the number of reducers to 1, and use the MapReduce framework to evenly distribute the sampling size to each node; In the Map phase, each node reads seismic traces at intervals according to the sampling scale, extracts the keyword information containing the selected keyword value in each seismic trace, and ensures that the number of seismic traces sampled and read meets the sampling scale; In the Reduce phase, the read keyword information is integrated, and the sampling results are used as the overall data scan results to generate the allocation table of the subsequent sorting data range of each node; Based on the allocation table, the keyword value is used as input and the node number specified for the value is used as output. After normalization, the data is passed into the machine learning model according to the proportion of the number of keyword values ​​for training. The cumulative distribution function of the keyword is fitted and saved to generate a node allocation model.

3. The seismic data query method according to claim 1, characterized in that: The step of performing distributed segment sorting on the seismic data based on the MapReduce framework and the node allocation model to generate an ordered index file includes: In the Map phase, the header data is evenly divided into multiple segments. Each Map task is responsible for reading a segment of data. These Map tasks are then evenly distributed to the Map Workers of each node in the cluster. After receiving the Map task, the Map Worker starts processing the corresponding trace header data segment, reading all trace headers, sequentially extracting the selected keyword data and the trace number in each trace as index items, and storing them using key / value pairs; The Map Worker obtains the node number corresponding to each index item by calling the node allocation model, and then sends each key / value to the corresponding Reduce Worker for sorting; After the Reduce Worker receives all the key / value pairs sent by the Map Worker, it sorts the index data stored in the key / value pairs according to the keyword order, and finally stores the sorted index data in the file corresponding to the Reduce Worker, and writes the statistics of each keyword information into the corresponding keyword information file.

4. The seismic data query method according to claim 1, characterized in that: The step of obtaining keywords to be queried and inputting them into the learning index structure to obtain query results includes: Obtain the keyword to be queried input by the user, and input the keyword value into the root node model of the learning index structure, obtain the lower model number where the keyword is located, and so on downward to the bottom model; The underlying model is used to predict the offset of the keyword in the index data file, and the index file is read based on the predicted offset. If the index item value read is inaccurate, you can determine whether to search forward or backward item by item by comparing the current item with the search value until the correct value is matched.

5. The seismic data query method according to claim 4, characterized in that: The method further comprises: After the query, the query conditions and the actual offset are passed into the model again as training data for training to update the learning index structure.

6. The seismic data query method according to claim 1, characterized in that: The machine learning model is a multi-layer perceptron model.

7. A seismic data query device, characterized in that: The device comprises: A sampling learning module is configured to perform data sampling and data distributed learning on seismic data based on the MapReduce framework and the machine learning model to generate a node allocation model; A sorting module configured to perform distributed segment sorting on the seismic data based on the MapReduce framework and the node allocation model to generate an ordered index file; A construction module is configured to train a learning index including an underlying model and a non-underlying model based on a MapReduce framework and the ordered index file, and combine the learning index with the node allocation model to construct a learning index structure; the steps of training a learning index including an underlying model and a non-underlying model based on a MapReduce framework and the ordered index file, and combining the learning index with the node allocation model to construct a learning index structure include: in the Map stage, calling the MakeMLIndexInputFormat class to divide the data blocks, determining the number of data blocks according to the number of index files, and storing them in the InputSplit class; Map After receiving the task, the Worker uses the MakeMLIndexRecordReader class to read the index data of each data block. Every time a fixed number of index items are read, they are counted as a training set of an underlying model in the learning index. The MakeIndexLeafModel function is used to generate an underlying model in the learning index for training and storage. After all leaf nodes are created, the MakeIndexNodeModel function is used to complete the generation, training and storage of non-bottom-level models of the learning index from bottom to top. The node allocation model is used as the root node model of the learning index to obtain a learning index structure, wherein the learning index structure starts from the root node model when querying, each layer model selects the next layer model, and the last layer model outputs the approximate position of the query value in the overall index data. The query module is configured to obtain keywords to be queried and input them into the learning index structure to obtain query results.

8. An electronic device, characterized in that: include: at least one processor; as well as A memory storing a computer program executable in the processor, wherein the processor executes the method according to any one of claims 1 to 6 when executing the program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is performed.

Images