Seismic data query method and device, electronic equipment and medium
The learning index structure is constructed through the MapReduce framework and machine learning model, which solves the problem of inefficiency of traditional B+ trees in massive seismic data processing, realizes efficient index construction and query, and improves the performance of seismic data query.
Patent Information
- Application Number
- CN202310594488.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-05-24
AI Technical Summary
Traditional B+ trees are inefficient in building and querying when processing massive seismic data, especially in a multi-threaded environment, and cannot effectively support multi-keyword range search, resulting in a significant reduction in querying efficiency.
The MapReduce framework and machine learning model are used to generate node allocation models, and a learning index structure is constructed through distributed segmentation sorting, and combined with underlying and non-underground models for training to form a learning index that can be queried efficiently.
It significantly reduces the storage space of the index structure, improves the efficiency of index construction and query speed, can learn and store data distribution rules in a small amount of space, supports efficient multi-keyword range search, and improves the user experience of interactive applications.
Smart Images

Figure CN119025481A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of seismic exploration, and in particular to a seismic data query method, device, electronic equipment and medium. Background Art
[0002] Seismic data processing is an important technology in the oil exploration industry. Its function is to process and calculate the seismic data collected in the field according to a specific processing algorithm, so as to obtain an image of the underground geological structure, which is used to guide subsequent drilling and oil production. With the continuous application of new exploration technologies and high-precision acquisition technologies in oil exploration, the amount of original seismic data collected from the field has grown rapidly. At present, the scale of a single data body has exceeded the PB level, and the number of seismic channels can reach hundreds of billions. The object processed by seismic applications is usually such a huge seismic data body. The seismic data body is logically similar to a data table in a relational database. It is organized in row order, and each row of records is called a seismic channel. The seismic channel consists of two parts, the header and the body. The header stores the attribute information related to the seismic channel, including the shot point coordinates, the receiver point coordinates, the number of sampling points, the shot number, the channel number and other information. Each attribute is called a header keyword. The channel body is a floating point array, and each floating point number is called a sampling point. Since the seismic data volume is high-dimensional structured data, each seismic trace has hundreds of attribute information stored in different trace header keywords.
[0003] However, a large number of interactive seismic applications are usually only interested in part of the data set of the seismic data body when accessing the seismic data body. Therefore, a large number of seismic data accesses will specify the value range of some attributes to filter out specific data sets, and may also specify the order of certain attributes to determine the sorting method of the query results. Since multi-dimensional range query is the most common data query mode in seismic applications, its query speed is crucial to the performance of seismic applications, especially interactive applications, and user experience. Efficient index query is the basis for ensuring query efficiency and reducing seismic data query latency. As a balanced search tree designed for disks or other direct access auxiliary storage devices, B+ tree index can effectively reduce the number of disk I / O operations during queries. And because B+ tree can support fast range scanning along leaf nodes, it has better range query performance.
[0004] However, the traditional B+ tree construction method is to insert each record into an empty tree, and the process involves many complex operations, such as node splitting and rotation. For hundreds of billions of unordered data, the overhead of this traditional construction method is huge, and it is difficult to build in parallel on multiple machines, and the speedup ratio in a multi-threaded environment is also relatively limited. At the same time, if each piece of data is stored in a B+ tree, the final tree size will also be huge, and the search will still face the problem of low efficiency. In addition, the B+ tree can only be used to search for a single keyword. When searching in a multi-keyword range, it is necessary to first query all data that meets the first keyword condition, and then filter the subsequent keywords, which will greatly reduce the search efficiency. Therefore, for massive seismic data, it is necessary to design a distributed index construction method that is different from the traditional B+ tree structure to improve the index construction efficiency, speedup ratio, scalability, and query efficiency. Summary of the invention
[0005] In view of this, it is necessary to provide a seismic data query method, device, electronic device and medium to address the above technical issues.
[0006] According to a first aspect of the present invention, a seismic data query method is provided, the method comprising:
[0007] Based on the MapReduce framework and machine learning model, data sampling and data distributed learning are performed on seismic data to generate a node allocation model;
[0008] Based on the MapReduce framework and the node allocation model, the seismic data is sorted in distributed segments to generate an ordered index file;
[0009] Based on the MapReduce framework, the ordered index file trains a learning index including an underlying model and a non-underlying model, and combines the learning index with the node allocation model to construct a learning index structure;
[0010] The keywords to be queried are obtained and input into the learning index structure to obtain query results.
[0011] In some embodiments, the step of performing data sampling and data distributed learning on seismic data based on the MapReduce framework and the machine learning model to generate a node allocation model includes:
[0012] Set the sampling scale and set the number of Reducers to 1, and use the MapReduce framework to evenly distribute the sampling scale to each node;
[0013] In the Map stage, each node reads seismic traces at intervals according to the sampling scale, extracts the keyword information containing the selected keyword value in each seismic trace, and ensures that the number of seismic traces sampled and read meets the sampling scale;
[0014] In the Reduce phase, the read keyword information is integrated, and the sampling results are used as the overall data scanning results to generate the allocation table of the sorting data range of each node;
[0015] Based on the allocation table, the keyword value is used as input and the node number specified for the value is used as output. After normalization, it is passed into the machine learning model for training according to the proportion of the keyword value. The cumulative distribution function of the keyword is fitted and saved to generate a node allocation model.
[0016] In some embodiments, the step of performing distributed segment sorting of seismic data based on the MapReduce framework and the node allocation model to generate an ordered index file includes:
[0017] In the Map phase, the header data is evenly divided into multiple segments. Each Map task is responsible for reading a segment of data. These Map tasks are then evenly distributed to the Map Workers of each node in the cluster.
[0018] After receiving the Map task, the Map Worker starts processing the corresponding trace header data segment, reading all trace headers, extracting the selected keyword data and the trace number in each trace in sequence as index items, and storing them using key / value pairs;
[0019] The Map Worker obtains the node number corresponding to each index item by calling the node allocation model, and then sends each key / value to the corresponding Reduce Worker for sorting;
[0020] After the Reduce Worker receives all the key / value pairs sent by the Map Worker, it sorts the index data stored in the key / value pairs according to the keyword order, and finally stores the sorted index data into the file corresponding to the Reduce Worker, and writes the statistics of each keyword information into the corresponding keyword information file.
[0021] In some embodiments, the step of training a learning index including an underlying model and a non-underlying model based on the MapReduce framework and the ordered index file, and combining the learning index with the node allocation model to construct a learning index structure includes:
[0022] In the Map stage, call the MakeMLIndexInputFormat class to divide the data into blocks. The number of data blocks is determined based on the number of index files and stored in the InputSplit class.
[0023] After receiving the task, the Map Worker uses the MakeMLIndexRecordReader class to read the index data of each data block. Every time a fixed number of index items are read, they are counted as a training set of an underlying model in the learning index. The MakeIndexLeafModel function is used to generate an underlying model in the learning index for training and saving.
[0024] After all leaf nodes are created, use the MakeIndexNodeModel function to complete the generation, training, and saving of the non-bottom-level model of the learning index from bottom to top;
[0025] The node allocation model is used as the root node model of the learning index to obtain a learning index structure, wherein the learning index structure starts from the root node model when querying, each layer of models selects the next layer of models, and the last layer of models outputs the approximate position of the query value in the overall index data.
[0026] In some embodiments, the step of obtaining keywords to be queried and inputting them into the learning index structure to obtain query results includes:
[0027] Obtain the keyword to be queried input by the user, and input the keyword value into the root node model of the learning index structure, obtain the lower model number where the keyword is located, and search downwards in this way until the bottom model;
[0028] The underlying model is used to predict the offset of the keyword in the index data file, and the index file is read according to the predicted offset.
[0029] If the index item value read is inaccurate, you can determine whether to search forward or backward item by item by comparing the current item with the search value until the correct value is matched.
[0030] In some embodiments, the method further comprises:
[0031] After the query, the query conditions and the actual offset are passed into the model again as training data for training to update the learning index structure.
[0032] In some embodiments, the machine learning model is a multi-layer perceptron model.
[0033] According to a second aspect of the present invention, a seismic data query device is provided, the device comprising:
[0034] A sampling learning module, configured to perform data sampling and data distributed learning on seismic data based on a MapReduce framework and a machine learning model to generate a node allocation model;
[0035] A sorting module configured to perform distributed segment sorting of seismic data based on a MapReduce framework and the node allocation model to generate an ordered index file;
[0036] A construction module configured to train a learning index including an underlying model and a non-underlying model based on a MapReduce framework and the ordered index file, and to combine the learning index with the node allocation model to construct a learning index structure;
[0037] The query module is configured to obtain keywords to be queried and input them into the learning index structure to obtain query results.
[0038] According to a third aspect of the present invention, there is further provided an electronic device, the electronic device comprising:
[0039] at least one processor; and
[0040] The memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned seismic data query method when executing the program.
[0041] According to a fourth aspect of the present invention, there is further provided a computer-readable storage medium, the computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the aforementioned seismic data query method is executed.
[0042] The above-mentioned seismic data query method has the following beneficial technical effects: the learning index structure can learn and store the distribution law of data with a small amount of space, significantly reducing the storage space of the index structure. It regards the index structure as a black box and simply regards the index structure as a model from the perspective of input and output data. It predicts its position in the ordered data body by fitting the cumulative distribution function of the input data, and uses a machine learning model to learn and predict the distribution law of seismic data, thereby realizing the indexing function. The learning index can be reused to a certain extent, effectively reducing the index construction time and improving the efficiency of index construction.
[0043] In addition, the present invention also provides a seismic data query device, an electronic device and a computer-readable storage medium, which can also achieve the above-mentioned technical effects and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.
[0045] Figure 1 A flowchart of a seismic data query method provided by one embodiment of the present invention;
[0046] Figure 2 A schematic diagram of the overall structure of a large-scale seismic data indexing system for implementing the method of the present invention provided in one embodiment of the present invention;
[0047] Figure 3 It is a schematic diagram of the implementation principle of the learning indexing function designed by the present invention;
[0048] Figure 4 It is a schematic diagram of a multi-layer perceptron model used in the learning index designed by the present invention;
[0049] Figure 5 It is a schematic diagram of the relationship between the first keyword value and the offset in PB-level seismic data using the indexing function of the method of the present invention;
[0050] Figure 6 It is a schematic diagram of the data sampling process based on the MapReduce framework and the machine learning model in the method of the present invention;
[0051] Figure 7 It is a schematic diagram of the distributed segment sorting process based on the MapReduce framework in the method of the present invention;
[0052] Figure 8 It is a schematic diagram of the process of building a learning index based on the MapReduce framework in the method of the present invention;
[0053] Fig. 9 It is a schematic diagram of the learning index structure and query flow in the method of the present invention;
[0054] Fig.10 A schematic structural diagram of a seismic data query device provided by another embodiment of the present invention;
[0055] Fig.11 is an internal structure diagram of an electronic device in another embodiment of the present invention;
[0056] Fig.12 This is a structural diagram of a computer-readable storage medium in another embodiment of the present invention. DETAILED DESCRIPTION
[0057] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0058] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. The subsequent embodiments will not explain this one by one.
[0059] In one embodiment, please refer to Figure 1 As shown, the present invention provides a seismic data query method 100. Specifically, the method comprises the following steps:
[0060] Step 101, performing data sampling and data distributed learning on seismic data based on the MapReduce framework and the machine learning model to generate a node allocation model;
[0061] Step 102, based on the MapReduce framework and the node allocation model, the seismic data is sorted in a distributed manner to generate an ordered index file;
[0062] Step 103, based on the MapReduce framework, the ordered index file is used to train a learning index including an underlying model and a non-underlying model, and the learning index is combined with the node allocation model to construct a learning index structure;
[0063] Step 104: Obtain the keyword to be searched and input it into the learning index structure to obtain the search result.
[0064] A seismic data query method of this embodiment has the following beneficial technical effects: the learning index structure can learn and store the distribution law of data with a small amount of space, significantly reducing the storage space of the index structure. It regards the index structure as a black box and simply regards the index structure as a model from the perspective of input and output data. It predicts its position in the ordered data body by fitting the cumulative distribution function of the input data, and uses a machine learning model to learn and predict the distribution law of seismic data, thereby realizing the indexing function. The learning index can be reused to a certain extent, effectively reducing the index construction time and improving the efficiency of index construction.
[0065] In some embodiments, the aforementioned step 101, performing data sampling and data distributed learning on seismic data based on the MapReduce framework and the machine learning model to generate a node allocation model, includes:
[0066] Set the sampling scale and set the number of Reducers to 1, and use the MapReduce framework to evenly distribute the sampling scale to each node;
[0067] In the Map stage, each node reads seismic traces at intervals according to the sampling scale, extracts the keyword information containing the selected keyword value in each seismic trace, and ensures that the number of seismic traces sampled and read meets the sampling scale;
[0068] In the Reduce phase, the read keyword information is integrated, and the sampling results are used as the overall data scanning results to generate the allocation table of the sorting data range of each node;
[0069] Based on the allocation table, the keyword value is used as input and the node number specified for the value is used as output. After normalization, it is passed into the machine learning model for training according to the proportion of the keyword value. The cumulative distribution function of the keyword is fitted and saved to generate a node allocation model.
[0070] In some embodiments, the aforementioned step 102, based on the MapReduce framework and the node allocation model, performs distributed segment sorting on the seismic data to generate an ordered index file, includes:
[0071] In the Map phase, the header data is evenly divided into multiple segments. Each Map task is responsible for reading a segment of data. These Map tasks are then evenly distributed to the Map Workers of each node in the cluster.
[0072] After receiving the Map task, the Map Worker starts processing the corresponding trace header data segment, reading all trace headers, extracting the selected keyword data and the trace number in each trace in sequence as index items, and storing them using key / value pairs;
[0073] The Map Worker obtains the node number corresponding to each index item by calling the node allocation model, and then sends each key / value to the corresponding Reduce Worker for sorting;
[0074] After the Reduce Worker receives all the key / value pairs sent by the Map Worker, it sorts the index data stored in the key / value pairs according to the keyword order, and finally stores the sorted index data into the file corresponding to the Reduce Worker, and writes the statistics of each keyword information into the corresponding keyword information file.
[0075] In some embodiments, the aforementioned step 103, based on the MapReduce framework, the ordered index file is used to train a learning index including an underlying model and a non-underlying model, and the learning index is combined with the node allocation model to construct a learning index structure, including:
[0076] In the Map stage, call the MakeMLIndexInputFormat class to divide the data into blocks. The number of data blocks is determined based on the number of index files and stored in the InputSplit class.
[0077] After receiving the task, the Map Worker uses the MakeMLIndexRecordReader class to read the index data of each data block. Every time a fixed number of index items are read, they are counted as a training set of an underlying model in the learning index. The MakeIndexLeafModel function is used to generate an underlying model in the learning index for training and saving.
[0078] After all leaf nodes are created, use the MakeIndexNodeModel function to complete the generation, training, and saving of the non-bottom-level model of the learning index from bottom to top;
[0079] The node allocation model is used as the root node model of the learning index to obtain a learning index structure, wherein the learning index structure starts from the root node model when querying, each layer of models selects the next layer of models, and the last layer of models outputs the approximate position of the query value in the overall index data.
[0080] In some embodiments, the aforementioned step 104, obtaining the keyword to be queried and inputting it into the learning index structure to obtain the query result, includes:
[0081] Obtain the keyword to be queried input by the user, and input the keyword value into the root node model of the learning index structure, obtain the lower model number where the keyword is located, and search downwards in this way until the bottom model;
[0082] The underlying model is used to predict the offset of the keyword in the index data file, and the index file is read according to the predicted offset.
[0083] If the index item value read is inaccurate, you can determine whether to search forward or backward item by item by comparing the current item with the search value until the correct value is matched.
[0084] In some embodiments, the method further comprises:
[0085] After the query, the query conditions and the actual offset are passed into the model again as training data for training to update the learning index structure.
[0086] In some embodiments, the machine learning model is a multi-layer perceptron model.
[0087] In another embodiment, in order to facilitate understanding of the solution of the present invention, the seismic data query method of the present invention is used on a high-performance cluster as an example. Figure 2 As shown, Figure 2 The large-scale seismic data indexing system is generally divided into two parts: index construction and index query, wherein the index construction is further divided into two parts: data sorting and index structure construction. Similarly, the present invention summarizes the index system into two levels: index data layer and index structure layer, both of which are distributedly constructed using the MapReduce framework. First, the selected keyword value and the sequence number of the seismic track are extracted from all seismic tracks as index items, and after sorting all index items, the ordered index files are stored in centralized storage to complete the construction of the index data layer; thereafter, an index structure is established for the ordered index item data set.
[0088] The cumulative distribution function, also known as the distribution function, is the integral of the probability density function and can fully describe the probability distribution of a real random variable X. For all real discrete variables X, the cumulative distribution function represents the sum of the probabilities of all values less than or equal to x. Its definition is as follows:
[0089] F X (x) = P(X≤x)
[0090] Therefore, for an ordered data set, its cumulative distribution function can be expressed as the ratio of the position of a certain value x in the set in the overall data set. After knowing the number of elements in the data set, the product of this ratio and the total number of elements is the specific position of the value x in the overall data set. Assume that the cumulative distribution function of an ordered data set with a total number of elements N is F X (x), then the position O(x) of one value x in the overall data can be calculated by the following formula:
[0091] O(x)=N·F X (x)
[0092] Therefore, the index of each value in a data set can be calculated by the above formula. To ensure that the formula is valid, three conditions must be met: the overall order of the data set, the overall data size N is known, and the cumulative distribution function F X(x) can be known. Among them, the overall order of the data set can be achieved by sorting the data set as a whole; at the same time, the overall data size can also be obtained after sorting; finally, the cumulative distribution function of the data set can be obtained by scanning and collecting data samples. When the data scale is small, the precise cumulative distribution can be obtained by a complete scan. When the data scale is large, the approximate cumulative distribution function can be obtained by partial sampling of the data body. From this, it can be concluded that by learning the precise or approximate cumulative distribution function of the data body through the machine learning model (i.e., function fitting), a mapping relationship between keywords (values) and positions (offsets) can be established. At the same time, the position relationship of unknown values can also be predicted, thereby achieving the effect of data indexing, such as Figure 3 As shown in Figure 2. Since the learning index directly calculates the index prediction value of the query content, the time complexity of querying it is O(1). Its model can be roughly understood as a hash function based on a machine learning algorithm. The mapping relationship is simply as follows: Figure 4 The multi-layer perceptron model shown can be implemented.
[0093] According to the above basic principles, the learning index takes the keyword value as input and the offset of the value in the index file as output, and uses the index items in the generated ordered index file as the training set to input the model for training. The scatter plot of the first keyword value and the index file offset of the value is shown in Figure 1. Figure 5 As shown in the figure, the value-offset relationship can be understood as the cumulative distribution function of the keyword. This graph is similar to the overall curve of the node allocation graph, so the same model structure can be used for learning and prediction. However, since the total amount of seismic traces is large, if all are input into one model for training, it will result in huge learning time and low learning accuracy. In view of this situation, the idea of divide and conquer can be adopted, and all index data can be divided into multiple segments. A query model is established for each segment index for learning and saving, which not only increases the learning accuracy but also enables the learning process to be executed in parallel, reducing the time consumption of the learning stage.
[0094] The design ideas of the learning index structure have the following main features:
[0095] [1.1] The construction of a learning index is based on a distributed training method. The construction process is divided into three parts: sampling prediction, data sorting, and learning index construction.
[0096] [1.2] Use machine learning models to predict data distribution during sampling scan prediction and sorting, and perform segmented sorting using node numbers assigned by the model to achieve efficient large-scale data sorting;
[0097] [1.3] All parts of the index construction process are distributed based on the MapReduce programming model;
[0098] [1.4] The number of index files is the same as the number of cluster nodes, and the number of sub-model trees of the learning index is the same as the number of index files;
[0099] [1.5] When querying, start from the root model, each layer of the model selects the next layer of the model, and the last layer of the model outputs the approximate position value of the query data;
[0100] [1.6] After the query, the data and position values are passed into the model again for training to improve the query efficiency of high-frequency queries.
[0101] [2] Sampling prediction based on MLP model.
[0102] In the MapReduce framework, the Map stage can use the GetPartition function to send the Value of the same Key value to the same Reducer for unified processing. Therefore, when using the MapReduce framework for sorting, each index item data can be sent to the Reducer with a specified number. Set an ordered, continuous and non-intersecting keyword range for each Reducer. Each Mapper sends the index item to the Reducer responsible for processing the index item. After sorting, the Reducer will generate multiple ordered index files, and each file is also ordered. In this way, the parallelism of the cluster is maximized during sorting.
[0103] However, in actual seismic trace data, different keywords have different properties such as type, value range and distribution law. In order to avoid the barrel effect caused by some nodes processing too much data, which makes the overall efficiency counterproductive, it is necessary to ensure that the amount of sorted data allocated to each node is as even as possible. This requires scanning the entire data before sorting, determining all the values of the keyword and the number of traces where each value appears, and then allocating them to different nodes as evenly and orderly as possible according to the number of traces of each value. The scanning of all trace headers can also be realized based on the MapReduce framework. First, in the Map stage, each node reads the trace header keyword in parallel, counts the read keyword information (i.e., the keyword value and the number of traces contained in the value), and sends the keyword information of each node to a unique node; then in the Reduce stage, the node responsible for receiving information collects all keyword information and merges it into a total keyword information table. Allocate as evenly as possible the number of traces for each node according to the keyword value, and finally generate a distribution table of the sorted data range of each node. After that, each node reads the distribution table in the sorting stage and sends different values to the designated node to achieve segmented sorting. Then, a search structure is constructed for each file. The orderliness between the structures allows them to be directly merged into an overall index search structure, and the result can be obtained by performing a single search.
[0104] Since the efficiency of scanning all the data is very low, and given that there are many channels with the same keyword value in the actual data, it is not possible to evenly distribute the same number of channels to each node for sorting, a method combining sampling and machine learning can be used to predict the distribution of data. The main process of this solution is as follows: Figure 6 As shown, the specific steps are as follows:
[0105] (Step 2.1) Set the sampling scale and set the number of Reducers to 1, and use the MapReduce framework to evenly distribute the sampling scale to each node;
[0106] (Step 2.2) In the Map stage, each node reads the seismic traces at intervals according to the sampling scale, extracts the selected keyword value in each seismic trace, and ensures that the number of seismic traces sampled and read meets the sampling scale;
[0107] (Step 2.3) In the Reduce phase, the read keyword information is integrated, and the sampling results are regarded as the overall data scanning results to generate the allocation table of the sorting data range of each node;
[0108] (Step 2.4) The keyword value is used as input, and the node number specified for the value is used as output. After normalization, it is passed into the machine learning model for training according to the number of keyword values, and the cumulative distribution function of the keyword is fitted and saved.
[0109] [3] Distributed sorting based on the MapReduce programming model.
[0110] The traditional distributed sorting algorithm is divided into two parts: internal sorting of each node and merge sorting. In the final stage of sorting, the ordered data of each node needs to be merged and sorted. However, the merge step will concentrate all data IO on a single node, resulting in a serious reduction in sorting efficiency and a significant increase in time consumption. In this regard, the segmented sorting method can be used to avoid merging and improve sorting efficiency. First, the data distribution is determined by scanning the data as a whole, and then data of different non-intersecting numerical ranges are evenly allocated to each node, so that the sorted files are ordered both internally and between files, thereby obtaining an overall ordered sorting result.
[0111] As the second part of the learning index construction process, the data sorting part has two purposes: sorting and generating index data files. The specific sorting process is as follows: Figure 7 As shown, it includes the following steps:
[0112] (Step 3.1) First, the header data is evenly divided into multiple segments in the Map phase. Each Map task is responsible for reading a segment of data. These Map tasks are then evenly distributed to the Map Workers of each node in the cluster.
[0113] (Step 3.2) After receiving the Map task, the Map Worker starts processing the corresponding trace header data segment. It reads all trace headers, extracts the selected keyword data and the trace number in each trace in sequence as index items, and stores them using key / value pairs;
[0114] (Step 3.3) The Map Worker obtains the node number corresponding to each index item by calling the node allocation model, and sends each key / value to the corresponding Reduce Worker for sorting, thereby achieving segmented sorting;
[0115] (Step 3.4) After receiving all the key / value pairs sent by the Map Worker, the Reduce Worker sorts the index data stored in the key / value pairs according to the keyword order, and finally stores the sorted index data in the file corresponding to the Reduce Worker. At the same time, the information of each keyword (non-repeated value, next keyword maximum value, next keyword minimum value, etc.) is counted and written into the corresponding keyword information file. At this point, all the header index data is extracted and sorted into multiple index files.
[0116] [4] Distributed learning index construction based on MapReduce framework.
[0117] The present invention designs a distributed learning index construction scheme based on the MapReduce framework, and its flow chart is as follows: Figure 8 As shown, the main steps are as follows:
[0118] (Step 4.1) Set the maximum training time time_max, maximum loss loss_max and prediction tolerance offset_tol of the model;
[0119] (Step 4.2) After the data is sorted, read the index items and corresponding offsets of all different values of the index data;
[0120] (Step 4.3) All index items and corresponding offsets are passed into the model as a training set;
[0121] (Step 4.4) Start model training until the training time is greater than time_max, or the loss value is less than loss_max and the tolerance of the training output is less than offset_tol;
[0122] (Step 4.5) Save the model.
[0123] Combine the index model generated in this step with the node allocation model generated in the sampling prediction step, and use the node allocation model as the root node model to achieve the following Fig. 9 The hierarchical index model shown in the figure. When querying, start from the root model, each layer of models selects the next layer of models, and the last layer of models outputs the approximate position of the query value in the overall index data.
[0124] [5] Query algorithm based on learned index structure.
[0125] The query process for the learning index structure is as follows Fig. 9 As shown, the specific steps are as follows:
[0126] (Step 5.1) Input the keyword value into the root model (i.e., the node allocation model), obtain the number of the lower model where the keyword is located, and search downwards in this way until the bottom model;
[0127] (Step 5.2) Predict the offset of the keyword in the index data file through the underlying model, and read the index file according to the predicted offset;
[0128] (Step 5.3) If the index item value read is inaccurate, the current item can be compared with the search value to determine whether to search forward or backward until the correct value is matched;
[0129] (Step 5.4) After the query, the query conditions and the true offset are passed into the model again for training.
[0130] [6] Compared with traditional index structures such as B+ trees, the present invention has the following advantages:
[0131] (1) The learned index structure only needs to save the underlying multi-layer perceptron model, and its space occupancy is significantly smaller than that of the traditional index structure.
[0132] (2) After the learning index query, the data and the true offset are passed into the model again for training, which can effectively improve the query efficiency of the index for the same conditions, that is, it has higher high-frequency query performance.
[0133] (3) We designed and implemented a distributed index construction algorithm based on the MapReduce programming model, which can utilize the computing resources of multiple nodes to quickly and concurrently build indexes with good scalability, thereby improving the efficiency of index construction and enhancing the user experience of interactive applications.
[0134] (4) The efficiency of index construction can be greatly improved by using GPU.
[0135] (5) After being built once, data with similar distribution patterns can be reused, reducing model training time.
[0136] The seismic data query method of this embodiment has the following beneficial technical effects:
[0137] (1) Check whether there is an index that meets the query conditions according to the specified query. If the index exists, it is used directly; otherwise, it is constructed.
[0138] (2) Taking full advantage of high-performance clusters, the rapid construction of learning indexes is achieved based on the parallelization of the MapReduce programming model, so that the index construction tasks can be evenly scheduled to each node for execution as much as possible. This achieves good performance scalability in a large-scale cluster environment and a near-linear acceleration ratio.
[0139] (3) For the constructed learning index structure, the corresponding query method proposed by the present invention is used to quickly perform a query.
[0140] (4) The query performance of the learned index is basically the same under different data scales.
[0141] (5) For data with similar distribution patterns, the learned index can be reused to a certain extent, effectively reducing the index construction time.
[0142] In another embodiment, the present invention also provides a seismic data query method, which specifically includes the construction of a learned index and the use of the learned index after construction. The implementation example of the present invention mainly divides the construction of the learned index into two parts: data sorting and index construction, wherein the data sorting part can be further divided into sampling scanning and segmented sorting. First, based on the MapReduce framework and the machine learning model, data sampling and distribution rules are learned to generate a node allocation model; then, based on the MapReduce framework, the node allocation model is used to generate an index and segmented sorting to generate multiple ordered index files; finally, the MapReduce framework is used to parallelize each index file to build a learned index. When querying a learned index, start from the root node model, each layer of the model selects the next layer of the model, and the last layer of the model outputs the approximate position of the query value in the overall index data. The specific implementation steps of each part are as follows:
[0143] Data sampling and data distribution learning based on MapReduce framework and machine learning model:
[0144] (Step 1) In the Map phase, the MLTraceScanInputFormat class is called to divide the data into blocks. The number of divided data blocks is determined based on the number of available nodes in the cluster, and the blocks are stored in the InputSplit class.
[0145] (Step 2) After receiving the task, the Map Worker uses the MLTraceScanRecordReader class to read the header data of each data block by random sampling at intervals. It reads the header data of one track at a time and extracts the selected keyword value in the track header and stores it in the key / value pair.
[0146] (Step 3) Map Worker sends each key / value pair to a unique Reduce task through the partitioning function GetPartition(const std::string&key,const std::string&value,int numPartitions) of the Partitioner class.
[0147] (Step 4) After the Reduce Worker receives all the key / value pairs sent by the Map Worker, it uses the Compare(const std::string&key1,const std::string&key2) function of the MLTraceScanComparator class to sort the received index item sample data.
[0148] (Step 5) After the Reduce part is completed, the number of different values of each keyword of the ordered index sample is counted, and the index item samples are ordered and distributed to each node as evenly as possible to generate a keyword-node allocation table.
[0149] (Step 6) An MLP model is constructed based on the Libtorch framework, and the keyword value and node number in the keyword-node assignment table are respectively passed into the model as the input and output of the model for training.
[0150] (Step 7) Finally, the trained model and parameter information are stored in non-volatile storage.
[0151] Data sorting based on MapReduce programming model:
[0152] (Step 1) In the Map phase, the MLTraceSortInputFormat class is called to divide the data into blocks. The number of divided data blocks is determined based on the number of available nodes in the cluster, and the blocks are stored in the InputSplit class.
[0153] (Step 2) After receiving the task, the Map Worker uses the MLTraceSortRecordReader class to read the header data of each data block, extract the selected keyword value in each header, and store it together with the current track number in the key / value pair.
[0154] (Step 3) Map Worker reads the node allocation model through the partitioning function GetPartition(const std::string&key,const std::string&value,int numPartitions) of the MLTraceSortPartitioner class, inputs the keyword value of each index item into the model, obtains the corresponding node number, and sends each key / value pair to the corresponding Reduce task.
[0155] (Step 4) After the Reduce Worker receives the key / value pair sent by the Map Worker, it uses the Compare(const std::string&key1,const std::string&key2) function of the MLTraceSortComparator class to sort the received index data, and finally uses the MLTraceSortRecordWriter class to write the index data and keyword information into non-volatile storage.
[0156] Learning index construction based on MapReduce programming model:
[0157] (Step 1) In the Map phase, call the MakeMLIndexInputFormat class to divide the data into blocks, determine the number of data blocks based on the number of index files, and store them in the InputSplit class.
[0158] (Step 2) After receiving the task, the Map Worker uses the MakeMLIndexRecordReader class to read the index data of each data block. Each time a fixed number of index items are read, they are counted as a training set for the underlying model. <std::string>&input_index_data,intmodel_id,double time_max,std::string save_path) function generates an underlying model for training and saving. After all leaf nodes are created, use MakeIndexNodeModel(std::vector <std::vector <std::string>>&index_data_strs,double time_max,std::string save_path) function completes the generation, training and saving of non-bottom-level models of learning indexes from the bottom up.
[0159] (Step 3) Map Worker sends each key / value pair to a unique Reduce task (before the task starts, the number of Reduce tasks has been set to 1 through the SetNumReduceTasks(int num) function).
[0160] Learning index query:
[0161] (Step 1) Acquire information of the current index through a given index path, and read the first keyword information file and the top-level keyword-node allocation model of the learning index.
[0162] (Step 2) By search_index(vector <bakeyvalue>&vFromKey,vector <bakeyvalue>&vToKey,std::vector<IndexData_v> &keyDatas) function, starting from the top-level model, searches for the given start and end ranges of each keyword, and stores all eligible index items in the keyDatas array.
[0163] (Step 3) Summarize all index item results, continue to filter other attributes (such as grouping, tolerance, etc.), and store the index items that meet all conditions in the BATraceIndexs class.
[0164] In some embodiments, please refer to Fig.10 As shown, the present invention also provides a seismic data query device 200, the device comprising:
[0165] A sampling learning module 201 is configured to perform data sampling and data distributed learning on seismic data based on a MapReduce framework and a machine learning model to generate a node allocation model;
[0166] A sorting module 202 is configured to perform distributed segment sorting on the seismic data based on the MapReduce framework and the node allocation model to generate an ordered index file;
[0167] A construction module 203 is configured to train a learning index including an underlying model and a non-underlying model based on a MapReduce framework and the ordered index file, and combine the learning index with the node allocation model to construct a learning index structure;
[0168] The query module 204 is configured to obtain keywords to be queried and input them into the learning index structure to obtain query results.
[0169] A seismic data query device of this embodiment has the following beneficial technical effects: the learning index structure can learn and store the distribution pattern of data with a small amount of space, significantly reducing the storage space of the index structure. It uses the index structure as a black box and regards the index structure as a model purely from the perspective of input and output data. It predicts its position in the ordered data body by fitting the cumulative distribution function of the input data, and uses a machine learning model to learn and predict the distribution pattern of seismic data, thereby realizing the indexing function. The learning index can be reused to a certain extent, effectively reducing the index construction time and improving the efficiency of index construction.
[0170] It should be noted that the specific definition of the seismic data query device can be found in the definition of the seismic data query method above, which will not be repeated here. Each module in the above seismic data query device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0171] According to another aspect of the present invention, an electronic device is provided. The electronic device may be a server. Fig.11 As shown. The electronic device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above-mentioned seismic data query method is implemented.
[0172] According to another aspect of the present invention, a computer readable storage medium is provided. Fig.12 As shown, a computer program is stored thereon, and when the computer program is executed by a processor, the above-mentioned seismic data query method is implemented.
[0173] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0174] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0175] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.< / bakeyvalue> < / bakeyvalue> < / std::string> < / std::string>
Claims
1. A seismic data query method, characterized in that: The method comprises: Based on the MapReduce framework and machine learning model, data sampling and data distributed learning are performed on seismic data to generate a node allocation model; Based on the MapReduce framework and the node allocation model, the seismic data is sorted in distributed segments to generate an ordered index file; Based on the MapReduce framework, the ordered index file trains a learning index including an underlying model and a non-underlying model, and combines the learning index with the node allocation model to construct a learning index structure; The keywords to be queried are obtained and input into the learning index structure to obtain query results.
2. The seismic data query method according to claim 1, characterized in that: The step of performing data sampling and data distributed learning on seismic data based on the MapReduce framework and the machine learning model to generate a node allocation model includes: Set the sampling scale and set the number of Reducers to 1, and use the MapReduce framework to evenly distribute the sampling scale to each node; In the Map stage, each node reads seismic traces at intervals according to the sampling scale, extracts the keyword information containing the selected keyword value in each seismic trace, and ensures that the number of seismic traces sampled and read meets the sampling scale; In the Reduce phase, the read keyword information is integrated, and the sampling results are used as the overall data scanning results to generate the allocation table of the sorting data range of each node; Based on the allocation table, the keyword value is used as input and the node number specified for the value is used as output. After normalization, it is passed into the machine learning model for training according to the proportion of the number of keyword values, and the cumulative distribution function of the keyword is fitted and saved to generate a node allocation model.
3. The seismic data query method according to claim 1, characterized in that: The step of performing distributed segment sorting on the seismic data based on the MapReduce framework and the node allocation model to generate an ordered index file comprises: In the Map phase, the header data is evenly divided into multiple segments. Each Map task is responsible for reading a segment of data. These Map tasks are then evenly distributed to the Map Workers of each node in the cluster. After receiving the Map task, the Map Worker starts processing the corresponding trace header data segment, reading all trace headers, extracting the selected keyword data and the trace number in each trace in sequence as index items, and storing them using key / value pairs; The Map Worker obtains the node number corresponding to each index item by calling the node allocation model, and then sends each key / value to the corresponding Reduce Worker for sorting; After the Reduce Worker receives all the key / value pairs sent by the Map Worker, it sorts the index data stored in the key / value pairs according to the keyword order, and finally stores the sorted index data into the file corresponding to the Reduce Worker, and writes the statistics of each keyword information into the corresponding keyword information file.
4. The seismic data query method according to claim 1, characterized in that: The step of training a learning index including an underlying model and a non-underlying model based on the MapReduce framework and the ordered index file, and combining the learning index with the node allocation model to construct a learning index structure includes: In the Map stage, call the MakeMLIndexInputFormat class to divide the data into blocks. The number of data blocks is determined based on the number of index files and stored in the InputSplit class. After receiving the task, the Map Worker uses the MakeMLIndexRecordReader class to read the index data of each data block. Every time a fixed number of index items are read, they are counted as a training set of an underlying model in the learning index. The MakeIndexLeafModel function is used to generate an underlying model in the learning index for training and saving. After all leaf nodes are created, use the MakeIndexNodeModel function to complete the generation, training, and saving of the non-bottom-level model of the learning index from bottom to top; The node allocation model is used as the root node model of the learning index to obtain a learning index structure, wherein the learning index structure starts from the root node model when querying, each layer of models selects the next layer of models, and the last layer of models outputs the approximate position of the query value in the overall index data.
5. The seismic data query method according to claim 4, characterized in that: The step of obtaining the keyword to be queried and inputting it into the learning index structure to obtain the query result includes: Obtain the keyword to be queried input by the user, and input the keyword value into the root node model of the learning index structure, obtain the lower model number where the keyword is located, and search downwards in this way until the bottom model; The underlying model is used to predict the offset of the keyword in the index data file, and the index file is read according to the predicted offset. If the index item value read is inaccurate, you can determine whether to search forward or backward item by item by comparing the current item with the search value until the correct value is matched.
6. The seismic data query method according to claim 5, characterized in that: The method further comprises: After the query, the query conditions and the actual offset are passed into the model again as training data for training to update the learning index structure.
7. The seismic data query method according to claim 1, characterized in that: The machine learning model is a multi-layer perceptron model.
8. A seismic data query device, characterized in that: The device comprises: A sampling learning module, configured to perform data sampling and data distributed learning on seismic data based on a MapReduce framework and a machine learning model to generate a node allocation model; A sorting module configured to perform distributed segment sorting of seismic data based on a MapReduce framework and the node allocation model to generate an ordered index file; A construction module configured to train a learning index including an underlying model and a non-underlying model based on a MapReduce framework and the ordered index file, and to combine the learning index with the node allocation model to construct a learning index structure; The query module is configured to obtain keywords to be queried and input them into the learning index structure to obtain query results.
9. An electronic device, characterized in that: include: at least one processor; as well as A memory storing a computer program executable in the processor, wherein the processor executes the method according to any one of claims 1 to 7 when executing the program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is performed.
Citation Information
Patent Citations
Distributed file management system based on seismic data processing
CN102880658A
Data transmission method and system based on seismic data processing
CN103412739A
Indexing method and system for large-scale data
CN103870456A
Device and method for storing seismic data cube and reading seismic channels
CN104317536A
Prestack migration parallel processing method and device based on MapReduce
CN106250101A
Cited By
Mass seismic data index construction method, index database and query method
CN119356647A
Data index construction method and device, data query method and device, medium and product
CN120743913A
A data index construction, data query method, device, medium and product
CN120743913B
Seismic data query method
CN121301643A
