Spark-based seismic data parallel granularity division method and application thereof

Abstract

The invention belongs to the field of seismic exploration data processing, and particularly relates to a Spark-based seismic data parallel granularity division method and application thereof, and the method comprises the steps: determining the window number of non-overlapping windows on trace data RDD, and dividing the non-overlapping windows; judging whether each channel of data in the non-overlapping windows is an overlapping channel or not; calculating other non-overlapping windows to which the overlapping channels belong; expanding each non-overlapping window into a complete window according to the overlapping channels, so as to convert the channel data RDD into a window RDD; the parallel granularity of the window RDD is channel data in a complete window. According to the method, the three-dimensional data volume can be accurately and clearly divided into a plurality of small overlapped three-dimensional volumes, and the parallel granularity is optimized and is smaller when parallel computing is performed by using the spark framework.

Description

Technical Field

[0001] The present invention belongs to the field of seismic exploration data processing, and in particular relates to a Spark-based parallel granularity partitioning method for seismic data and an application thereof. Background Art

[0002] RDD (Resilient Distributed Dataset) is the basic data structure of Spark, representing an immutable, partitionable set whose elements can be calculated in parallel. In seismic data processing, if the seismic data is processed one by one, the spark framework can directly convert the seismic data into a channel data RDD; if the entire channel set is processed, it can be converted into a channel set RDD. These two situations can be completed by the flatMap function in spark with a simple RDD flow. When processing post-stack data, sometimes it is necessary to process the data through sliding windows, and there are overlaps between windows, which also increases the difficulty of three-dimensional data division. Summary of the invention

[0003] The purpose of the present invention is to solve the above-mentioned problems existing in the prior art and provide a parallel granularity partitioning method for seismic data based on Spark.

[0004] The present invention is implemented by the following technical scheme: a parallel granularity partitioning method for seismic data based on Spark, comprising the following steps:

[0005] Determine the number of non-overlapping windows on the track data RDD and divide the non-overlapping windows;

[0006] Determine whether each data track in the non-overlapping window should be an overlapping track;

[0007] Calculate other non-overlapping windows to which the overlapping channels belong;

[0008] Expand each non-overlapping window to a complete window based on the overlapping channels, thereby converting the channel data RDD into a form RDD;

[0009] The parallel granularity of the form RDD is the data within a complete window.

[0010] Furthermore, the method for constructing the track data RDD includes the following steps:

[0011] The seismic data is input into a distributed file system and partitioned, each partition corresponds to multiple lines of data, and each line of data consists of multiple channels of data;

[0012] According to the data in the partition, an Inline RDD is constructed. The key-value pair corresponding to each element in the Inline RDD is represented as key1 -value 1 ,key 1 The value is InlineNum, which is the line number in the Inline direction. 1 The value is all the channel data with the same Inline number;

[0013] Convert the Inline number RDD to a channel data RDD. The key-value pair corresponding to each element in the channel data RDD is represented as key 2 -value 2 ,key 2 The value is (InlineNum, CrosslineNum), value 2 The value is key 2 The corresponding data, InlineNum represents the line number in the Inline direction, and CrosslineNum represents the line number in the Crossline direction.

[0014] Furthermore, the flatMap conversion function is used to transform each key in the Inline RDD. 1 -value 1 For each data in the data, extract the Inline number and Crossline number from the header as the key 2 The value of .

[0015] Furthermore, the number of non-overlapping windows is determined as follows:

[0016] The number of blocks in the Inline direction nInline is calculated according to the following formula:

[0017] nInline=(InlineN-OverlapInline) / (WinInline–OverlapInline)+1

[0018] Where, InlineN represents the total number of Inline channels, WinInline represents the number of Inline channels that a complete window should have, and OverlapInline represents the number of overlapping channels in the Inline direction in a single overlapping window.

[0019] The number of blocks in the Crossline direction nCrossline is calculated according to the following formula:

[0020] nCrossline=(CrossN-OverlapCrossline) / (WinCrossline–verlapCrossline)+1

[0021] In the formula, CrossN represents the total number of crossline channels, WinCrossline represents the number of crossline channels that a complete window should have, and OverlapInline represents the number of overlapping channels in the crossline direction within a single overlapping window.

[0022] The product of the number of blocks nInline and the number of blocks nCrossline is the number of non-overlapping windows.

[0023] Furthermore, the window position of the non-overlapping window is determined by the window position iInline in the Inline direction and the window position icrossline in the Crossline direction, and both iInline and icrossline are numbered starting from 0.

[0024] Furthermore, the key-value pair corresponding to each element in the form RDD is represented as key 3 -value 3 ,key 3 The value is the window number, value 3 The value is the corresponding key 3 Click on all the data in the complete window;

[0025] key 3 The following formula is used to calculate:

[0026] key 3 =iInline*nCrossline+iCrossline

[0027] Where iInline represents the window position in the Inline direction, iCrossline represents the window position in the Crossline direction, and nCrossline represents the number of blocks in the Crossline direction.

[0028] Furthermore, before determining whether it is an overlapping track, mark it first:

[0029] For each data, first mark flagInline=0, flagCrossline=0;

[0030] Then calculate its window position (iInline, iCrossline);

[0031] When iInline>=1, iCrossline>=1, determine whether the channel is an overlapping channel:

[0032] If InlineNum<=(iInline-1)*(WinInline-OverlapInline)+WinInline-1, then the Inline direction is marked as flagInline=-1;

[0033] If CrosslineNum<=(iCrossline-1)*(WinCrossline-OverlapCrossline)++WinCrossline-1, then the Crossline direction is marked as flagCrossline=-1.

[0034] Furthermore, whether it is an overlapping channel and the non-overlapping window it belongs to is determined based on the mark:

[0035] If flagInline+flagCrossline==0, then flagInline=0, flagCrossline=0, indicating that the data of this track is not an overlapping track;

[0036] Otherwise, if flagInline*flagCrossline==0, then when flagInline=0, flagCrossline=-1, the track data belongs to the overlapping area in the Crossline direction; when flagInline=-1, flagCrossline=0, the track data belongs to a certain overlapping area of ​​Inline, so the track data in the overlapping area also belongs to the window: (iInline+flagInline)*nCrssline+(iCrossline+flagCrssline);

[0037] If flagInline*flagCrossline!=0, then flagInline=-1, flagCrossline=-1, indicating that the trace belongs to the overlapping area in two directions, so the overlapping area also belongs to the window: (iInline+flagInline)*nCrssline+(iCrossline+flagCrossline), (iInline+0)*nCrossline+(iCrossline+flagCrssline), (iInline+flagInline)*nCrossline+(iCrossline+0).

[0038] The present invention also provides an application of a Spark-based parallel granularity partitioning method for seismic data, characterized in that it is used to partition pre-stack seismic data, post-stack seismic data, three-dimensional seismic data or two-dimensional seismic data.

[0039] Furthermore, after the data processing is completed according to the parallel granularity, if it is necessary to output according to the Inline number RDD, the window RDD is first converted into a data RDD with (InlineNum, CrosslineNum) as the key value, and the data is deduplicated, and then converted into an RDD with Inline as the key value.

[0040] Compared with the prior art, the beneficial effects of the present invention include:

[0041] The present invention can accurately and clearly divide a three-dimensional data volume into a plurality of small overlapping three-dimensional volumes, thereby ensuring that when the spark framework is used for parallel computing, the parallel granularity is optimized and made smaller. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the structure after the seismic data is input into the distributed system;

[0043] Figure 2 This is a schematic diagram of arranging channel data with the same Inline number according to the Crossline number;

[0044] Figure 3 It is a schematic diagram of dividing the 3D seismic data into small 3D volumes;

[0045] Figure 4 Schematic diagram of different windows;

[0046] Figure 5 It is a relationship diagram between window position and window number;

[0047] Figure 6 Schematic diagram marking the overlapping area. DETAILED DESCRIPTION

[0048] A parallel granularity partitioning method for seismic data based on Spark, comprising the following steps:

[0049] Determine the number of non-overlapping windows on the track data RDD and divide the non-overlapping windows;

[0050] Determine whether each data track in the non-overlapping window should be an overlapping track;

[0051] Calculate other non-overlapping windows to which the overlapping channels belong;

[0052] Expand each non-overlapping window to a complete window based on the overlapping channels, thereby converting the channel data RDD into a form RDD;

[0053] The parallel granularity of the form RDD is the data within a complete window.

[0054] The present invention can be used in the scenario where data needs to be re-blocked during seismic data processing, and can be used for both pre-stack seismic data and post-stack seismic data; and can be used for both three-dimensional seismic data and two-dimensional seismic data.

[0055] This specific implementation manner is described by taking three-dimensional post-stack seismic data as an example.

[0056] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0057] This specific implementation shows the method of dividing the Inline and Crossline directions of post-stack seismic data. The time direction is assumed to be a complete data without division. If the time direction needs to be further divided, the same marking method can be used to mark the overlapping data in the time direction.

[0058] First, refer to Figure 1 As shown in the figure, the seismic trace data is input into the distributed file system, and the input data is partitioned. Each partition corresponds to multiple lines of data. The format of post-stack seismic data is stored by trace. Each line consists of multiple traces of data, and each three-dimensional data volume consists of multiple lines. The data in the same Inline line is in the same partition.

[0059] refer to Figure 2 As shown in the figure, a data is determined by (Inline, Crossline). The number of tracks in the Inline direction is the number of Inline lines. Track data with the same Inline number are arranged according to the Crossline number to form data in an Inline line.

[0060] refer to Figure 3 As shown, the 3D seismic data is divided into small 3D volumes of size WinInline*WinCrossline, and the lengths of the single-side overlapping windows are OverlapInline and OverlapCrossline respectively.

[0061] Example 1

[0062] The method for constructing the track data RDD includes the following steps:

[0063] The seismic data is input into a distributed file system and partitioned, each partition corresponds to multiple lines of data, and each line of data consists of multiple channels of data;

[0064] According to the data in the partition, an Inline RDD is constructed. The key-value pair corresponding to each element in the Inline RDD is represented as key 1 -value 1 ,key 1The value is InlineNum, which is the line number in the Inline direction. 1 The value is all the channel data with the same Inline number;

[0065] Convert the Inline number RDD to a channel data RDD. The key-value pair corresponding to each element in the channel data RDD is represented as key 2 -value 2 ,key 2 The value is (InlineNum, CrosslineNum), value 2 The value is key 2 The corresponding data, InlineNum represents the line number in the Inline direction, and CrosslineNum represents the line number in the Crossline direction.

[0066] (1) Construct an RDD whose key value is the Inline number and whose value is all the channel data corresponding to the Inline number. The specific method is: extract the Inline number and channel data of each channel data, and find all the channel data corresponding to the same Inline number. Each element corresponds to a key-value pair.

[0067] (2) Convert the Inline number RDD to the track data RDD. The key value of the track data RDD is the composite key (Inline, Crossline), and the value is a track data (track header + track data). Use the flatMap conversion function to extract the Inline number and Crossline number from the track header for each key-value pair (i.e., each line) in the Inline number RDD. Use them as the key value, and the corresponding track data is the value.

[0068] Example 2

[0069] The number of non-overlapping windows is determined as follows:

[0070] The number of blocks in the Inline direction nInline is calculated according to the following formula:

[0071] nInline=(InlineN-OverlapInline) / (WinInline–OverlapInline)+1

[0072] Where, InlineN represents the total number of Inline channels, WinInline represents the number of Inline channels that a complete window should have, and OverlapInline represents the number of overlapping channels in the Inline direction in a single overlapping window.

[0073] The number of blocks in the Crossline direction nCrossline is calculated according to the following formula:

[0074] nCrossline=(CrossN-OverlapCrossline) / (WinCrossline–verlapCrossline)+1

[0075] In the formula, CrossN represents the total number of crossline channels, WinCrossline represents the number of crossline channels that a complete window should have, and OverlapInline represents the number of overlapping channels in the crossline direction within a single overlapping window.

[0076] The product of the number of blocks nInline and the number of blocks nCrossline is the number of non-overlapping windows.

[0077] The specific steps are:

[0078] refer to Figure 4 As shown in , the number of blocks in the Inline direction nInline and the number of blocks in the Crossline direction nCrossline are calculated based on the non-overlapping situation between the windows. Figure 4 As shown, nInline=3, nCrossline=5.

[0079] The specific calculation process is as follows: There are 8 lines in the Inline direction and 9 lines in the Crossline direction (such as Figure 4 As shown in the figure). In each block window, there are 5 data tracks in the Inline direction (i.e. WinInline=5) and 3 data tracks in the Crossline direction (WinCrossline=3). The overlap between windows is as follows: the number of overlapping tracks in the Inline direction is 2 (OverlapInline) and the number of overlapping tracks in the Crossline direction is 1 (OverlapCrossline). Then nInline=(8-2) / (5-2)+1=3, nCrossline=(9-1) / (3-1)+1=5. A total of 3*5=15 non-overlapping windows are divided, and the window number starts from 0.

[0080] Example 3

[0081] Before determining whether it is an overlapping track, mark it first:

[0082] First, calculate the window position iInline (iInline = InlineNum / (WinInline-OverlapInline)) in the Inline direction and icrossline (iCrossline = CrosslineNum / (WinCrossline-OverlapCrossline)) in the Crossline direction when there is no overlap (iInline, iCrossline are numbered starting from 0). The data belongs to the window (iInline*nCrossline+iCrossline) (key 3 value). Figure 5 In window 10, the key value of the data in this window is 10. After the data is divided into windows, the window position in the Inline direction is iInline=1, and the Crossline window position is iCrossline=2.

[0083] Figure 5 The window on the right is the narrowest to indicate that the last few traces of data may not fit into a normal window size.

[0084] The final divided windows will overlap. The introduction of non-overlapping windows is just to calculate the number of windows based on the non-overlapping parts and facilitate the division of data, such as Figure 4 As shown, the middle is a complete window, and the left is the data after removing the overlapping part)

[0085] Given the number of windows, we can calculate iInline and iCrossline. iInline = key / nCrossline, iCrossline = key-nCrossline*iInline.

[0086] Then determine whether the data of the channel is an overlapping channel. If it is an overlapping channel, determine which window the overlapping channel belongs to (for example, in the final divided window, Figure 6 The overlapping area ① belongs to both window 32 and window 33. Figure 6 ② in the middle belongs to windows 25, 26, 33, and 34 at the same time.)

[0087] Set up flags: the overlapping area in the Inline direction is set to flagInline=-1, and the overlapping area in the Crossline direction is set to flagCrosseline=-1. If the channel data is not in the overlapping area, flagInline=0, flagCrossline=0.

[0088] The overlapping area is determined by calculation. For each data, first mark flagInline=0, flagCrossline=0, and then calculate its location (iInline, iCrossline). When iInline>=1, iCrossline>=1, determine whether the data is an overlapping data:

[0089] If InlineNum<=(iInline-1)*(WinInline-OverlapInline)+WinInline-1, then flagInline=-1. If CrosslineNum<=(iCrossline-1)*(WinCrossline-OverlapCrossline)+WinCrossline-1, then flagCrossline=-1.

[0090] Example 4

[0091] According to the mark, it is determined whether it is an overlapping track and the non-overlapping window it belongs to, and the track data RDD is converted into a three-dimensional window RDD.

[0092] The following judgment is made: if flagInline+flagCrossline==0, then flagInline=0, flagCrossline=0, indicating that the data of this track is not an overlapping track;

[0093] Otherwise, if flagInline*flagCrossline==0, then when flagInline=0, flagCrossline=-1, the data of this track belongs to the overlapping area in the Crossline direction. When flagInline=-1, flagCrossline=0, the data of this track belongs to a certain overlapping area of ​​Inline, so the data of this overlapping area also belongs to the window: (iInline+flagInline)*nCrssline+(iCrossline+flagCrssline);

[0094] If flagInline*flagCrossline!=0, then flagInline=-1, flagCrossline=-1, indicating that the trace belongs to the overlapping area in two directions, so the overlapping area also belongs to the window: (iInline+flagInline)*nCrssline+(iCrossline+flagCrossline), (iInline+0)*nCrossline+(iCrossline+flagCrssline), (iInline+flagInline)*nCrossline+(iCrossline+0).

[0095] Using flatMap to construct key 3 RDD whose value is window number (window number), value 3 The value is the data in a window. The window does not refer to the time window. It does not mean cutting a data RDD into multiple three-dimensional form RDDs. It means recombining the data RDD into a three-dimensional form RDD. The window is composed of multiple adjacent data. For example, the data in the red box in the figure below is in one window. Figure 2 Understand that in one window number, there are multiple data from different inlines and crosslines

[0096] Thus, all the data of each overlapping window are obtained. The key 3 The value is the number of windows, value 3 The value is the RDD of all the data in a window. At this time, the data can be processed in parallel, and the parallel granularity is the data of a complete window with overlapping data.

[0097] Each complete window contains overlapping and non-overlapping parts. The overlapping and non-overlapping parts are only for window division. After the window division is completed, the data in the window is processed in the same way.

[0098] Example 5

[0099] After the data processing is completed according to the parallel granularity, if it is necessary to output according to the Inline number RDD, the window RDD is first converted into a channel data RDD with (InlineNum, CrosslineNum) as the key value, and the data is deduplicated, and then converted into an RDD with Inline as the key value. Finally, the RDD is directly output or passed to the next module for processing.

[0100] Example 6

[0101] (1) Input the seismic data into a distributed file system and partition the input data. Each partition corresponds to multiple lines of data.

[0102] (2) Construct an RDD whose key value is the Inline number and whose value is all the channel data corresponding to the Inline number.

[0103] (3) Convert the Inline number RDD into a channel data RDD. The key value of the channel data RDD is the composite key (Inline, Crossline), and the value is a channel data (channel header + channel data).

[0104] (4) Convert the track data RDD to a window RDD. Use flatMap to construct an RDD whose key value is the number of windows and whose value is the track data in a window. First, calculate the total number of windows that need to be divided, and for each track data, calculate which window each track data belongs to when there is no overlap.

[0105] (5) Then determine whether the data is an overlapping track and set a mark. Finally, divide the overlapping track into another window where it overlaps according to the mark. At this time, the data can be processed in parallel, and the parallel granularity is the data of one window.

[0106] (6) After data processing is completed, if it is necessary to output according to the Inline number RDD, the window RDD is first converted into a channel data RDD with (Inline, Crossline) as the key value, and the data is deduplicated. Then it is converted into an RDD with the Inline number as the key value.

[0107] The above technical scheme is only a specific implementation method of the present invention. For those skilled in the art, it is easy to make various types of improvements or modifications based on the principles disclosed in the present invention, and it is not limited to the technical scheme described in the above specific embodiments of the present invention. Therefore, the above description is only preferred and does not have a restrictive meaning.

Claims

1. A parallel granularity partitioning method for seismic data based on Spark, characterized in that: The following steps are involved: Determine the number of non-overlapping windows on the track data RDD and divide the non-overlapping windows; Determine whether each data track in the non-overlapping window should be an overlapping track; Calculate other non-overlapping windows to which the overlapping channels belong; Expand each non-overlapping window to a complete window based on the overlapping channels, thereby converting the channel data RDD into a form RDD; The parallel granularity of the form RDD is the data within a complete window.

2. The method for parallel granularity partitioning of seismic data based on Spark according to claim 1, characterized in that: The method for constructing the track data RDD includes the following steps: The seismic data is input into a distributed file system and partitioned, each partition corresponds to multiple lines of data, and each line of data consists of multiple channels of data; According to the data in the partition, an Inline number RDD is constructed. The key-value pair corresponding to each element in the Inline number RDD is expressed as key1-value1. The key1 value is InlineNum, that is, the line number in the Inline direction, and the value1 value is all the track data with the same Inline number. Convert the Inline number RDD to the track data RDD. The key-value pair corresponding to each element in the track data RDD is expressed as key2-value2. The value of key2 is (InlineNum, CrosslineNum), and the value of value2 is a track data corresponding to key2. InlineNum represents the line number in the Inline direction, and CrosslineNum represents the line number in the Crossline direction.

3. According to the Spark-based parallel granularity partitioning method for seismic data in claim 2, the flatMap conversion function is used to extract the Inline number and Crossline number from the header as the value of key2 for each data in each key1-value1 in the Inline number RDD.

4. The method for parallel granularity partitioning of seismic data based on Spark according to claim 1, characterized in that: The number of non-overlapping windows is determined as follows: The number of blocks in the Inline direction nInline is calculated according to the following formula: nInline=(InlineN-OverlapInline) / (WinInline–OverlapInline)+1 Where, InlineN represents the total number of Inline channels, WinInline represents the number of Inline channels that a complete window should have, and OverlapInline represents the number of overlapping channels in the Inline direction in a single overlapping window. The number of blocks in the Crossline direction nCrossline is calculated according to the following formula: nCrossline=(CrossN-OverlapCrossline) / (WinCrossline–verlapCrossline)+1 In the formula, CrossN represents the total number of crossline channels, WinCrossline represents the number of crossline channels that a complete window should have, and OverlapInline represents the number of overlapping channels in the crossline direction within a single overlapping window. The product of the number of blocks nInline and the number of blocks nCrossline is the number of non-overlapping windows.

5. The method for parallel granularity partitioning of seismic data based on Spark according to claim 4, characterized in that: The window position of the non-overlapping window is determined by the window position iInline in the Inline direction and the window position icrossline in the Crosslin direction, and both iInline and icrossline are numbered starting from 0.

6. The method for parallel granularity partitioning of seismic data based on Spark according to claim 5, characterized in that: The key-value pair corresponding to each element in the form RDD is expressed as key3-value3, where key3 is the window number and value3 is the total data in the complete window corresponding to key3; The following formula is used to calculate key3: key3=iInline*nCrossline+iCrossline Where iInline represents the window position in the Inline direction, iCrossline represents the window position in the Crossline direction, and nCrossline represents the number of blocks in the Crossline direction.

7. The method for parallel granularity partitioning of seismic data based on Spark according to claim 1, characterized in that: Before determining whether it is an overlapping track, mark it first: For each data, first mark flagInline=0, flagCrossline=0; Then calculate its window position (iInline, iCrossline); When iInline>=1, iCrossline>=1, determine whether the channel is an overlapping channel: If InlineNum<=(iInline-1)*(WinInline-OverlapInline)+WinInline-1, then the Inline direction is marked as flagInline=-1; If CrosslineNum<=(iCrossline-1)*(WinCrossline-OverlapCrossline)++WinCrossline-1, then the Crossline direction is marked as flagCrossline=-1.

8. The method for parallel granularity partitioning of seismic data based on Spark according to claim 7, characterized in that: Determine whether it is an overlapping channel and the non-overlapping window it belongs to based on the mark: If flagInline+flagCrossline==0, then flagInline=0, flagCrossline=0, indicating that the data of this track is not an overlapping track; Otherwise, if flagInline*flagCrossline==0, then when flagInline=0, flagCrossline=-1, the data of this track belongs to the overlapping area in the Crossline direction; when flagInline=-1, flagCrossline=0, the data of this track belongs to a certain overlapping area of ​​Inline, so the data of this overlapping area also belongs to the window: (iInline+flagInline)*nCrssline+(iCrossline+flagCrssline); If flagInline*flagCrossline!=0, then flagInline=-1, flagCrossline=-1, indicating that the trace belongs to the overlapping area in two directions, so the overlapping area also belongs to the window: (iInline+flagInline)*nCrssline+(iCrossline+flagCrossline), (iInline+0)*nCrossline+(iCrossline+flagCrssline), (iInline+flagInline)*nCrossline+(iCrossline+0).

9. An application of the parallel granularity partitioning method for seismic data based on Spark according to claims 1 to 8, characterized in that: It is used to block pre-stack seismic data, post-stack seismic data, 3D seismic data or 2D seismic data.

10. The application of the Spark-based parallel granularity partitioning method for seismic data according to claim 9, characterized in that: After the data processing is completed according to the parallel granularity, if it is necessary to output according to the Inline number RDD, the window RDD is first converted into a channel data RDD with (InlineNum, CrosslineNum) as the key value, and the data is deduplicated, and then converted into an RDD with Inline as the key value.

Images