SEGY disc data parallel loading device and method

Abstract

This invention discloses a parallel loading device for SEGY disk data, comprising a parameter parsing module, a trace information statistics module, a parallel partitioning strategy module, a data positioning module, and a data reading module. The parameter parsing module is connected to the trace information statistics module, the trace information statistics module is connected to the parallel partitioning strategy module, the parallel partitioning strategy module is connected to the data positioning module, and the data positioning module is connected to the data reading module. This invention also discloses a parallel loading method for SEGY disk data, based on the aforementioned parallel loading device. This invention reduces human intervention, automatically partitions the scope of the data to be loaded, adapts to the loading of massive amounts of SEGY data, and improves the loading efficiency of seismic data processing projects. This invention is applicable to improving the input efficiency of massive seismic data by using a parallel method to load SEGY disk data.

Description

Technical Field

[0001] This invention belongs to the field of geophysical exploration technology and relates to seismic data platform technology, specifically a SEGY disk data parallel loading device and method. Background Technology

[0002] With the continuous advancement of exploration technology and the emergence of new acquisition technologies, the amount of acquired data has grown rapidly. In particular, the acquisition method of continuous recording by wireless nodes has ultimately led to an ever-increasing amount of data to be loaded into seismic data processing and production projects.

[0003] Currently, no similar processing software in the industry can implement parallel loading of SEGY data within the workflow. When loading multiple SEGY data sets, the input SEGY files must be manually divided into multiple groups based on the number of output data volumes, which is inefficient. It is powerless to load single-file SEGY data with a large number of volumes. For example, the commonly used data processing system GeoEast does not provide effective parallel loading technology for SEGY data. Therefore, to improve the efficiency of processing production projects and simplify the seismic data loading process for personnel, it is imperative to provide an effective method and device for loading SEGY seismic data. Summary of the Invention

[0004] To address the aforementioned shortcomings in the existing technology, this invention aims to provide a parallel loading device and method for SEGY disk data, thereby improving the efficiency of massive seismic data input by using a parallel method to load SEGY disk data.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A parallel loading device for SEGY disk data includes a parameter parsing module, a trace information statistics module, a parallel partitioning strategy module, a data positioning module, and a data reading module; the parameter parsing module is connected to the trace information statistics module, the trace information statistics module is connected to the parallel partitioning strategy module, the parallel partitioning strategy module is connected to the data positioning module, and the data positioning module is connected to the data reading module.

[0007] The present invention also provides a SEGY disk data parallel loading method, which is based on the above-mentioned SEGY disk data parallel loading device, and the method includes the following steps performed in sequence:

[0008] S1. Parse parallel loading parameters

[0009] The filenames of the SEGY disk files collected in the field are placed in the SEGY file container vFiles. The user-input SEGY file parameters and parallel loading parameters are parsed to determine the parallel loading mode used by the job.

[0010] The SEGY file parameters include the total number of subjobs (Jobs) and the job number (JobId) for each subjob;

[0011] The parallel loading parameters include the total number of subjobs (Jobs) and the sequence number (JobId) of each subjob;

[0012] The parallel loading mode includes loading at the file level and loading at the data channel level;

[0013] S2. Data Channel Information Statistics

[0014] S21. Traverse each element in the SEGY file container vFiles, open the corresponding SEGY file, read the C-Card and file header block, and extract the sampling interval, number of samples and track length;

[0015] The number of sample points is used to calculate the byte length of each sample point;

[0016] S22. Calculate the byte length of any seismic data stream;

[0017] S23. Obtain the size of the SEGY file in bytes, and subtract the length of the C-Card and the length of the file header block from the size of the file in bytes to get the length of the data track;

[0018] S24. Calculate the number of channels in each SEGY file;

[0019] S25. Place the number of channels for each SEGY file into the data channel container vTraces;

[0020] S3. Develop a data range delineation strategy

[0021] Determine the loading range of SEGY data for each subjob with the job number JobId. For SEGY data loaded at the file level, calculate the range of SEGY files for the subjob with the job number JobId; for SEGY data loaded at the data channel level, calculate the range of data channels for the subjob with the job number JobId.

[0022] S4. Data Positioning

[0023] Open the first file in the SEGY file container vFiles for data analysis, allocate a data channel buffer, and process SEGY data loaded at the file granularity and SEGY data loaded at the data channel granularity respectively.

[0024] The allocation is implemented through a C++ program, requesting a contiguous block of memory to store data.

[0025] S5. Input SEGY data according to the normal procedure and read the data.

[0026] As a limitation, step S1, determining the parallel loading mode used by the job, includes the following steps performed sequentially:

[0027] S11. Parse the input SEGY disk file parameters

[0028] The filenames of the SEGY disk files collected in the field are placed in the SEGY file container vFiles, and the number of SEGY disk files is counted.

[0029] SEGY disk files collected in the field are generally one file per shot or one file per receiving point, with input SEGY data reaching tens of thousands; while the result data is also a large data file, and these file names are placed in the SEGY file container vFiles;

[0030] S12. Parse the total number of subjobs and the sequence number of each subjob.

[0031] Set the starting value of the subjob sequence number to 1, and count the total number of subjobs (Jobs) and the sequence number (JobId) of each subjob;

[0032] If the number of files in the SEGY disk file is greater than or equal to the total number of jobs, load the SEGY data for each subjob at the file level; if the number of files in the SEGY disk file is less than the total number of jobs, load the SEGY data for each subjob at the data track level.

[0033] As another limitation, in step S22, the byte length of the seismic data is calculated as the sum of the length of the trace header (240 bytes) and the byte length of each trace sample point.

[0034] As a third limitation, in step S3, for SEGY data loaded at the file level, the following steps are performed:

[0035] S311. Calculate the average number of jobs fNum that each subjob is responsible for, which is the number of SEGY disk files in the filename container vFiles / the total number of subjobs Jobs;

[0036] S312. Calculate the starting file number CurSeq of the subjob, i.e., the value of fNum*(JobId-1), and round it to the nearest whole number.

[0037] S313. Calculate the starting file number NextSeq of the next subjob, that is, round the value of fNum*JobId;

[0038] S314. Calculate the number of files handled by the subjob, CurNum, which is NextSeq-CurSeq.

[0039] As a fourth limitation, in step S3, the SEGY data loaded at the data channel granularity will undergo the following steps:

[0040] S321. Statistical data: TotalTraces, the total number of tracks in the track container vTraces;

[0041] S322. Calculate the average number of data tracks nNum that each subjob is responsible for loading, which is TotalTraces / Jobs;

[0042] S323. Calculate the starting channel number CurSeq of the current subjob JobId, where the starting channel number is nNum*(nJobId-1);

[0043] S324. If JobId is not the last subjob, the number of tracks CurNum that the subjob is responsible for is nNum; otherwise, the number of tracks CurNum that the subjob is responsible for is TotalTraces-CurSeq-1.

[0044] S325. Calculate the SEGY file number of the data channel responsible for the subjob, as well as its starting channel number (TraceSeq) and ending channel number in the file, based on the SEGY data channel number recorded in the data channel container vTraces.

[0045] As a fifth limitation, in step S4, the SEGY data loaded at the file level will undergo the following steps:

[0046] S411. Remove elements from the file container vFiles that are not within the scope of this subjob, and also remove the keyword control selection range of the corresponding SEGY file;

[0047] If the starting file number of the subjob is 0, execute S412; otherwise, execute S413.

[0048] The file number of the subjob is set by the user, and the starting file number is 0;

[0049] S412. If the SEGY data contains a C-Card and needs to be output, then output the C-Card and the data positioning ends;

[0050] S413. Open the second file in the file container vFiles. Data location is complete.

[0051] As a sixth limitation, in step S4, the SEGY data loaded at the data channel granularity will undergo the following steps:

[0052] S421. If the starting file number of the subjob is not 0, execute S422; otherwise, execute S423.

[0053] S422. Delete elements that are not within the scope of this subjob from the file container vFiles, and delete the keyword control selection range of the corresponding SEGY file, then open the next file for positioning;

[0054] S423. Locate the data track in the current file, specifically including the following steps:

[0055] S4231. If the starting trace number TraceSeq in the currently located file is 0, the data location ends;

[0056] S4232. Calculate the byte position in the location file, and use the file's starting track number * data track length to locate it. If the user has not set a track set keyword, the location ends; otherwise, execute S4233.

[0057] The key to the set is a parameter input by the user. Its purpose is to ensure that data belonging to the same set is loaded in only one subjob.

[0058] S4233. Read each data point and obtain the CurValue corresponding to the key of the data set;

[0059] S4234. Read the next data, obtain the trace head value PreValue corresponding to the trace key. If PreValue is not equal to CurValue, it means that the data trace being located is a new trace set, and the location ends; otherwise, execute S4235.

[0060] S4235. Read the next trace, obtain the trace header value NextValue corresponding to the trace key. When the SEGY file is read to the end or the number of traces read exceeds CurNum, the subjob ends; when NextValue is not equal to CurValue, the positioning ends and backtracks by one trace.

[0061] When NextValue is not equal to CurValue, it means that the set of Tao containing NextValue is not the same set of Tao containing CurValue.

[0062] The reason for the rollback is that the data track NextValue has just been read, and the position marker in the file has reached the position of the next track, but this track is needed, so it is necessary to roll back to the beginning of the NextValue track.

[0063] The concept of this invention is as follows: Statistically analyze the distribution of traces within each SEGY file to be loaded. Based on the quantity of output seismic data and the statistical information, determine whether to load at the file level or at the trace level. During the job preparation phase, remove input data and data control ranges irrelevant to the current job. If loading is done at the trace level, the correct trace location must be identified. For trace-based loading, the integrity of the trace gathers must be ensured in independent job runs, and the uniqueness of the trace gathers must be guaranteed across all jobs.

[0064] By adopting the above-described technical solution, the beneficial effects achieved by this invention compared to the prior art are as follows:

[0065] ① The parallel loading method for SEGY disk data provided by this invention has different designs for loading a single large SEGY data volume and multiple small SEGY data volumes. By pre-analyzing the information of all SEGY data to be loaded, the data trace range responsible for each job is reasonably divided to ensure that data traces belonging to the same trace set are loaded in only one job. This method utilizes the hardware resources of cluster nodes through a multi-node, multi-job approach, simplifies the SEGY seismic data loading process for processing personnel, and improves the work efficiency of seismic data processing projects.

[0066] ② The parallel loading method for SEGY disk data provided by this invention uses automatic SEGY data partitioning technology, combined with a multi-node, multi-job approach, to make full use of cluster node hardware resources, simplify the SEGY seismic data loading process for processing personnel, and improve the work efficiency of seismic data processing projects.

[0067] ③ The parallel loading method for SEGY disk data designed in this invention reduces the intervention of processing personnel, automatically divides the scope of data to be loaded, adapts to the loading of massive amounts of SEGY data, and improves the loading efficiency of seismic data processing projects. This innovative achievement will play a certain supporting role in the development of the company's main business of seismic data processing.

[0068] This invention is applicable to improving the efficiency of massive seismic data input by using a parallel method to load SEGY disk data. Attached Figure Description

[0069] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0070] Figure 1 This is a schematic diagram of the SEGY disk data parallel loading device in an embodiment of the present invention;

[0071] Figure 2This is a schematic diagram of the overall process of the SEGY disk data parallel loading method in an embodiment of the present invention;

[0072] Figure 3 This is a schematic diagram showing the efficiency comparison in an embodiment of the present invention. Detailed Implementation

[0073] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and understanding only and are not intended to limit the present invention.

[0074] Example 1: A SEGY disk data parallel loading device

[0075] It includes a parameter parsing module, a channel information statistics module, a parallel partitioning strategy module, a data location module, and a data reading module; the parameter parsing module is connected to the channel information statistics module, the channel information statistics module is connected to the parallel partitioning strategy module, the parallel partitioning strategy module is connected to the data location module, and the data location module is connected to the data reading module.

[0076] In this embodiment, the parameter parsing module is used for parameter initialization. The function of this module is to analyze the SEGY file parameters and parallel loading parameters input by the user, and determine the parallel loading mode used by the job: loading at the file level and loading at the data track level.

[0077] In this embodiment, the channel information statistics module is used for data channel information statistics. The function of this module is to count the number of data channels in each SEGY file, while ensuring that the key information of GeoEast data after each subjob conversion is consistent.

[0078] In this embodiment, the data location module is used for data location. The function of this module is to locate the SEGY file that the current subjob is responsible for and the starting channel number of the SEGY.

[0079] In this embodiment, the parallel partitioning strategy module is used for partitioning strategies. The function of this module is to determine the loading range of SEGY data that the current subjob is responsible for.

[0080] In this embodiment, the data reading module is used to read data.

[0081] Example 2: A method for parallel loading of SEGY disk data

[0082] In this invention, each subjob statistically analyzes the data track distribution within each SEGY file to be loaded. Based on the number of output data volumes and the statistical information, it determines whether to load at the file level or at the data track level. For the data track-level loading method, the integrity of the track set must be guaranteed during independent job execution, and the uniqueness of the track set must be guaranteed across all jobs. A schematic diagram of the overall process of the SEGY disk data parallel loading method in this embodiment is shown in Figure 2.

[0083] To verify the effectiveness of this invention, we deployed the software on a high-performance cluster and conducted multiple experiments on it. Each node in the cluster was equipped with two Intel(R) Xeon(R) E5-2660 v3 2.6GHz CPUs, each processor containing 10 physical cores, for a total of 20 physical cores per node, 128GB of memory, and shared storage using Huawei storage. The operating system on the nodes was Red Hat Enterprise Linux Server release 6.8 (Santiago). The test data was the actual data from the processing project, 240GB in size, using the GeoEast 4.0 system. The experiment involved loading SEGY data in a single-node single-job and 8-node 8-job parallel processing configurations to generate GeoEast system data.

[0084] This implementation specifically includes the following steps:

[0085] S1. Parse parallel loading parameters

[0086] S11. Parse the input SEGY disk file parameters

[0087] The filenames of the SEGY disk files collected in the field were placed in the SEGY file container vFiles. The number of SEGY disk files was counted as 33.

[0088] S12. Parse the total number of subjobs and the sequence number of each subjob.

[0089] Set the starting value of the subjob sequence number to 1, and parse the SEGY file parameters input by the user (total number of subjobs is 33 and the sequence number of each subjob is JobId) and parallel loading parameters (total number of subjobs is 8 and the sequence number of each subjob is JobId).

[0090] If the number of files in the SEGY disk file is greater than or equal to the total number of jobs, load the SEGY data of each subjob at the file level; if the number of files in the SEGY disk file is less than the total number of jobs, load the SEGY data of each subjob at the data track level.

[0091] In this embodiment, the number of SEGY disk files is 33, and the total number of subjobs is 8. Therefore, the SEGY data of each subjob is loaded at the file level.

[0092] S2. Data Channel Information Statistics

[0093] S21. Traverse each element in the SEGY file container vFiles, open the corresponding SEGY file, read the C-Card (3200 bytes) and the file header block (400 bytes), and extract the sampling interval, number of samples, and track length;

[0094] In this embodiment, the sampling interval is 2ms, the number of sampling points is 4000, and the data is not SEGY data with variable track length.

[0095] S22. Calculate the byte length of any seismic data track: the sum of the track header length (240 bytes) and the byte length of each sample point;

[0096] In this embodiment, the byte length of one seismic data point is: 240 + 4000 * 4, which is 16240.

[0097] S23. Obtain the size of the SEGY file in bytes, and subtract the length of the C-Card and the length of the file header block from the size of the file in bytes to get the length of the data track;

[0098] In this embodiment, the size of one of the SEGY files is 8515301440 bytes, so the data channel length is: 8515301440-3200-400, that is, 8515297840;

[0099] S24. Calculate the number of channels in each SEGY file;

[0100] In this embodiment, the number of channels in each SEGY file is as follows: 524341, 524341, 524341, 524341, 246001, 524341, 524341, 353326, 524341, 524341, 524341, 524341, 524341, 524341, 524341, 353339, 524341, 524341, 524341, 313916, 524341, 524341, 524341, 524341, 524341, 524341, 355865, 524341, 524341, 94065;

[0101] S25. Place the number of channels for each SEGY file into the data channel container vTraces;

[0102] S3. Develop a data range delineation strategy

[0103] Determine the loading range of SEGY data for each subjob with the job number JobId. For SEGY data loaded at the file level, calculate the range of SEGY files for the subjob with the job number JobId; for SEGY data loaded at the data channel level, calculate the range of data channels for the subjob with the job number JobId.

[0104] Since this embodiment loads the SEGY data of each subjob at the file level, the following steps are performed:

[0105] S311. Calculate the average number of jobs fNum that each subjob is responsible for, which is the number of SEGY disk files in the filename container vFiles / the total number of subjobs Jobs, 33 / 8 = 4.125;

[0106] S312. Calculate the starting file number CurSeq of the subjob, i.e., the value of fNum*(JobId-1), and round it to the nearest whole number.

[0107] S313. Calculate the starting file number NextSeq of the next subjob, that is, round the value of fNum*JobId;

[0108] S314. Calculate the number of files handled by the subjob, CurNum, which is NextSeq-CurSeq;

[0109] S4. Data Positioning

[0110] Open the first file in the SEGY file container vFiles for data analysis, allocate a data channel buffer, and process the SEGY data loaded at the file level.

[0111] Since this embodiment loads the SEGY data of each subjob at the file level, the following steps are performed:

[0112] S411. Remove elements from the file container vFiles that are not within the scope of this subjob, and also remove the keyword control selection range of the corresponding SEGY file;

[0113] If the starting file number of the subjob is 0, execute S412; otherwise, execute S413.

[0114] S412. If the SEGY data contains a C-Card and needs to be output, then output the C-Card and the data positioning ends;

[0115] S413. Open the second file in the file container vFiles. Data location is complete.

[0116] S5. Input SEGY data according to the normal procedure and read the data.

[0117] Figure 3 shows the comparison results of the running time efficiency of this embodiment.

[0118] Depend on Figure 3 It can be seen that the efficiency of loading SEGY data is basically proportional to the number of nodes. Therefore, this method can improve the efficiency and flexibility of seismic data processing personnel in loading SEGY data.

Claims

1. A method for parallel loading of SEGY disk data, characterized in that, The method includes the following steps performed sequentially: S1. Parse parallel loading parameters The filenames of the SEGY disk files collected in the field are placed in the SEGY file container vFiles. The user-input SEGY file parameters and parallel loading parameters are parsed to determine the parallel loading mode used by the job. The SEGY file parameters include the total number of subjobs (Jobs) and the job number (JobId) for each subjob; The parallel loading parameters include the total number of subjobs (Jobs) and the job number (JobId) of each subjob. The parallel loading mode includes loading at the file level and loading at the data channel level; S2. Data Channel Information Statistics S21. Traverse each element in the SEGY file container vFiles, open the corresponding SEGY file, read the C-Card and file header block, and extract the sampling interval, number of samples and track length; S22. Calculate the byte length of any seismic data stream; S23. Obtain the size of the SEGY file in bytes, and subtract the length of the C-Card and the length of the file header block from the size of the file in bytes to get the length of the data track; S24. Calculate the number of channels in each SEGY file; S25. Place the number of channels for each SEGY file into the data channel container vTraces; S3. Develop a data range delineation strategy Determine the loading range of SEGY data for each subjob with the job number JobId. For SEGY data loaded at the file level, calculate the range of SEGY files for the subjob with the job number JobId; for SEGY data loaded at the data channel level, calculate the range of data channels for the subjob with the job number JobId. S4. Data Positioning Open the first file in the SEGY file container vFiles for data analysis, allocate a data channel buffer, and process SEGY data loaded at the file granularity and SEGY data loaded at the data channel granularity respectively. S5. Input SEGY data according to the normal procedure and read the data.

2. The SEGY disk data parallel loading method according to claim 1, characterized in that, In step S1, determining the parallel loading mode used by the job includes the following steps performed sequentially: S11. Parse the input SEGY disk file parameters The filenames of the SEGY disk files collected in the field are placed in the SEGY file container vFiles, and the number of SEGY disk files is counted. S12. Parse the total number of subjobs and the sequence number of each subjob. Set the starting value of the subjob sequence number to 1, and count the total number of subjobs (Jobs) and the sequence number (JobId) of each subjob; If the number of files in the SEGY disk file is greater than or equal to the total number of jobs, load the SEGY data for each subjob at the file level; if the number of files in the SEGY disk file is less than the total number of jobs, load the SEGY data for each subjob at the data track level.

3. The SEGY disk data parallel loading method according to claim 1, characterized in that, In step S22, the byte length of the seismic data is calculated as the sum of the length of the trace header (240 bytes) and the byte length of each trace sample point.

4. The SEGY disk data parallel loading method according to claim 1, characterized in that, In step S3, for SEGY data loaded at the file level, perform the following steps: S311. Calculate the average number of jobs fNum that each subjob is responsible for, which is the number of SEGY disk files in the filename container vFiles / the total number of subjobs Jobs; S312. Calculate the starting file number CurSeq of the subjob, i.e., the value of fNum*(JobId-1), and round it to the nearest whole number. S313. Calculate the starting file number NextSeq of the next subjob, that is, round the value of fNum*JobId; S314. Calculate the number of files handled by the subjob, CurNum, which is NextSeq-CurSeq.

5. The SEGY disk data parallel loading method according to claim 1, characterized in that, In step S3, the SEGY data loaded at the data channel granularity will proceed to the following steps: S321. Statistical data: TotalTraces, the total number of tracks in the track container vTraces; S322. Calculate the average number of data tracks nNum that each subjob is responsible for loading, which is TotalTraces / Jobs; S323. Calculate the starting channel number CurSeq of the current subjob JobId, where the starting channel number is nNum*(nJobId-1); S324. If JobId is not the last subjob, the number of tracks CurNum that the subjob is responsible for is nNum; otherwise, the number of tracks CurNum that the subjob is responsible for is TotalTraces-CurSeq-1. S325. Calculate the SEGY file number of the data channel responsible for the subjob, as well as its starting channel number (TraceSeq) and ending channel number in the file, based on the SEGY data channel number recorded in the data channel container vTraces.

6. The SEGY disk data parallel loading method according to claim 1, characterized in that, In step S4, the SEGY data loaded at the file level will undergo the following steps: S411. Remove elements from the file container vFiles that are not within the scope of this subjob, and also remove the keyword control selection range of the corresponding SEGY file; If the starting file number of the subjob is 0, execute S412; otherwise, execute S413. S412. If the SEGY data contains a C-Card and needs to be output, then output the C-Card and the data positioning ends; S413. Open the second file in the file container vFiles. Data location is complete.

7. The SEGY disk data parallel loading method according to claim 1, characterized in that, In step S4, the SEGY data loaded at the data channel granularity will proceed to the following steps: S421. If the starting file number of the subjob is not 0, execute S422; otherwise, execute S423. S422. Delete elements that are not within the scope of this subjob from the file container vFiles, and delete the keyword control selection range of the corresponding SEGY file, then open the next file for positioning; S423. Locate the data track in the current file, specifically including the following steps: S4231. If the starting trace number TraceSeq in the currently located file is 0, the data location ends; S4232. Calculate the byte position in the location file, and use the file's starting track number * data track length to locate it. If the user has not set a track set keyword, the location ends; otherwise, execute S4233. The keywords of the Dao collection are parameters input by the user; S4233. Read each data point and obtain the CurValue corresponding to the key of the data set; S4234. Read the next data, obtain the trace head value PreValue corresponding to the trace key. If PreValue is not equal to CurValue, it means that the data trace being located is a new trace set, and the location ends; otherwise, execute S4235. S4235. Read the next trace, obtain the trace header value NextValue corresponding to the trace key. When the SEGY file is read to the end or the number of traces read exceeds CurNum, the subjob ends; when NextValue is not equal to CurValue, the positioning ends and backtracks by one trace.

8. A SEGY disk data parallel loading apparatus, used to implement the SEGY disk data parallel loading method according to any one of claims 1 to 7, characterized in that, The device includes a parameter parsing module, a channel information statistics module, a parallel partitioning strategy module, a data positioning module, and a data reading module; the parameter parsing module is connected to the channel information statistics module, the channel information statistics module is connected to the parallel partitioning strategy module, the parallel partitioning strategy module is connected to the data positioning module, and the data positioning module is connected to the data reading module.

Images