Seismic data processing automation parallel development device and method
By designing an automated parallel development device for seismic data processing, and utilizing the Spark parallel framework and Resilient Distributed Dataset (RDD), the implementation challenges of Spark parallel technology in seismic exploration data processing were solved, realizing automated parallel processing of seismic data and improving development efficiency and computing performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA PETROLEUM & CHEMICAL CORP
- Filing Date
- 2024-11-07
- Publication Date
- 2026-05-08
AI Technical Summary
In existing technologies, Spark parallel technology is difficult to apply directly to seismic exploration data processing, especially since seismic data processing algorithms are mostly developed in programming languages such as C++ and Fortran, which makes implementation difficult, and the large-scale computing requirements pose challenges to computing resources and algorithm design capabilities.
An automated parallel development device for seismic data processing was designed, including a gather data construction module, an RDD construction module, a C++ base class construction module, a JNI interface module, a data parallel input/output module, and a module development project construction plugin. Parallel processing of seismic data is achieved through an elastic distributed dataset RDD, and automated parallelization is performed using the Spark parallel framework.
It enables the rapid integration of algorithms written in traditional languages into the Spark parallel framework, automating the parallel program development of algorithms, improving module development efficiency, parallel processing capabilities, simplifying the development process, and enhancing computing performance.
Smart Images
Figure CN121996217A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of seismic exploration, specifically relating to an automated parallel development device and method for seismic data processing. Background Technology
[0002] With the continuous development of oil exploration and development operations, exploration and development technologies, and information technology, the demand for exploration precision and accuracy in oil exploration and development is constantly increasing. Advances in geophysical technology, especially in seismic data acquisition, processing, and interpretation technologies, have led to an exponential increase in the volume of seismic exploration data and computational workload, placing ever greater demands on large-scale high-performance computing. Large-scale high-performance computing not only places significant demands on computer resources, including networks, storage, and CPUs, but also challenges the design capabilities of algorithm developers. When the problem size is small or the computing cluster is small, some problems with parallel programs, especially performance issues, are generally not apparent; however, as the problem size increases or the number of nodes in the cluster increases, many problems become apparent. These include the scalability of parallel programs, fault tolerance of hardware and software, and resource utilization efficiency. Spark parallel technology provides an efficient solution for big data processing and analysis and has been widely used. However, because Spark technology is developed based on the Scala language, while seismic data processing algorithms are mostly developed in programming languages such as C++ and Fortran, Spark parallel technology faces significant implementation challenges in seismic exploration data processing. Summary of the Invention
[0003] The purpose of this invention is to solve the problems existing in the prior art. In view of the shortcomings of the prior art, an automated parallel device for earthquake data processing is designed to realize the rapid integration and automatic parallelization of earthquake data processing algorithms.
[0004] This invention is achieved through the following technical solution:
[0005] In a first aspect, the present invention provides an automated parallel development apparatus for seismic data processing, the apparatus comprising:
[0006] The Daoji data construction module is used to obtain and pass functions;
[0007] RDD building blocks are used for data processing;
[0008] The C++ base class construction module is used to receive seismic data passed from the RDD construction module through the gather data construction module;
[0009] The JNI interface module is used for data transfer between Scala code and C++ code;
[0010] The data parallel input module is used to call the C++ base class construction module and RDD construction module to process seismic data and to perform parallel partitioning of the seismic data;
[0011] The data parallel output module is used to output the processed calculation results in parallel.
[0012] A module development project build plugin for building parallel development templates.
[0013] Furthermore, the Dao set data construction module is constructed using an Elastic Distributed Dataset (RDD). The smallest record unit of the Elastic Distributed Dataset (RDD) is a Dao set, which consists of a Dao set number and a Dao set data body forming a key-value pair. The Dao set number key records the unique identifier of the Dao set, and the Dao set data value records the data body of the Dao set.
[0014] Furthermore, the data construction module for the collection uses the map function as its core function.
[0015] Furthermore, the seismic data in the trace set consists of multiple data packets, each of which consists of a trace header and a trace data body. The trace header records the acquisition information of the trace data, including coordinate position, sampling interval, sampling length, and grid information. The trace data body records the amplitude value of the sampling point.
[0016] Furthermore, the track header and the track data body are stored independently; the track header is stored in byte data format, and the track data body is stored in floating-point array format.
[0017] Furthermore, the operations performed by the RDD construction module include:
[0018] Building Resilient Distributed Datasets (RDDs): Provides basic classes and templates for creating resilient distributed datasets (RDDs) from earthquake data sources;
[0019] Parallel computation of Resilient Distributed Datasets (RDDs): Define templates for parallel computation and implement user-defined parallel computation logic;
[0020] Data computation: Performs calculations for seismic data preprocessing, feature extraction, and filtering using the base class for data processing;
[0021] Processing computation results data of Resilient Distributed Datasets (RDDs): Summarizing, statistically analyzing, and transforming the computation results data;
[0022] Output Resilient Distributed Dataset (RDD): Define the base class for data output and save the computation results of the processed Resilient Distributed Dataset (RDD) to the storage system.
[0023] Furthermore, the C++ base class construction module is used to provide the core algorithm interface, and through the core algorithm interface, the calculation function of the core algorithm is directly embedded into the C++ base class.
[0024] Furthermore, the data transfer of the JNI interface module is completed through the Record of the Resilient Distributed Dataset (RDD), which enables the complete transfer of seismic data in the trace collection while also transferring the file header data.
[0025] Furthermore, the module development project building plugin is based on the Eclipse development module building template plugin, and directly builds a parallel development template; the parallel development template includes Spark parallel framework classes, C++ core function integration classes, and module parameter interface information setting files.
[0026] A second aspect of the present invention provides an automated parallel development method for seismic data processing, implemented using an automated parallel seismic data processing apparatus as described in any one of claims 1 to 9, the method comprising:
[0027] Step 1: Build a Spark parallel framework: Build a Spark parallel framework, using the map function, and implement parallel processing of seismic data through the Resilient Distributed Dataset (RDD).
[0028] Step 2: Processing Input Data: Read seismic data through the data parallel input module; construct an elastic distributed dataset (RDD);
[0029] Step 3: Processing data using core computational methods: Obtain seismic data from the Resilient Distributed Dataset (RDD); integrate the core computational methods into a C++ class; perform computations using Spark parallel framework functions;
[0030] Step 4: Output Completed Data: Obtain the seismic data of the new elastic distributed dataset RDD processed by the core computing method through the data parallel output module, and output it to the distributed file system HDFS.
[0031] Compared with the prior art, the beneficial effects of the present invention are:
[0032] This invention rapidly integrates algorithms written in traditional languages into the Spark parallel framework, automating the development of parallel programs for algorithms. It combines big data parallel technology with traditional earthquake processing algorithms, eliminating the need for module developers to learn the Spark parallel process and data reading and output processes. They only need to focus on the core algorithm's performance, significantly improving module development efficiency and enabling large-scale parallel processing of modules. Attached Figure Description
[0033] Figure 1 This is the RRD data structure of the present invention.
[0034] Figure 2 This is a schematic diagram of the device of the present invention.
[0035] Figure 3 This is a diagram illustrating the implementation of the parallel framework of this invention.
[0036] Figure 4 This is a flowchart of the parallel operation process of this invention. Detailed Implementation
[0037] This invention utilizes Spark parallel technology to design an automated parallel development device for seismic data processing. The device is based on the Spark parallel architecture and constructed according to the characteristics of seismic data processing. It includes: a gather data construction module, i.e., a seismic data gather data RDD structure (such as...). Figure 1 As shown, RDD stands for Resilient Distributed Dataset; the JNI interface module is used to encapsulate... Figure 3 The JNI native interface shown; C++ base class building module, i.e. Figure 2 The example shown is the C++ base class for the processing module; the RDD building module, i.e. Figure 2 The Scala base class shown; the data-parallel input module, i.e. Figure 2 The input module shown; the data parallel output module, i.e. Figure 2 The output module is shown; the module development project build plugin. Furthermore, the automated parallel seismic data processing development device also includes a core algorithm interface (functions written in C++) provided to users, and predefined Spark modules within the framework for user use or as examples.
[0038] like Figure 2 As shown, when using it, users first need to define the core algorithm; and through the initial parallel partitioning, distributed parallel data is input into the development framework, creating a parallel development template based on Spark parallel transformation functions as the framework.
[0039] Users input predefined core algorithms into the automated parallel development device for seismic data processing through the core algorithm interface. Specifically, it uses C++ to create algorithms based on C++ base classes and Scala to create algorithms based on Scala base classes. C++ base class building modules and RDD building modules can be used to create seismic data processing modules related to RDDs (Resilient Distributed Datasets), ensuring modules adhere to the framework's specifications and structure. The JNI-based core algorithm interface enables rapid integration of core computational methods from languages such as C++ and Fortran into Spark applications. Parallel development templates automatically convert the originally sequential core algorithms into parallel-executable versions, achieving automated parallel data processing and improving development efficiency and computational performance. The automated parallel development device for seismic data processing also provides utility class plugins and parameter base class plugins to ensure parameter consistency and correctness, simplifying development. The automated parallel development device for seismic data processing is fully encapsulated, allowing users to focus on business logic implementation without needing to concern themselves with the underlying complexity.
[0040] Example 1
[0041] The automated parallel development device for earthquake data processing of the present invention will be described in detail below with reference to the accompanying drawings.
[0042] (1) Data construction module
[0043] As the foundation of the entire development framework, Spark's distributed computing capabilities are used to support parallel processing.
[0044] Given that most seismic data processing algorithms use traces or trace sets as the basic computational unit, performing calculations one trace or one trace set at a time, we adopt the basic transformation function map in Spark as the core function of the Spark parallel framework. This ensures that the processing algorithm can be divided and processed in parallel according to trace sets.
[0045] Furthermore, the automated parallel development device for earthquake data processing designs an earthquake dataset structure built using a Resilient Distributed Dataset (RDD) within the Spark parallel framework, employing an retrieval function and a transfer function to facilitate parallel data flow. Specifically, the retrieval function in the RDD obtains the results calculated by the previous module; the transfer function in the RDD passes the calculation results to the next calculation module.
[0046] Specifically, the basic unit of seismic data is the trace, with each trace representing a seismic wave signal received by a geophone over a specific period during seismic exploration. In seismic data processing, traces with the same order are typically grouped into a trace set, which is the most commonly used data unit in seismic data processing. Based on the organizational characteristics of seismic data, a resilient distributed dataset (RDD) for seismic data is designed, using trace sets as the basic recording unit. The resilient distributed dataset (RDD) uses a key-value pair approach, where the key records the unique identifier of the trace set (i.e.,...). Figure 1 The value records the data body of the Dao set (i.e., the Dao set number). Figure 1 The seismic dataset structure is built using a trace gather as the smallest record in a Resilient Distributed Dataset (RDD). The trace gather is used as a key-value pair, with the trace gather number and the trace gather data body forming the key-value pair. This allows for comprehensive management of the seismic data. Seismic data is loaded into Spark's RDD, and data is transferred via RDD Records to ensure the integrity of a trace gather. Simultaneously, header data is transferred to guarantee the completeness of the seismic data information required for core function calculations.
[0047] Furthermore, the seismic data stored in the data gather consists of multiple data packets, each consisting of a trace header and a trace data body. The trace header records the acquisition information of the trace data, including coordinate location, sampling interval, sampling length, grid information, etc.; the trace data body records the amplitude values of the sampling points. Based on the characteristics of seismic traces, the trace header is a fixed-byte data structure, and the trace data body is a one-dimensional floating-point array. Since the trace header and trace data body have different structures, for ease of management, the trace gather data structure of this invention stores the trace header and trace data body separately. Specifically, the trace header is stored as byte data, and the trace data body is stored as a floating-point array. Specifically, a trace gather record class, Record, is constructed. The Record class records the trace header and trace data body of the seismic data, and also records the number of traces in the trace gather data and the sample point data for each trace.
[0048] Furthermore, the automated parallel development device for earthquake data processing incorporates module parameter verification functions and an error reporting mechanism within its Spark parallel framework. These verification functions validate the rationality of algorithm parameters. If parameters are deemed unreasonable, an error window pops up, terminating program execution and improving program stability.
[0049] (2) C++ base class construction module
[0050] The C++ base class building block receives seismic data from the Scala base class in the Spark parallel framework and provides an integration interface for core data processing functions, i.e., the core algorithm interface. Through this interface, the computation functions of the core algorithms are directly embedded into the C++ base class. The data required by the core computation functions is provided by the core function integration interface, and the computation results are output through the data output interface. The data interface is a JNI native interface, allowing data to be exchanged between the Scala and C++ code. Data is then passed to the Spark parallel framework for processing, updating the Resilient Distributed Dataset (RDD) and creating a new RDD.
[0051] The C++ base class building blocks ensure that users developing processing modules only need to focus on the core algorithm, without having to consider data parallelism and data acquisition and output. After module integration, the Spark parallel framework automatically implements parallel processing.
[0052] (3) RDD building module
[0053] The RDD build module is responsible for data processing in the Spark parallel framework. It is provided by the Scala language and offers the following operations:
[0054] Build Resilient Distributed Datasets (RDDs): Provides base classes or templates to create RDDs from various data sources, such as file systems, databases, etc.
[0055] Parallel computing of Resilient Distributed Datasets (RDDs): Define parallel development templates that allow users to implement custom parallel computing logic.
[0056] Data computation: This is a base class for data processing, used to perform computational operations such as preprocessing, feature extraction, and filtering of seismic data.
[0057] Processing Resilient Distributed Datasets (RDD) computation results: Provides processed data results, including data summarization, statistics, and transformation results.
[0058] Output Resilient Distributed Dataset (RDD) data: Defines the base class for data output, used to save the processed data to a file system, database, or other storage system.
[0059] (4) JNI Interface Module
[0060] The automated parallel development device for earthquake data processing uses the JNI native interface to facilitate data transfer between Scala and C++ code. The JNI native interface's function call interface class is implemented in Scala. Data transfer via the Resilient Distributed Dataset (RDD) Record ensures the complete transfer of a trace gather, including the header data, guaranteeing the integrity of the data required for core function calculations.
[0061] (5) Parallel data input module
[0062] This invention utilizes a parallel data input module to perform parallel partitioning of seismic data. The seismic data input module directly reads seismic data from the HDFS distributed file system, constructing a Resilient Distributed Dataset (RDD) for each trace gather, one record per gather. Each record in the RDD is structured as a key-value pair consisting of a gather number and the gather data body, as follows: Figure 2 As shown. After the Resilient Distributed Dataset (RDD) is constructed, it is passed to the next processing module through the RDD transfer interface. Specifically, after the module completes the computation, the data is still organized in the RDD according to the key-value pair format. The device of this invention passes the data to the next processing module through the RDD. If further processing is required, it can be completed by subsequent processing modules, and so on.
[0063] Because seismic data processing is a complex process requiring numerous methods to work together to obtain results, each method is developed as a processing module using a framework. Users can then connect different processing modules into various processing flows to complete the seismic data processing. Given the need to develop numerous processing modules and ensure that these modules can be arbitrarily connected, a unified development framework is required to guarantee interoperability. Furthermore, the computational demands of seismic data processing are substantial, necessitating a parallel module development framework. Processing modules developed based on this framework are directly connected to the input data module to obtain the Resilient Distributed Dataset (RDD) passed from the input module, thereby acquiring the parallel-partitioned seismic data. The parallel input data module, also developed based on the framework, is solely responsible for generating the RDD and storing the seismic data in the RDD after splitting it according to its structure. Other processing modules based on the framework can then directly connect to the input module to obtain data from the RDD.
[0064] (6) Data Parallel Output Module
[0065] The seismic data processing module developed using the parallel framework, after completing data computation, passes the new Resilient Distributed Dataset (RDD) to another processing module developed using the same framework through the computation result output interface. For example... Figure 3 The STACK module is a processing module developed based on a parallel framework. The parallel data output module receives the computation results, stores them in the Resilient Distributed Dataset (RDD) according to its data type, and outputs the results in parallel through the data output interface. This avoids requiring module users to directly write code for outputting seismic data, significantly simplifying development and improving module development efficiency.
[0066] (7) Module development project build plugin
[0067] This Eclipse-based development module template plugin allows for the direct creation of parallel development templates. The template includes Spark parallel framework classes, a C++ core function ensemble class, and a parameter interface configuration file for the module. Users simply need to set the interface parameters according to their algorithm requirements and embed the core algorithm into the computation interface functions of the C++ function ensemble class to complete the entire development process.
[0068] Example 2
[0069] The implementation process of automated parallel development methods based on the Spark development framework is as follows: Figure 2 and Figure 3 As shown, it includes:
[0070] The first step is to build a Spark parallel framework. This framework, based on an automated parallel device for seismic data processing, utilizes a Resilient Distributed Dataset (RDD) to achieve parallel processing of seismic data. Specifically, the parallel transformation function for the RDD employs the `map` function; for each record in the RDD, the `map` function passes data to the C++ core computation function and Scala types via JNI functions.
[0071] The second step is to process the input data. Using the base classes of the Spark parallel framework, the data-parallel input module reads the seismic data and constructs a resilient distributed dataset (RDD). The input module is a pre-provided, fixed module; users developing processing modules do not need to modify it. They only need to develop their own processing modules, and the input data is obtained by concatenating the input module.
[0072] The third step involves processing the data using core computational methods: This involves developing a core C++ computational class to obtain seismic data from a single record of the Resilient Distributed Dataset (RDD). The core computational methods are integrated into the C++ class by connecting the user-developed seismic data processing module to the input module. The system then reads the parallel-partitioned seismic data transmitted from the RDD, which is returned via the JNI interface. Finally, it uses Spark parallel framework functions to perform computations, resulting in a new seismic data RDD.
[0073] The fourth step is to output the completed data. Using the base class of the Spark parallel framework, the seismic data of the new Resilient Distributed Dataset (RDD) processed by the core computing method is obtained through the data parallel output module. According to the seismic data format, it is output to the distributed file system HDFS, completing the data output.
[0074] Furthermore, when performing parallel processing of seismic data, after developing the seismic data processing module using a parallel development template, it is compiled and deployed via a compilation script. The deployed processing module connects to the data input and output modules to establish a parallel workflow. After data and parameter settings are completed, the parallel processing workflow can be executed to perform parallel processing of seismic data.
[0075] The above technical solution is only one embodiment of the present invention. For those skilled in the art, based on the principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the technical solutions described in the specific embodiments of the present invention. Therefore, the foregoing description is only a preferred option and is not restrictive.
Claims
1. An automated parallel development device for seismic data processing, characterized in that, The device includes: The Daoji data construction module is used to obtain and pass functions; RDD building blocks are used for data processing; The C++ base class construction module is used to receive seismic data passed from the RDD construction module through the gather data construction module; The JNI interface module is used for data transfer between Scala code and C++ code; The data parallel input module is used to call the C++ base class construction module and RDD construction module to process seismic data and to perform parallel partitioning of the seismic data; The data parallel output module is used to output the processed calculation results in parallel. A module development project build plugin for building parallel development templates.
2. The automated parallel development device for seismic data processing according to claim 1, characterized in that, The Dao set data construction module is constructed using an Elastic Distributed Dataset (RDD). The smallest record unit of an Elastic Distributed Dataset (RDD) is a Dao set, which consists of a Dao set number and a Dao set data body forming a key-value pair. The Dao set number key records the unique identifier of the Dao set, and the Dao set data value records the data body of the Dao set.
3. The automated parallel development device for seismic data processing according to claim 1 or 2, characterized in that, The data construction module for the collection uses the map function as its core function.
4. The automated parallel development device for seismic data processing according to claim 2, characterized in that, The seismic data in the trace set consists of multiple data packets, each consisting of a trace header and a trace data body. The trace header records the acquisition information of the trace data, including coordinate position, sampling interval, sampling length, and grid information. The trace data body records the amplitude value of the sampling point.
5. The automated parallel development device for seismic data processing according to claim 4, characterized in that, The track header and the track data body are stored independently; the track header is stored in byte data format, and the track data body is stored in floating-point array format.
6. The automated parallel development device for seismic data processing according to claim 1, characterized in that, The operations performed by the RDD construction module include: Building Resilient Distributed Datasets (RDDs): Provides basic classes and templates for creating resilient distributed datasets (RDDs) from earthquake data sources; Parallel computation of Resilient Distributed Datasets (RDDs): Define templates for parallel computation and implement user-defined parallel computation logic; Data computation: Performs calculations for seismic data preprocessing, feature extraction, and filtering using the base class for data processing; Processing computation results data of Resilient Distributed Datasets (RDDs): Summarizing, statistically analyzing, and transforming the computation results data; Output Resilient Distributed Dataset (RDD): Define the base class for data output and save the computation results of the processed Resilient Distributed Dataset (RDD) to the storage system.
7. The automated parallel development device for seismic data processing according to claim 1, characterized in that, The C++ base class construction module is used to provide the core algorithm interface, and through the core algorithm interface, the calculation function of the core algorithm is directly embedded into the C++ base class.
8. The automated parallel development device for seismic data processing according to claim 1, characterized in that, The data transfer of the JNI interface module is completed through the Record of the Resilient Distributed Dataset (RDD), which enables the complete transfer of seismic data from the trace collection while also transferring the file header data.
9. The automated parallel development device for seismic data processing according to claim 1, characterized in that, The module development project building plugin is based on the Eclipse module building template plugin and directly builds a parallel development template; the parallel development template includes Spark parallel framework classes, C++ core function integration classes, and module parameter interface information setting files.
10. An automated parallel development method for seismic data processing, implemented using the automated parallel seismic data processing apparatus as described in any one of claims 1 to 9, characterized in that, The method includes: Step 1: Build a Spark parallel framework: Build a Spark parallel framework, using the map function, and implement parallel processing of seismic data through the Resilient Distributed Dataset (RDD). Step 2: Processing Input Data: Read seismic data through the data parallel input module; construct an elastic distributed dataset (RDD); Step 3: Processing data using core computational methods: Obtain seismic data from the Resilient Distributed Dataset (RDD); integrate the core computational methods into a C++ class; perform computations using Spark parallel framework functions; Step 4: Output Completed Data: Obtain the seismic data of the new elastic distributed dataset RDD processed by the core computing method through the data parallel output module, and output it to the distributed file system HDFS.