Method for realizing three-dimensional median filter and storage device

Abstract

The invention provides a three-dimensional median filter implementation method and a storage device, and belongs to the field of seismic exploration data processing. The method for realizing the three-dimensional median filter comprises the following steps of: 1, inputting seismic data, and determining an element dimension and a fragmentation dimension; 2, initializing a cluster environment, setting calculation parameters, and segmenting data fragments; step 3, performing data fragmentation calculation on each calculation node; step 4, summarizing data fragments of each computing node; and step 5, outputting a calculation result data set. According to the method, the logic of the algorithm itself is taken as the center, a parallel mode and a series of system-side-biased elements such as data division, network communication and node management caused by the parallel mode do not need to be considered too much, and the calculation task implementation cost is reduced; various problems possibly caused by unreasonable operation of a program and related to the use of computing hardware are reduced, and the sensitivity and dependency on the hardware are reduced.

Description

Technical Field

[0001] This invention belongs to the field of seismic exploration data processing, specifically relating to a method and storage device for implementing a three-dimensional median filter. Background Technology

[0002] The 3D median filter is a versatile signal processing tool widely used in seismic data processing, image processing, and many other fields. The calculation process involves point-by-point calculation of the median of the data within the 3D neighborhood of each data point and replacing the value of that data point with the median. In seismic data processing, a typical 3D data volume is the 3D post-stack data volume. Gathers processed by various migration algorithms typically contain information such as offset distance and azimuth. By stacking these gathers, a post-stack data volume in a specific imaging space can be obtained. Taking a single 2D line from this volume yields a stacked profile reflecting the distribution of subsurface structures. In time-domain 3D post-stack data, the position of a sampling point within the entire 3D data volume is determined using inline, xline, and travel time. A single 3D median filter requires the same calculation for all sampling points.

[0003] In recent years, with the changing landscape of oil and gas resource extraction in my country, seismic exploration targets have gradually shifted towards complex, concealed, deep and ultra-deep reservoirs, as well as various unconventional oil and gas reservoirs. These targets pose new challenges to current seismic data processing methods. Simultaneously, with the increasing size of exploration areas and the continuous improvement of spatial and temporal resolution, processing actual field data using single workstations or small clusters is no longer practically feasible. Modern seismic data processing pursues large-scale clusters and high-performance computing methods. Programs running on large systems differ significantly in design and implementation from those running on single machines or small clusters.

[0004] Efficient implementation of three-dimensional median filters is currently approached from both software and hardware perspectives. Software-wise, the focus is typically on designing new data structures and algorithms to enhance the program's parallel computing capabilities. Hardware-wise, various dedicated or general-purpose acceleration hardware such as GPUs, FPGAs, and DSPs are considered to accelerate computation. These methods have achieved good acceleration results on many different types of local filters, and some of the results have been integrated into various commercial or industrial computing software, achieving good practical applications.

[0005] However, solutions for high-performance computing environments running on large clusters are currently relatively scarce. While the Spark framework has been used in various commercial big data scenarios, its application in high-performance scientific computing has not yet been widely adopted. As a successful commercial big data computing framework, Spark offers many features not provided by traditional solutions, such as higher-level data structure abstractions and richer computational libraries. Therefore, exploring the implementation of a three-dimensional median filter based on the Spark framework is of great significance for research on scientific computing models in the context of big data. Summary of the Invention

[0006] The purpose of this invention is to solve the problems existing in the prior art. Addressing the shortcomings of existing technologies, this invention proposes a three-dimensional median filter implementation method based on the Apache Spark computing framework. This invention fully utilizes the Spark framework's MapReduce computing paradigm to automatically segment input data and construct partitions. Because the Hadoop system provides resource management and task planning and scheduling components at the underlying level, parallel processing programs for the Spark framework do not require much operation at the cluster management level.

[0007] This invention is achieved through the following technical solution:

[0008] A first aspect of the present invention provides a method for implementing a three-dimensional median filter, characterized in that the method includes:

[0009] Step 1: Input seismic data and determine element dimensions and patch dimensions;

[0010] Step 2: Initialize the cluster environment, set computing parameters, and split the data into shards;

[0011] Step 3: Perform data sharding and computation for each computing node;

[0012] Step 4: Summarize the data shards from each computing node;

[0013] Step 5: Output the calculation result dataset.

[0014] Furthermore, the earthquake data in step 1 is three-dimensional data; the storage method of the three-dimensional data in the distributed dataset RDD is defined, and the construction of the distributed dataset RDD is completed.

[0015] Furthermore, in step 2, initializing the cluster environment involves dividing the data into shards based on the available computing resources in the cluster; the computing resources include: memory, memory bandwidth, network bandwidth, and CPU performance of a single node.

[0016] Furthermore, the calculation parameters set in step 2 include: the radius parameter of the filtering window of the three-dimensional median filter, and the range of the input data in the element dimension and the piecewise dimension.

[0017] Furthermore, step 3 involves using the Spark framework to perform a three-dimensional median filtering operation on each data shard in parallel, including: numbering the data shards as split1, split2, split3, ..., splitN; repartitioning the entire distributed dataset RDD so that each partition contains only one data shard; and for each data point, calculating and collecting the values ​​of all points in its neighborhood, sorting these values ​​in ascending order, and selecting the median value as the new value for that data point.

[0018] Furthermore, the calculation involves using the JNI interface to transfer the fragmented data from the JVM into local memory, where it is processed by a program written in the local language.

[0019] Furthermore, step 4 includes:

[0020] Confirm that the calculation results between the data shards do not overlap in any of the three dimensions;

[0021] Get the position of each data shard in the original dataset;

[0022] Use the calculation results to replace all the original data points in the original dataset.

[0023] Furthermore, step 4 also includes sharding the completed data, transferring it from local memory to the JVM, managing it through the Spark framework, and uniformly incorporating it into the computation result dataset RDD that collects the computation results.

[0024] Furthermore, step 5 includes: outputting the computation result dataset RDD as a separate file; and converting the computation result dataset RDD into the storage format internal to the application system.

[0025] A second aspect of the present invention provides a computer-readable storage medium, characterized in that: the computer-readable storage medium stores at least one computer-executable program, which, when executed by the computer, causes the computer to perform the steps of the method for implementing a three-dimensional median filter as described in any one of claims 1-9.

[0026] Compared with the prior art, the beneficial effects of the present invention are:

[0027] This invention leverages the flexible data sharding support of the Spark framework's Distributed Dataset (RDD) structure, enabling a computationally centered approach. Compared to parallel computing tasks without framework support, which require defining parallelism and addressing system-side factors such as data partitioning, network communication, and node management, this invention reduces the cost of computational task implementation. The introduction of the Spark framework, combined with Hadoop Yarn's resource coordination, allows program implementation to avoid considering system-side factors, enabling program writing to focus on computational logic. The implementation of three-dimensional median filtering based on modern big data architecture offers better granularity in cluster hardware management compared to applications directly managing computing hardware using various libraries. Furthermore, Hadoop Yarn's excellent management of computing resources reduces various hardware-related problems caused by improper program operations, lowering the program's hardware sensitivity and dependency. The introduction of JVM native method interfaces overcomes, to some extent, the shortcomings of Java / Scala in memory management and computational efficiency when writing Spark tasks. Combined with native parallel design, this allows multi-threaded computation to fully utilize the parallel computing potential of multi-core processors. Attached Figure Description

[0028] Figure 1 This is the overall flowchart of the present invention. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the accompanying drawings:

[0030] This invention performs three-dimensional median filtering on seismic data, representing a highly efficient seismic data processing method based on Spark. The technical solution of this invention is implemented through the following steps:

[0031] Step 1, Data Input

[0032] Input 3D seismic data and determine which dimension will be used as the element dimension and which two dimensions will be used as the tiling dimensions. Taking post-stack data as an example, there are inline and crossline directions, as well as time and depth directions; these three directions constitute the three dimensions of the seismic data. Time or depth can be chosen as the element dimension because these dimensions typically contain the value of a single data point. Inline and crossline are used as tiling dimensions because these two dimensions can be processed in parallel, and each tiling can contain a certain range of inline and crossline data. Define the storage method of the 3D data in a distributed dataset (RDD), thus completing the construction of the distributed dataset (RDD).

[0033] Step 2: Initialize the cluster environment and computing parameters, and split the data into shards.

[0034] The data is partitioned based on the available resources in the cluster, and a corresponding computation process is designed to describe the entire Spark task. After determining one dimension as the element dimension, the radius parameter of the filtering window of the three-dimensional median filter is set in the other two dimensions; the size of the data partitions that a parallel task needs to process is determined by the range of the input data in the element dimension and the partition dimension.

[0035] The radius of the filtering window is determined based on the characteristics of the seismic data. The window radius should be large enough to smooth noise, but not so large as to blur important features. Preferably, the selected filtering window radius should be able to utilize the locality of the data, i.e., the correlation between adjacent data points, to improve filtering efficiency.

[0036] The size of the filtering window radius directly affects the computational load; a larger window radius increases the computational burden, so it needs to be chosen appropriately based on the cluster's computing resources. Computational resources also affect the size of data shards, including the impact of single-node memory. Specifically, a Distributed Dataset (RDD) is a distributed data structure whose data is stored in the memory of each computing node during program execution; therefore, the memory size of a single computing node is a primary consideration. In addition, factors such as memory bandwidth, network bandwidth, and CPU performance are also considered. Network bandwidth is a key consideration because it determines the transmission of data shards managed by the RDD within the cluster. CPU performance takes into account the parallelism control strategy of the Spark framework itself. For example, how many Executors a Spark task requires, combined with the number and performance of CPUs available in the actual cluster, can determine the resource consumption of a Spark application and the basic parallelism control.

[0037] Step 3: Perform sharding calculations on each node.

[0038] The Spark framework is used to perform a three-dimensional median filter operation in parallel on each data shard. The three-dimensional median filter is calculated for a given data point and its neighborhood. Specifically, for each data point, the algorithm collects the values ​​of all points in its neighborhood, sorts these values, and finally selects the median value as the new value for that data point.

[0039] Specifically, the values ​​of all points within the neighborhood of each data point are sorted, usually in ascending order. After sorting, if the number of points in the neighborhood is odd, then the middle value is the median; if the number of points in the neighborhood is even, then the average of the two middle values ​​can be chosen as the median, or the smaller of the two middle values ​​can be chosen.

[0040] The parameters of this 3D median filter include the size of the filter window in each dimension of the data, and it can be globally applied to every data point in each data slice. Furthermore, by leveraging hardware such as GPUs or other parallel function libraries (parallel function libraries provide applications that utilize various parallel computing frameworks or accelerate computing hardware functions; such libraries or accelerated computing hardware are part of a cluster configuration), the efficiency of computing each data slice can be further improved, thereby enhancing the overall computational efficiency of the job.

[0041] For each shard, the JNI interface can be used to transfer shard data from the JVM (Java Virtual Machine) to native memory, and then routed via the native language. native The program, written in the local language, processes the data to complete the computational task. Even when using a program written in the local language, the data is reorganized into a custom format according to various needs. After the data is in the custom format, computation still needs to be performed and the results stored.

[0042] Step 4: Summarize the data shards from each computing node.

[0043] After completing the calculation for each data piece, the calculation results need to be reset to the corresponding positions in the original dataset based on the position of each data piece in the original dataset. The 3D median filtering calculation itself, with padding details determined, can be viewed as the data at each position in the original dataset being replaced by the data at the corresponding position in the calculation results, i.e., the updated original dataset. Specifically, if the calculation results between data pieces do not overlap in the three dimensions, the calculation results from each data piece are used to reset the original data corresponding to each data point in the original dataset.

[0044] Preferably, if the calculation results between shards overlap, the partitioning strategy is re-evaluated and adjusted to ensure that the data range processed by each partition does not overlap with other partitions. Specifically, this includes: increasing padding; and optimizing cross-shard calculations.

[0045] Step 5: Output the calculation results.

[0046] After summarizing the calculation results, the dataset can be output as a separate file or in the application system's internal storage format, depending on the specific needs.

[0047] The following is a specific example using the radius of the three-dimensional median filter window as the input parameter.

[0048] Step 1: Input seismic data. In this invention, the input seismic data is three-dimensional data, meaning it has three dimensions: X, Y, and Z. For the input three-dimensional data, determine which dimension will be used as the element dimension and which two dimensions will be used as the sharding dimensions to complete the construction of the distributed dataset (RDD).

[0049] S11. Select the two dimensions to be partitioned.

[0050] In this invention, the three-dimensional median filter window is defined by input parameters r1, r2, and r3. r1 is the radius of the three-dimensional median filter window in the X dimension, r2 is the radius of the three-dimensional median filter window in the Y dimension, and r3 is the radius of the three-dimensional median filter window in the Z dimension.

[0051] In a distributed environment, data is typically partitioned for parallel processing. For three-dimensional data, sharding is performed only in two dimensions to obtain sharded data that can be used for parallel computation by the Spark framework. In this embodiment, sharding can be performed in the X and Y dimensions.

[0052] S12. Define the storage method for 3D data in a distributed dataset (RDD).

[0053] The Spark framework's computation revolves around the Distributed Dataset (RDD) data structure. An RDD is a resilient data structure that can store elements in various formats. In terms of computational partitioning and storage, different parts of the RDD are stored in different partitions. Specifically, it's necessary to define how the three-dimensional data is stored in the RDD. Data partitioning is performed along the X and Y dimensions, so the Z dimension can be considered a separate data dimension. The entire three-dimensional data volume can be viewed as a dataset composed of records / elements along the Z dimension, while the X and Y dimensions define the size of the dataset.

[0054] Step 2: Initialize the cluster environment and computation parameters, and split the data into shards. In the sharding operation, the sharding logic focuses on the X and Y dimensions and the settings of the median filter parameters. Because the Spark framework needs to perform parallel data processing, it requires reasonable data sharding so that the data in each shard is independent and parallel in its contribution to the result. The three-dimensional median filter, as a local filter, does not require the participation of global data and there is no need for global computation. In a distributed environment, each shard can independently apply the median filter to the data points within its region.

[0055] Therefore, the entire dataset can be partitioned along the X and Y dimensions, so that each data segment retains a certain amount of Z-axis data as its data records. When partitioning, the spatial relationship between the X and Y dimensions needs to be considered. The data in each segment must be adjacent in both the X and Y dimensions to maintain local data continuity and facilitate local operations such as median filtering.

[0056] Step 3: Perform calculations on the data shards obtained in Step 2. Specifically, use the Spark framework to perform a three-dimensional median filter operation on each data shard.

[0057] Furthermore, the data slices obtained in step 2 are numbered as split1, split2, split3, ..., splitN, for a total of N slices; the entire distributed dataset (RDD) is repartitioned so that each partition contains only one data slice. For each data point, the values ​​of all points in its neighborhood are collected, these values ​​are sorted in ascending order, and finally the median value is selected as the new value for that data point.

[0058] The specific steps and reasons are as follows:

[0059] Sorting: Sort the values ​​of all points in the neighborhood of each data point, usually in ascending order. This is because the median is the middle value of a set of numbers; if sorted in ascending order, the value in the middle position is the median.

[0060] Choosing the median: After sorting, if the number of points in the neighborhood is odd, then the middle value is the median; if the number of points in the neighborhood is even, then you can choose the average of the two middle values ​​as the median, or choose the smaller median value.

[0061] Update data points: Assign the median obtained after sorting to the corresponding data points to complete median filtering.

[0062] The first embodiment of the present invention is that, for each shard, the shard data can be transferred from the JVM (Java Virtual Machine) via the JNI interface, through the native language ( native The program (code) is used to process and complete the computation task. This part of the processing takes into account the possibility that the input dataset may be large while the computation cluster size is small during the implementation of the three-dimensional median filter. In this case, if the cluster size cannot be increased, introducing dedicated acceleration computing hardware on each node can further expand the cluster's computing power to shorten the computation task time; furthermore, various acceleration methods for native code can be used to further accelerate the computation, such as using GPUs for accelerated computation.

[0063] The second embodiment of the invention does not use native code to optimize computation, but instead employs the JVM for computation. Specifically, the JVM computation process means that the implementation of the three-dimensional median filter exists as a complete Spark application, without involving any JNI-related operations or components such as native memory. This approach is heavily reliant on Yarn's resource management and computation scheduling, and also requires a large-scale cluster computing resource. In this embodiment, the computation can be simply considered as the computation performed by a program implementation in a language supported by the JVM of a three-dimensional median filter, after the data has been processed by the Spark framework.

[0064] Step 4: Summarize the calculation results for each fragment.

[0065] The computed data shards are then transferred from local memory to the JVM (Java Virtual Machine) for aggregation. After being transferred from local memory to the JVM, the data is managed by the Spark framework and uniformly incorporated into a distributed dataset RDD (a data structure defined by the Spark framework) for collecting and organizing results. Since the computation process involves distributing data across various nodes in the cluster, the data needs to be reorganized according to its original dimensional information. This means determining the position of each data point in the final result and writing it to the correct location. Specifically, the computation results need to be reset to the corresponding positions based on the position of each data shard in the original dataset. If the sharding is reasonable and the computation results between shards do not overlap in the three dimensions, the computation results in each data shard can be used to reset the original data at the original position in the original dataset.

[0066] Without using native code to optimize the calculation process, data can be directly converted into custom formats according to various needs.

[0067] Step 5: Output the calculation results. The entire 3D median filtering calculation is divided into two cases: The first case is when the data does not leave the JVM memory, in which case all data calculations are completed in the JVM. Specifically, this refers to the JVM process of the machine where each Executor node in the Spark computing task resides. The second case is when the data flows to local memory, that is, depending on the specific output needs, the dataset is output as a separate file or in the application system's internal storage format.

Claims

1. A method for implementing a three-dimensional median filter, characterized in that: The method includes: Step 1: Input seismic data and determine element dimensions and patch dimensions; Step 2: Initialize the cluster environment, set computing parameters, and split the data into shards; Step 3: Perform data sharding and computation for each computing node; Step 4: Summarize the data shards from each computing node; Step 5: Output the calculation result dataset.

2. The method for implementing a three-dimensional median filter according to claim 1, characterized in that: The seismic data in step 1 is three-dimensional data; the storage method of the three-dimensional data in the distributed dataset RDD is defined, and the construction of the distributed dataset RDD is completed.

3. The method for implementing a three-dimensional median filter according to claim 1, characterized in that: In step 2, the cluster environment is initialized by dividing the data into shards based on the allocable computing resources in the cluster. The computing resources include: memory of a single node, memory bandwidth, network bandwidth, and CPU performance.

4. The method for implementing a three-dimensional median filter according to claim 1, characterized in that: The calculation parameters set in step 2 include: the radius parameter of the filtering window of the three-dimensional median filter, and the range of the input data in the element dimension and the piecewise dimension.

5. The method for implementing a three-dimensional median filter according to claim 1, characterized in that: Step 3 involves using the Spark framework to perform a three-dimensional median filtering operation on each data shard in parallel, including: numbering the data shards as split1, split2, split3, ..., splitN; repartitioning the entire distributed dataset RDD so that each partition contains only one data shard; and for each data point, calculating and collecting the values ​​of all points in its neighborhood, sorting these values ​​in ascending order, and selecting the median value as the new value for that data point.

6. The method for implementing a three-dimensional median filter according to claim 5, characterized in that: The calculation involves using the JNI interface to transfer fragmented data from the JVM into local memory, where it is then processed by a program written in the local language.

7. The method for implementing a three-dimensional median filter according to claim 1, characterized in that: Step 4 includes: Confirm that the calculation results between the data shards do not overlap in any of the three dimensions; Get the position of each data shard in the original dataset; Use the calculation results to replace all the original data points in the original dataset.

8. The method for implementing a three-dimensional median filter according to claim 1, characterized in that: Step 4 also includes sharding the completed data, transferring it from local memory to the JVM, managing it through the Spark framework, and uniformly incorporating it into the computation result dataset RDD that collects the computation results.

9. The method for implementing a three-dimensional median filter according to claim 1, characterized in that: Step 5 includes: outputting the computation result dataset RDD as a separate file; and converting the computation result dataset RDD into the storage format internal to the application system.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer-executable program, which, when executed by the computer, causes the computer to perform the steps of the method for implementing a three-dimensional median filter as described in any one of claims 1-9.

Images