Method and system for calculating noise power spectrum of massive seismic observation data based on distributed architecture

Through distributed architecture and Spark cluster, computing tasks are automatically allocated and HBase is used to store results, which solves the problem of inefficient efficiency of single-machine nodes when processing massive earthquake observation data, and realizes efficient data processing and storage, which significantly improves computing capabilities.

CN112308317BActive Publication Date: 2024-11-22CHINA EARTHQUAKE NETWORKS CENT CENT +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011188298.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-10-30
Publication Date
2024-11-22
Estimated Expiration
2040-10-30

AI Technical Summary

Technical Problem

The existing single-machine nodes have low efficiency in calculating the noise power spectrum of massive earthquake observation data, slow processing speed, and cannot effectively process and store a large amount of earthquake network data.

Method used

Using a distributed architecture-based method, the Spark distributed cluster and HDFS file system are automatically allocated to multiple computing nodes, the RestFul interface or command line is used to submit computing tasks, and the calculation results are stored in HBase distributed database to optimize data processing. and storage.

Benefits of technology

It significantly improves the computing and processing capabilities of massive earthquake observation data, solves the IO and computing bottlenecks of single-machine nodes processing massive data, and realizes fast and efficient data processing and storage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112308317B_ABST
    Figure CN112308317B_ABST
Patent Text Reader

Abstract

The present invention provides a method and system for calculating the noise power spectrum of massive seismic observation data based on a distributed architecture. The method includes: submitting the noise power spectrum calculation task of massive waveform data in the target time and space range to the distributed cluster based on the RestFul interface or through the command line Spark submit method, automatically assigning the calculation task to the calculation node after creating the elastic distributed data set RDD, parsing the seismic waveform data stored in the distributed storage HDFS file system, calling the noise power spectrum method in units of nodes to carry out the calculation task, and putting the calculation result into the distributed database HBASE using the designed RowKey. The method and system for calculating the noise power spectrum of massive seismic observation data based on a distributed architecture provided by the present invention can solve the low calculation efficiency caused by using a single node to perform massive noise power spectrum calculations, and at the same time, the storage and reading analysis of the calculation results based on HBase further improves the ability to carry out long-period noise power spectrum change analysis of massive seismic data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of earthquake prediction, and in particular to a method and system for calculating noise power spectrum of massive earthquake observation data based on a distributed architecture. Background Art

[0002] The main purpose of building a seismic network is to provide high-quality observation data. The quality of seismic waveform data depends largely on whether the seismic instrument system consisting of seismometers and data collectors is normal. In a broadband recording system, transient changes in the system, instrument glitches, steps and spikes in data recording, signal distortion caused by system failures (such as superposition of high-frequency current signals in the recorded signal, etc.) and the presence of major environmental interference are all important factors affecting the quality of data recording.

[0003] Internationally, the noise power spectrum PSD and power spectrum density probability density function (PDF) methods are mainly used to monitor the ground noise level of stations. The Data Management Center (DMC) of the American Institute of Seismological Research (IRIS) integrated the relevant methods into a stand-alone software PQLX and script program. However, the calculation efficiency is slow, and the operating database is MYSQL. Domestic related research and development are all based on stand-alone program development and compilation such as MATLAB.

[0004] At present, the noise power spectrum method used at home and abroad is mainly based on a single computing node for calculation, which has low calculation efficiency. It takes about half a month to process 1TB of data, and the amount of data and calculation results that can be processed and saved are very limited. Generally, it can only process data from the national monitoring network for half a year to a year. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for calculating the noise power spectrum of massive seismic observation data based on a distributed architecture, which can solve the problems of low calculation efficiency and slow processing caused by using a single node to perform massive noise power spectrum calculations.

[0006] To solve the above technical problems, the present invention provides a method for calculating the noise power spectrum of massive seismic observation data based on a distributed architecture, the method comprising: submitting the noise power spectrum calculation task of massive waveform data in the target time and space range to the distributed cluster based on the RestFul interface or through the command line Spark submit, automatically allocating the calculation task to the computing node after creating a resilient distributed data set RDD, parsing the seismic waveform data stored in the distributed storage HDFS file system, calling the noise power spectrum method in units of nodes to carry out the calculation task, and putting the calculation results into the distributed database HBASE using the designed RowKey.

[0007] In some embodiments, the seismic waveform data includes: real-time seismic streaming data, or archived seismic historical observation data.

[0008] In some implementations, after receiving the parameters that need to be processed and calculated from the foreground, the Spark master node allocates the resources required to process the task, creates a resilient distributed data set RDD, distributes the task to each computing node, and each node completes the calculation process and feeds the results back to the master node.

[0009] In some implementations, the processing process of each working node includes: data preprocessing operations and data calculation operations.

[0010] In some embodiments, the data preprocessing operation includes: unpacking the raw data of a given range at a sampling rate and reassembling it into a complete time series value.

[0011] In some embodiments, the data calculation operation includes: noise power spectrum PSD calculation and PDF calculation.

[0012] In some embodiments, the data calculation operation includes: when a PSD calculation request is received, the algorithm scheduling module queries the specific location of each file that needs to be calculated based on the time range, network and station parameters in the request parameters, and pushes the file location and task ID to the computing node; after receiving the calculation request, the computing node directly obtains the minimized file to be processed from HDFS through the path in the parameters, and calculates each minimized file.

[0013] In addition, the present invention also provides a noise power spectrum calculation system for massive seismic observation data based on a distributed architecture, the system comprising: a Spark distributed cluster, submitting the noise power spectrum calculation task of massive waveform data in the target time and space range to the distributed cluster based on the RestFul interface or through the command line Spark submit, automatically allocating the calculation task to the computing node after creating a resilient distributed data set RDD, parsing the seismic waveform data stored in the distributed storage HDFS file system, calling the noise power spectrum method in units of nodes to carry out the calculation task, and putting the calculation results into the distributed database HBASE using the designed RowKey.

[0014] After adopting such a design, the present invention has at least the following advantages:

[0015] The technical solution of the present invention combines the current mainstream big data technology, based on Hadoop and Spark technology, optimizes the data access and processing links in the original program data processing flow, and solves the IO bottleneck and computing bottleneck of single-machine node processing massive data. Based on HDFS, a distributed archiving and distributed computing architecture of seismic waveform data is designed, and based on the calculation results, a KEY-VALUE power spectrum storage result suitable for distributed database storage is designed. Overall, a set of distributed computing and processing methods for massive seismic waveform data is designed, which greatly improves the computing power and data processing capacity. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0017] Figure 1 It is a flow chart of a method for processing massive seismic waveform data based on a distributed architecture provided by an embodiment of the present invention;

[0018] Figure 2 It is a data processing flow diagram of a method for processing massive seismic waveform data based on a distributed architecture provided in an embodiment of the present invention;

[0019] Figure 3 It is a system operation interface provided by an embodiment of the present invention;

[0020] Figure 4 This is an interface diagram for submitting a computing task based on a RESTful interface provided by an embodiment of the present invention;

[0021] Figure 5 It is an interface diagram of the Spark computing cluster environment and the computing task submitted and generated provided by the embodiment of the present invention;

[0022] Figure 6 This is an interface diagram of a Spark computing cluster generating a computing task of Application:computePsd provided by an embodiment of the present invention;

[0023] Figure 7 It is a schematic diagram of a Spark computing node computing task output log provided by an embodiment of the present invention;

[0024] Figure 8 is a schematic diagram of an HBase distributed data storage cluster provided by an embodiment of the present invention;

[0025] Fig. 9 It is a schematic diagram of a RowKey storage data table provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0027] In the whole algorithm, the Spark computing program is mainly used for task control and result statistics. After the task is started, one mseed is used as a task unit and is assigned to all subordinate worker nodes based on the Spark elastic data set. After receiving the computing task set, each worker node communicates with the Python computing algorithm module and sends the mseed path to be calculated. After the sending is successful, the computing thread remains in a waiting state and polls the computing results at a certain time. After successfully obtaining the computing results, they are persisted and saved in hbase.

[0028] Spark computing process: After submitting a Spark job, this job will start a corresponding Driver process. It will be started on a working node in the cluster. The Driver process itself will occupy a certain amount of memory and CPU cores according to the parameters we set. The first thing the Driver process has to do is to apply to the cluster manager for the resources needed to run the Spark job. The resources here refer to the Executor process. The YARN cluster manager will start a certain number of Executor processes on each working node according to the resource parameters we set for the Spark job. Each Executor process occupies a certain amount of memory and CPU cores.

[0029] After applying for the resources required for job execution, the Driver process will start scheduling and executing the job code. The Driver process will split the Spark job code we wrote into multiple stages. Each stage executes a part of the code snippet, creates a batch of tasks for each stage, and then assigns these tasks to each Executor process for execution. Task is the smallest computing unit, responsible for executing the same computing logic, and each task processes a MINISEED data. A CPU core can only execute one thread at the same time. The multiple tasks assigned to each Executor process are all multi-threaded and run concurrently in the form of one thread per task. After all tasks of a stage are executed, the intermediate results of the calculation will be written to the local disk files of each node, and then the Driver will schedule the next stage.

[0030] The main contents of the technical solution of the present invention are:

[0031] 1) Seismic waveform data migration: The seismic waveform data is reliably transmitted to the distributed file system at an extremely fast speed; an efficient distributed cloud computing system is used to realize the fast and efficient processing of massive seismic waveform data information; and it is stored in the distributed cloud storage system, which has very high performance advantages during warehousing and retrieval.

[0032] 2) Distributed parallel computing of seismic waveform data based on Spark: The system adopts a distributed data computing platform to establish a set of high reliability, high processing performance, online elastic scalability, and uninterrupted task reception for the centralized computing and mining of massive data, providing convenient, fast and efficient technical support for the calculation of various types of seismic data.

[0033] Figure 1 The general process of the method provided by the embodiment of the present invention is shown.

[0034] Since the PSD and RMS algorithms involve complex iterative processes, compared to Hadoop's MapRedurce which repeatedly reads and writes data on the hard disk, Spark has a more obvious speed processing advantage in this regard through memory computing. In order to ensure the efficiency of the calculation, the system's processing platform uses the distributed memory computing platform Spark. Figure 3 The application interface of the system platform is shown.

[0035] The processing layer consists of two parts: algorithm scheduling module and algorithm module. The algorithm module is written in Python and is implemented by calling the obspy library. The algorithm module calculates PSD, PDF and RMS values; the algorithm scheduling module is responsible for receiving calculation tasks, task initialization and task scheduling.

[0036] After receiving the parameters that need to be processed and calculated (date, station, and channel) from the front end, the Spark master node allocates the resources required to process the task and distributes the task to each working node. Each node completes the calculation process and feeds the results back to the master node. Figure 4 The interface for submitting computing tasks based on the RESTful interface is shown.

[0037] Seismic Waveform Data Analysis

[0038] Real-time earthquake streaming data or earthquake historical waveform data files are waveform data in Miniseed format, which consists of 512 bytes and is divided into two parts: a data header and a data body. The data header is 64 bytes long and contains the attributes of the waveform data, such as the network, station, longitude and latitude information, and the time when the waveform data is generated; the data body is 448 bytes of compressed earthquake waveform data, and the data in the data body is decompressed according to the Steim2 algorithm; the basic information and data information of the waveform can be obtained by analyzing each waveform data. Figure 5 The Spark computing cluster environment and the interface for submitting generated computing tasks are shown.

[0039] Data processing and computing

[0040] Data processing flow

[0041] Since the PSD and PDF algorithms involve complex iterative processes, Spark has a clear advantage in this regard compared to Hadoop's MapReduce. In order to ensure the efficiency of the calculation, the system's processing platform uses the distributed memory computing platform SPRAK.

[0042] The processing layer consists of two parts: algorithm scheduling module and algorithm module. The algorithm module is written in Python and is implemented by calling the obspy library. The algorithm module calculates PSD and PDF values; the algorithm scheduling module is responsible for receiving calculation tasks, task initialization, and task scheduling.

[0043] After receiving the parameters that need to be processed and calculated (date, station, and channel) from the front end, the Spark master node allocates the resources required to process the task and distributes the task to each working node. Each node completes the calculation process and feeds the results back to the master node.

[0044] Data collation

[0045] Since the data calculation part cuts the waveform data into hours, and the data currently in the database is not archived and stored in the form of whole seconds, the raw data in a given range needs to be unpacked by milliseconds and reassembled into a complete time series value before calculation.

[0046] Data Calculation

[0047] PSD calculation

[0048] When the processing layer receives a PSD calculation request, the algorithm scheduling module queries the specific location of each file that needs to be calculated based on the time range, network, and station parameters in the request parameters, and pushes the file location and task ID to the server end of the calculation module; after receiving the calculation request, the calculation module directly obtains the mseed file to be processed from HDFS through the path in the parameters, and calculates each mseed file.

[0049] The PSD is calculated using the WELCH method, which is an improved piecewise average period method. The power spectrum density calculation process is: calculation of fast Fourier transform (FFT) data characteristic parameters, fast Fourier transform (FFT), FFT conversion PSD and data smoothing. Figure 6 The figure shows the interface of the computing task generated by the Spark computing cluster, Application:computePsd.

[0050] Calculate the FFT data characteristic parameters:

[0051] This step uses the Hanning window function to obtain sampling points with different window widths;

[0052] FFT

[0053] Perform fast Fourier transform on the obtained sampling points;

[0054] Convert FFT to Psd

[0055] This step converts the Fourier transformed sampling points into PSD data. The conversion formula is:

[0056] PSD=(X^2*2*dt / N)

[0057] smooth

[0058] This step performs a smoothing operation on the calculated PSD data to obtain the velocity power spectrum density result of each sampling point.

[0059] When the PSD data calculation is completed, the calculation module generates the calculation results in a temporary file (frequency and PSD data value), and notifies the algorithm scheduling module of the calculation results and the path of the temporary file through serve (no path is generated if the calculation fails). The algorithm scheduling module parses the temporary file according to the temporary file path and stores it in the HBase data table together with the rowkey composed of the query parameters; at the same time, a shared file is generated and stored in the shared file area for subsequent PDF data calculation. The content stored in the shared file is the period and PSD value. The example is as follows:

[0060]

[0061] PDF Calculation

[0062] When the processing layer receives a PDF calculation request, the algorithm scheduling module queries whether there is a shared PSD file in the shared file area based on the time range, network and station parameters in the request parameters. If not, the query result is fed back to the algorithm scheduling module, and then the algorithm scheduling module calculates the PSD data first; if a shared PSD file exists, the period and PSD data in the file are parsed for calculation. The calculation content is three columns of data: frequency (the inverse of the period), PSD data (discarding the decimal part and leaving only the integer part), and PDF data (the frequency of the same noise value in the same period within a period of time), and the calculation result is saved as a temporary file; and the calculation result and the path of the temporary file are notified to the algorithm scheduling module through serve (no path is generated if the calculation fails). The algorithm scheduling module parses the temporary file according to the temporary file path and stores it in the HBase data table together with the rowkey composed of the query parameters.

[0063] Business Information Processing

[0064] According to business needs, the system calculates the data needed by the system while importing historical data or accessing real-time streams, mainly including: automatic interception of network, station, and channel information and generation of corresponding dictionary tables. Figure 7 The output log of the Spark computing node computing task is shown.

[0065] Data storage

[0066] Distributed database technology and relational database are mainly used to realize efficient and secure online storage of seismic waveform data and calculation results according to data models. The distributed database uses HBase and the relational database is MySql. Figure 8 An HBase distributed data storage cluster is shown.

[0067] Seismic waveform data storage

[0068] Considering the actual business needs and data retrieval efficiency, the HBase-based seismic waveform data storage format is designed as shown in the following table. Considering performance and maintainability, the design idea of ​​storing monthly data in one table is adopted. Each table has two columns, one is ROWKEY and the other stores the original data of each real-time stream. The designed table structure is shown in the following table:

[0069]

[0070] The ROWKEY designed month_timestamp_network code_station code_measurement item_sub-item_sampling rate can be parsed from the data header in the real-time stream; when a large amount of waveform data is in the HBase database, the HBase table will be divided into 1...n Regions and hosted in the RegionServer. Two important properties of Region: StartKey and EndKey represent the rowKey range maintained by this Region. When we want to read / write data, if the rowKey falls within a certain start-endkey range, then we will locate the target region and read / write the relevant data. Fig. 9 The RowKey storage data table is shown.

[0071] Numerical calculation storage

[0072] Since the data processing module needs to calculate the PSD value and PDF value of each direction of the national stations every hour, in order to ensure the query efficiency, these large amounts of result data will be stored in the distributed database HBase. The table design is as follows:

[0073]

[0074]

[0075] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Those skilled in the art may make some simple modifications, equivalent changes or modifications using the technical contents disclosed above, which all fall within the protection scope of the present invention.

Claims

1. A method for calculating noise power spectrum of massive seismic observation data based on distributed architecture, characterized in that: include: Submit the noise power spectrum calculation task of massive waveform data in the target time and space range to the distributed cluster based on the RestFul interface or through the command line Spark submit method. After creating the elastic distributed dataset RDD, the calculation task is automatically assigned to the computing node, and the seismic waveform data stored in the distributed storage HDFS file system is parsed. The noise power spectrum method is called in units of nodes to carry out the calculation task, and the calculation results are put into the distributed database HBASE using the designed RowKey; specifically including: After receiving the parameters that need to be processed and calculated from the foreground, the Spark master node allocates the resources required for processing the task, creates a resilient distributed dataset (RDD), and distributes the task to each computing node. Each node completes the calculation process and feeds the results back to the master node. The processing process of each working node includes: data preprocessing operations and data calculation operations; The data preprocessing operation includes: unpacking the raw data of a given range by milliseconds and reassembling it into a complete time series value; The data calculation operation includes: noise power spectrum PSD calculation and power spectrum density probability density function PDF calculation; When receiving a PSD calculation request, the algorithm scheduling module queries the specific location of each file to be calculated based on the time range, network and station parameters in the request parameters, and pushes the file location and task ID to the computing node. After receiving the calculation request, the computing node directly obtains the miniseed files to be processed from HDFS through the path in the parameters, and calculates each miniseed file. The noise power spectrum PSD is calculated using the WELCH method, which is an improved segmented average period method. The calculation process is: Calculate the fast Fourier transform (FFT) data characteristic parameters: This step uses the Hanning window function to obtain sampling points with different window widths; Perform fast Fourier transform on the obtained sampling points; The Fourier transformed sampling points are converted into PSD data. The conversion formula is: PSD=(X^2*2*dt / N) Smooth the calculated PSD data to obtain the velocity power spectrum density result of each sampling point; When the PSD calculation is completed, the calculation results of the frequency and PSD data values ​​are generated in a temporary file, and the calculation results and the path of the temporary file are notified to the algorithm scheduling module. The algorithm scheduling module parses the temporary file according to the temporary file path and stores it in the HBase data table together with the RowKey composed of the query parameters; at the same time, a shared file is generated and stored in the shared file area for subsequent PDF data calculation. The content stored in the shared file is the period and PSD value: When receiving a PDF calculation request, the algorithm scheduling module queries whether there is a shared PSD file in the shared file area according to the time range, network and station parameters in the request parameters. If not, the query result is fed back to the algorithm scheduling module, and then the algorithm scheduling module calculates the PSD data first; if a shared PSD file exists, the period and PSD data in the file are parsed for calculation. The calculation content is three columns of data: frequency, PSD data, and PDF data, and the calculation result is saved as a temporary file; the calculation result and the path of the temporary file are notified to the algorithm scheduling module, and the algorithm scheduling module parses the temporary file according to the temporary file path and stores it in the HBase data table together with the RowKey composed of the query parameters.

2. The method for calculating noise power spectrum of massive seismic observation data based on distributed architecture according to claim 1 is characterized in that: Seismic waveform data include: real-time seismic streaming data, or archived seismic observation waveform data.

3. A noise power spectrum calculation system for massive seismic observation data based on a distributed architecture using the calculation method described in claim 1 or 2, characterized in that: include: Spark distributed cluster, based on RestFul interface or through the command line Spark submit method to submit the massive waveform data noise power spectrum calculation task in the target time and space range to the distributed cluster, create a resilient distributed dataset RDD and automatically assign the calculation task to the computing node, parse the seismic waveform data stored in the distributed storage HDFS file system, call the noise power spectrum method in units of nodes to carry out the calculation task, and put the calculation results into the distributed database HBASE using the designed RowKey.