A data completion system and data completion method based on reftek instrument

Abstract

The application discloses a data completion system and method based on Reftek instruments, which comprises a real-time acquisition module, a data storage module, a quality calculation module, a data pulling module and a data fusion module. The real-time acquisition module is connected with the Reftek instruments and is used for executing a data acquisition task started by a user, acquiring data generated by the Reftek instruments in real time and writing the data into a cache. The data storage module is used for consuming and storing data files in the real-time acquisition module. The quality calculation module is used for calculating the completeness of the data files in the data storage module and judging the completeness threshold. The data pulling module is used for calling the data files stored in the Reftek instruments. The data fusion module is used for fusing the pulled data files with the data files judged as having a completeness lower than the completeness threshold, and then storing the data files with higher completeness to the data storage module. The application solves the problem that the completeness of the collected data of the Reftek instruments is low.

Description

Technical Field

[0001] This invention relates to the field of seismic data processing technology, specifically to a data completion system and method based on the Reftek instrument. Background Technology

[0002] The Reftek instrument is a widely used seismograph, compact, lightweight, and low-power, suitable for field use. It features a data sampling frequency of up to 500 Hz and supports SeedLink protocol transmission. SeedLink is a protocol for earthquake monitoring data transmission, primarily used for data streaming services between seismic network systems. This protocol enables researchers to access data from different seismic network systems for research and application.

[0003] Although the Seedlink protocol itself supports resuming interrupted transmissions, unstable transmission networks, instrument power outages and restarts, and frequent instrument malfunctions can lead to low data integrity rates when Reftek instruments receive data in real time. Data integrity directly impacts the accuracy of researchers' scientific research success.

[0004] Therefore, there is an urgent need for a data completion system and method based on Reftek instruments to solve the problem of low data integrity rate of the collected Reftek instruments. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a data completion system and method based on Reftek instruments to solve the problem of low data integrity rate from Reftek instruments.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A data completion system based on a Reftek instrument includes a real-time acquisition module, a data storage module, a quality calculation module, a data retrieval module, and a data fusion module. The real-time acquisition module connects to the Reftek instrument and executes user-initiated data acquisition tasks, acquiring observation data generated by the Reftek instrument in real time via the SeedLink protocol and writing it to a cache. The data storage module consumes and stores data files from the real-time acquisition module. The quality calculation module calculates the integrity rate of the data files stored in the data storage module and performs integrity rate threshold judgment. The data retrieval module is triggered by the quality calculation module when it determines that the integrity rate is lower than a preset integrity rate threshold, and retrieves data files for the required time period stored in the Reftek instrument. The data fusion module is triggered by the data retrieval module and merges the retrieved data files with the data files whose integrity rate is determined to be lower than the integrity rate threshold, and then stores the data files with higher integrity rates to the data storage module.

[0008] To optimize the above technical solution, the specific measures also include:

[0009] Furthermore, the real-time acquisition module supports the breakpoint resume function of the Seedlink protocol.

[0010] Furthermore, the data fusion module is used to simultaneously parse the retrieved data file and the data file that is determined to have a completeness rate lower than the completeness rate threshold, and generate a fused data file according to the strategy of "reorganizing the data within the data file based on the time point, splicing the data points of the two data files in chronological order and deduplicating".

[0011] Furthermore, it also includes a quality alarm module. The quality calculation module is used to perform further calculations on data files with higher integrity rates and to determine the integrity rate threshold. The quality alarm module is used to trigger an alarm when the quality calculation module determines that the data file is still below the integrity rate threshold and to send the data file to the user.

[0012] Furthermore, it also includes a file upload module, which allows users to upload data files obtained from the local storage of the Reftek instrument to the data fusion module through a human-machine interface when they receive an alarm sent by the quality alarm module.

[0013] Furthermore, it also includes a batch processing module, which is used to poll and obtain data files that the user periodically backs up from the local storage of the Reftek instrument, and upload them to the data fusion module.

[0014] Furthermore, it also includes an FTP server, which is used to store data files that users periodically back up locally from the Reftek instrument. The batch processing module is used to poll for data files in the FTP server and upload them in batches to the data fusion module.

[0015] Furthermore, a data completion method includes the following steps:

[0016] Users configure acquisition parameters and start acquisition tasks through the human-computer interaction page. After receiving the task instruction, the real-time acquisition module automatically connects to the Reftek instrument and acquires data in real time through the seedlink protocol. Then, it returns the data to the real-time acquisition module.

[0017] After receiving real-time streaming data, the real-time acquisition module writes it to the cache, and the data storage module consumes the data to achieve orderly data storage.

[0018] The quality calculation module calculates the integrity rate of the data files stored in the data storage module and performs an integrity rate threshold judgment. If the integrity rate is lower than the preset integrity rate threshold, the data retrieval module is triggered.

[0019] When the quality calculation module determines that the integrity rate of the data file is lower than the preset integrity rate threshold, the data retrieval module automatically connects to the Reftek instrument to retrieve the data file for the required time period stored in the Reftek instrument;

[0020] The retrieved data file triggers the data fusion mechanism, which merges the retrieved data file with the data file whose integrity rate is determined to be lower than the integrity rate threshold through the data fusion module, and stores the merged data file with a higher integrity rate into the data storage module.

[0021] Furthermore, it also includes the following steps:

[0022] The system also includes a quality alarm module and a file upload module.

[0023] The quality calculation module obtains data files with higher integrity rates from the data storage module, recalculates the quality, and performs integrity rate threshold judgment.

[0024] If the data file is determined to be below the integrity rate threshold, an alarm will be automatically triggered through the quality alarm module and the merged data file will be sent to the user.

[0025] Users can download the data file from the Reftek instrument based on the data file after one fusion and upload the data file through the file upload module using the human-computer interaction page;

[0026] The uploaded data file automatically triggers the data fusion mechanism, which merges the uploaded data file with the data file that has already been merged, and then stores the merged data file in the data storage module.

[0027] Furthermore, it also includes the following steps:

[0028] The system also includes a batch processing module and an FTP server.

[0029] Users should periodically back up the data files stored locally on the Reftek instrument and upload the copied data files in batches to the FTP server;

[0030] The batch processing module polls and retrieves data files from the FTP server and triggers a data fusion task. The data fusion module then merges the uploaded data files with the stored data files and finally stores the data files with higher integrity rates to the data storage module.

[0031] The beneficial effects of this invention are:

[0032] This invention supports four methods simultaneously for acquiring Reftek device data: breakpoint resume, online file acquisition, offline file upload, and batch processing of backup files. It can acquire observational data files for maximizing data fusion through multiple channels, supports real-time sending of low-quality file alarms to Reftek instrument users (i.e., maintenance personnel), improves the response efficiency of instrument maintenance personnel, and can promptly supplement data when real-time data acquisition is unsatisfactory, reducing the impact on researchers. It supports multi-file data fusion strategies, ensuring that files from multiple channels can be fused to generate the most complete usable data file, thus solving the problem of low data integrity rate of Reftek instruments. Attached Figure Description

[0033] Figure 1 This is a connection diagram of a data completion system based on a Reftek instrument proposed in this invention;

[0034] Figure 2 This is a flowchart illustrating a data completion method proposed in this invention. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0036] As attached Figure 1As shown in the figure, an embodiment of the present invention provides a data completion system based on a Reftek instrument, comprising a real-time acquisition module, a data storage module, a quality calculation module, a data retrieval module, and a data fusion module. The real-time acquisition module connects to and authenticates with the Reftek instrument, and is used to execute user-initiated data acquisition tasks. It acquires observation data generated by the Reftek instrument in real time via the SeedLink protocol and writes it to a cache. The data storage module consumes and stores data files from the real-time acquisition module. The quality calculation module can be configured with a data file integrity threshold, and is used to calculate the integrity rate of the data files stored in the data storage module and perform integrity rate threshold judgment. The data retrieval module is triggered by the quality calculation module when it determines that the integrity rate is lower than the preset integrity rate threshold, and retrieves a data file fragment of the required time period stored in the Reftek instrument through the Reftek instrument's interface. The data fusion module is triggered by the data retrieval module and is used to fuse the retrieved data file fragment with the data file whose integrity rate is determined to be lower than the integrity rate threshold, and then stores the data file with a higher integrity rate to the data storage module.

[0037] In the above scheme, the integrity rate refers to the data integrity rate within a certain time period. The formula for calculating the integrity rate is: number of data points within the time period / (number of seconds within the time period * sampling rate).

[0038] In a further specific implementation based on the above, the real-time acquisition module supports the breakpoint resume function of the Seedlink protocol. It records a data cursor on the local disk at regular intervals, for example, every 3 seconds, to update and record the data receiving position. When the process fails or restarts, the data cursor can be sent to the Reftek instrument, so that the Reftek instrument can start uploading data again from this position, thereby realizing breakpoint resume and ensuring data integrity.

[0039] In a further specific implementation based on the above, the data fusion module is used to simultaneously parse the retrieved data file fragments and the data files that are determined to have a completeness rate lower than the completeness rate threshold. Following the strategy of "reorganizing the data within the data files according to the time points, splicing the data points of the two data files in chronological order and deduplicating them", a fused data file is generated.

[0040] In a further specific implementation based on the above, a quality alarm module is also included. The quality calculation module is used to further calculate the data file with a higher integrity rate and determine the integrity rate threshold. The quality alarm module is used to trigger an alarm when the quality calculation module determines that the data file is still below the integrity rate threshold, and send the data file to the user via email, SMS, in-site message, etc., so that the user can handle it in a timely manner after receiving the alarm.

[0041] This also includes a file upload module, which allows users to upload data files obtained from the Reftek instrument's local storage to the data fusion module via a human-machine interface when they receive an alarm sent by the quality alarm module.

[0042] In a further specific implementation based on the above, a batch processing module is also included. The batch processing module is used to poll and obtain the data files that the user periodically backs up from the local storage of the Reftek instrument, and upload them to the data fusion module.

[0043] This also includes an FTP server, which stores data files that users periodically back up locally from the Reftek instrument. The batch processing module is used to poll for data files on the FTP server and upload them in batches to the data fusion module.

[0044] A data completion method is applied to the aforementioned Reftek instrument-based data completion system, comprising the following steps:

[0045] The system periodically updates and records the data reception location in local disk files, sending data reception location information to the Reftek device when the acquisition task starts, enabling breakpoint resume functionality. This ensures data integrity in case of acquisition task failure. Data quality is calculated based on the integrity rate algorithm for the received Reftek instrument data. When the integrity rate falls below a set threshold, a data completion task is triggered, which retrieves data for missing time periods via the Reftek instrument's API. If the file integrity rate after automatic data completion is still below the threshold, an alarm is sent to instrument maintenance personnel. Instrument maintenance personnel can upload complete data from the device's local storage via human-machine interaction. Data in the Reftek local storage is backed up periodically. When the integrity rate of the acquired data falls below the set threshold within the backup period, backup files are uploaded in batches to the system for data fusion and completion. For multi-channel data, data fusion can be achieved based on file parsing, time-series data reassembly, deduplication and sorting, and file generation, generating files with higher integrity rates.

[0046] As attached Figure 2 As shown, specifically, when performing real-time automatic data fusion:

[0047] a. Data acquisition: Users configure acquisition parameters and start acquisition tasks through the human-computer interaction page. After receiving the task instruction, the real-time acquisition module automatically connects to the Reftek instrument and acquires data in real time through the seedlink protocol. Then, the data is returned to the real-time acquisition module.

[0048] b. Data storage: After receiving real-time streaming data, the real-time acquisition module writes it to the cache, and the data storage module consumes the data to achieve orderly data storage.

[0049] c. Calculate real-time data quality. The quality calculation module calculates the integrity rate of the data files stored in the data storage module and judges the integrity rate threshold. If the integrity rate is lower than the preset integrity rate threshold, the data retrieval module is triggered.

[0050] d. Data retrieval: When the quality calculation module determines that the integrity rate of the data file is lower than the preset integrity rate threshold, the data retrieval module automatically connects to the Reftek instrument to retrieve the data file for the required time period stored in the Reftek instrument.

[0051] e. Real-time automatic data fusion: The retrieved data file triggers the data fusion mechanism, which merges the retrieved data file with the data file whose integrity rate is determined to be lower than the integrity rate threshold through the data fusion module, and stores the merged data file with a higher integrity rate to the data storage module.

[0052] Furthermore, the system also includes a quality alarm module and a file upload module, which are used when performing non-real-time data fusion:

[0053] a) Offline data quality calculation: The quality calculation module obtains data files with higher integrity rates from the data storage module, recalculates the quality, and performs integrity rate threshold judgment.

[0054] b. Trigger an alarm: If it is determined that the data file is still below the integrity rate threshold, an alarm will be automatically triggered through the quality alarm module and the merged data file will be sent to the user.

[0055] c. File upload: Based on the data file after one fusion, the user downloads the data file from the Reftek instrument and uploads the data file through the file upload module on the human-computer interaction page;

[0056] d. Offline data fusion: The uploaded data file automatically triggers the data fusion mechanism, and the uploaded data file is merged with the data file that has been merged once through the data fusion module. The merged data file is then stored in the data storage module.

[0057] Furthermore, the system also includes a batch processing module and an FTP server, so when performing backup data fusion:

[0058] a. Regularly back up data. Users should back up the data files stored locally on the Reftek instrument monthly and upload the copied data files in batches to the FTP server;

[0059] b. Backup data fusion: The batch processing module polls and retrieves data files from the FTP server and triggers a data fusion task. The data fusion module then merges the uploaded data files with the stored data files and stores the data files with higher integrity rates to the data storage module.

[0060] The system and method of this invention can simultaneously support four methods for acquiring Reftek device data: breakpoint resume, online file acquisition, offline file upload, and batch processing of backup files. It can acquire observational data files for maximizing data fusion through multiple channels, supports real-time sending of low-quality file alarms to Reftek instrument users (i.e., maintenance personnel), improving the response efficiency of instrument maintenance personnel. When real-time data acquisition is unsatisfactory, it can promptly supplement the data, reducing the impact on researchers. It supports multi-file data fusion strategies, ensuring that files from multiple channels can be fused to generate the most complete and usable data file, thus solving the problem of low data integrity rates from Reftek instruments.

[0061] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0062] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A data completion system based on a Reftek instrument, characterized in that: The system includes a real-time acquisition module, a data storage module, a quality calculation module, a data retrieval module, and a data fusion module. The real-time acquisition module connects to the Reftek instrument and executes user-initiated data acquisition tasks. It acquires observation data generated by the Reftek instrument in real time via the SeedLink protocol and writes it to a cache. The data storage module consumes and stores the data files in the real-time acquisition module. The quality calculation module calculates the integrity rate of the data files stored in the data storage module and performs integrity rate threshold judgment. The data retrieval module is triggered by the quality calculation module when it determines that the integrity rate is lower than a preset integrity rate threshold, and retrieves the data files for the required time period stored in the Reftek instrument. The data fusion module is triggered by the data retrieval module and merges the retrieved data files with the data files whose integrity rate is determined to be lower than the integrity rate threshold, and then stores the data files with higher integrity rates to the data storage module. The data fusion module is used to simultaneously parse the retrieved data file and the data file that is determined to have a completeness rate lower than the completeness rate threshold. It generates a fused data file according to the strategy of "reorganizing the data within the data file based on the time point, splicing the data points of the two data files in time sequence and deduplicating". It also includes a quality alarm module. The quality calculation module is used to perform further calculations on data files with higher integrity rates and to determine the integrity rate threshold. The quality alarm module is used to trigger an alarm when the quality calculation module determines that the data file is still below the integrity rate threshold and to send the data file to the user.

2. The data completion system based on a Reftek instrument according to claim 1, characterized in that: The real-time acquisition module supports the breakpoint resume function of the Seedlink protocol.

3. The data completion system based on a Reftek instrument according to claim 1, characterized in that: It also includes a file upload module, which allows users to upload data files obtained from the local storage of the Reftek instrument to the data fusion module through a human-machine interface when they receive an alarm sent by the quality alarm module.

4. The data completion system based on a Reftek instrument according to claim 1, characterized in that: It also includes a batch processing module, which is used to poll and retrieve data files that the user periodically backs up from the local storage of the Reftek instrument, and upload them to the data fusion module.

5. A data completion system based on a Reftek instrument according to claim 4, characterized in that: It also includes an FTP server, which is used to store data files that users periodically back up locally from the Reftek instrument. The batch processing module is used to poll for data files in the FTP server and upload them in batches to the data fusion module.

6. A data completion method applied to the data completion system based on a Reftek instrument as described in any one of claims 1-5, characterized in that, Includes the following steps: Users configure acquisition parameters and start acquisition tasks through the human-computer interaction page. After receiving the task instruction, the real-time acquisition module automatically connects to the Reftek instrument and acquires data in real time through the seedlink protocol. Then, it returns the data to the real-time acquisition module. After receiving real-time streaming data, the real-time acquisition module writes it to the cache, and the data storage module consumes the data to achieve orderly data storage. The quality calculation module calculates the integrity rate of the data files stored in the data storage module and performs an integrity rate threshold judgment. If the integrity rate is lower than the preset integrity rate threshold, the data retrieval module is triggered. When the quality calculation module determines that the integrity rate of the data file is lower than the preset integrity rate threshold, the data retrieval module automatically connects to the Reftek instrument to retrieve the data file for the required time period stored in the Reftek instrument; The retrieved data file triggers the data fusion mechanism, which merges the retrieved data file with the data file whose integrity rate is determined to be lower than the integrity rate threshold through the data fusion module, and stores the merged data file with a higher integrity rate into the data storage module.

7. The data completion method according to claim 6, characterized in that, It also includes the following steps: The system also includes a quality alarm module and a file upload module. The quality calculation module obtains data files with higher integrity rates from the data storage module, recalculates the quality, and performs integrity rate threshold judgment. If the data file is determined to be below the integrity rate threshold, an alarm will be automatically triggered through the quality alarm module and the merged data file will be sent to the user. Users can download the data file from the Reftek instrument based on the data file after one fusion and upload the data file through the file upload module using the human-computer interaction page; The uploaded data file automatically triggers the data fusion mechanism, which merges the uploaded data file with the data file that has already been merged, and then stores the merged data file in the data storage module.

8. The data completion method according to claim 6, characterized in that, It also includes the following steps: The system also includes a batch processing module and an FTP server. Users should periodically back up the data files stored locally on the Reftek instrument and upload the copied data files in batches to the FTP server; The batch processing module polls and retrieves data files from the FTP server and triggers a data fusion task. The data fusion module then merges the uploaded data files with the stored data files and finally stores the data files with higher integrity rates to the data storage module.

Images