Data storage method and system based on SEGY file

Abstract

The invention discloses a data storage method and system based on an SEGY file, and the method comprises the steps: responding to a received target SEGY file, carrying out the Hash calculation of the target SEGY file, and obtaining a Hash value; and determining whether an asynchronous analysis task is generated or not according to the hash value, analyzing the target SEGY file into file header information and data channel information under the condition of determining to execute analysis on the target SEGY file, and storing an analysis result in a database. The database comprises at least one data table, and the data table is used for representing the incidence relation among the user, the target SEGY file, the file header information and the data channel information. According to the method, the uniqueness of the file is quickly detected through the hash value, and the storage and analysis overhead caused by repeated files is greatly reduced. And structured information such as a file header and a data channel of the SEGY file is independently stored, analysis logic is clear, and query is flexible and efficient.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a storage method and system based on SEGY data. Background Technology

[0002] SEGY (Society of Exploration Geophysicists Y) is an internationally recognized seismic exploration data format, widely used in the acquisition, processing, and interpretation of seismic waveforms. Traditional SEGY data processing tools, such as SeismicUnix (SU), OpendTect, and ProMAX, are mainly based on command-line or local processing methods. These methods have linear processing flows, low parsing efficiency, and lack support for concurrent processing and remote access to large-scale data.

[0003] Currently, the mainstream earthquake data processing tools on the market are mainly client software. These software are mostly based on local deployment and rely on high-computing-power terminal devices. They cannot meet the high-concurrency requests of web users. Furthermore, in the process of parsing SEGY data, they are mostly parsed locally in a serial manner. The parsing process is severely blocked, the system response is slow, and it is difficult to process large-volume SEGY files, especially when there are many data channels, the parsing time is seriously consumed. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a data storage method and system based on SEGY files, aiming to improve the parsing capability of SEGY files and reduce parsing overhead.

[0005] This invention discloses a data storage method based on SEGY files, comprising:

[0006] In response to receiving the target SEGY file, perform a hash calculation on the target SEGY file to obtain the hash value;

[0007] The hash value determines whether to generate an asynchronous parsing task, which is used to mark whether to perform parsing on the target SEGY file.

[0008] If it is determined that the target SEGY file will be parsed, the target SEGY file will be parsed into file header information and data channel information, and the parsing results will be stored in a database. The database includes at least one data table, which is used to represent the relationship between the user, the target SEGY file, the file header information and the data channel information.

[0009] Preferably, the data storage method based on SEGY files further includes: obtaining user information, wherein the user information is the identity identifier of the user who uploaded the target SEGY file;

[0010] Establish the association between user information and the target SEGY file.

[0011] Preferably, determining whether to generate an asynchronous parsing task based on the hash value includes:

[0012] The system sequentially matches historical SEGY files corresponding to hash values ​​from the memory cache and disk, and generates matching results, including successful and unsuccessful matches.

[0013] If the matching result is successful, the historical SEGY file will be associated with the target SEGY file;

[0014] If the matching result is a failure, an asynchronous parsing task is generated.

[0015] Preferably, the database includes a file attribute table, a user project table, a file header information table, and a data channel information table;

[0016] The file attribute table is used to record the hash value, storage URL, file parsing status and identifier information of the target SEGY file. The identifier information includes the file identifier, file header ID and data track ID.

[0017] The user project table records the project attributes corresponding to user information. Project attributes include project name, project description, file identifier, and upload time. The user project table and the file attribute table are linked through the file identifier as a logical foreign key.

[0018] The file header information table records the text file header fields and binary file header fields after parsing the target SEGY file; the file header information table and the file attribute table are linked through the file header ID as a logical foreign key.

[0019] The data trace information table is used to record the trace header information and data information after parsing the target SEGY file. The trace header information includes the data trace ID, source point number, and sampling interval. The data information includes the data trace information, which is stored based on the BLOB type. The data trace information table and the file attribute table are linked through the data trace ID as a logical foreign key.

[0020] Preferably, the historical SEGY files corresponding to the hash values ​​are matched sequentially from the memory cache and the disk, and the matching results are generated, including:

[0021] Retrieve historical SEGY files corresponding to the hash value of the target SEGY file from the file attribute table;

[0022] If the hash value of the target SEGY file exists in the file attribute table, the matching result is a successful match;

[0023] If the hash value of the target SEGY file does not exist in the file attribute table, the matching result is a failure.

[0024] Preferably, the database includes a user information table and a data channel mapping table;

[0025] The user information table is used to record user identification;

[0026] The user information table and the user project table are linked by using the identity identifier as a logical foreign key;

[0027] The data track mapping table is used to record the association between the target SEGY file and the data track information.

[0028] Preferably, storing the parsing results in the database includes:

[0029] Get the parsing status of the target SEGY file, including parsing in progress and parsing completed;

[0030] If the parsing status is "parsing in progress", the parsing status is stored in the memory cache;

[0031] If the parsing status is "parsing complete", obtain the file status of the target SEGY file and save the parsing results based on the file status of the target SEGY file. The file status includes "file normal" and "file abnormal".

[0032] Preferably, the parsing results are saved according to the file status of the target SEGY file, including:

[0033] If the file status is normal, the parsed file header information and data channel information are saved to the database;

[0034] If the file status is abnormal, an exception message will be generated.

[0035] Preferably, the data storage method based on SEGY files further includes:

[0036] The system uses a pre-defined procedure to verify the primary key values ​​of related fields in the database. Related fields are used to characterize fields in a data table that have a relationship with each other.

[0037] Add an index to the logical foreign key in the data table;

[0038] Delete isolated data from a data table. Isolated data is used to represent data that has no relation to each other.

[0039] This invention also discloses a data storage system based on SEGY files, used to execute the above-described data storage method based on SEGY files. The data storage system based on SEGY files includes:

[0040] The data processing module is configured to: in response to receiving a target SEGY file, perform a hash calculation on the target SEGY file to obtain a hash value;

[0041] The asynchronous parsing module is configured to determine whether to generate an asynchronous parsing task based on a hash value. The asynchronous parsing task is used to mark whether to perform parsing on the target SEGY file.

[0042] The data storage module is configured to: when it is determined that parsing of a target SEGY file is to be performed, parse the target SEGY file into file header information and data channel information, and save the parsing results in a database. The database includes at least one data table, which is used to represent the relationship between the user, the target SEGY file, the file header information, and the data channel information.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] 1. Efficient duplicate file handling: Quickly detects file uniqueness through hash values, significantly reducing the storage and parsing overhead caused by duplicate files.

[0045] 2. Flexible user project management: User project information is decoupled from files, which can easily support a file to be associated with multiple users or multiple project descriptions.

[0046] 3. Clear modular design: The structured information of the SEGY file, such as the file header and data channels, is stored independently, with clear parsing logic and flexible and efficient querying.

[0047] 4. High scalability: The use of logical foreign keys makes the system design more flexible and can easily support future functional expansion, such as data channel analysis and visualization, fine-grained management of user permissions, etc. Attached Figure Description

[0048] Figure 1 A flowchart illustrating the data storage method based on SEGY files provided by this invention;

[0049] Figure 2 This is a schematic diagram of the SEGY file parsing process provided by the present invention;

[0050] Figure 3 This is a schematic diagram of the data storage system based on SEGY files provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] The present invention will now be described in further detail with reference to the accompanying drawings.

[0053] like Figure 1 As shown, this embodiment of the invention provides a data storage method based on SEGY files, including the following steps:

[0054] Step S1: In response to receiving the target SEGY file, perform a hash calculation on the target SEGY file to obtain a hash value.

[0055] In this embodiment of the invention, after a user uploads a target SEGY file, a hash calculation is performed on the target SEGY file. A unique digital fingerprint, or hash value, is generated for the target SEGY file using a hash algorithm. The hash value can be used for data integrity verification, unique identification, and security verification.

[0056] In this embodiment of the invention, after receiving the target SEGY file, the system also obtains user information, which is the identity identifier of the user who uploaded the target SEGY file, and establishes an association between the user information and the target SEGY file. This allows the system to trace the source of each SEGY file, ensuring clear data ownership. When data access is required, the system can query based on the association between the user information and the target SEGY file, which helps in data management and maintenance and improves data management efficiency.

[0057] Step S2: Determine whether to generate an asynchronous parsing task based on the hash value.

[0058] In this embodiment of the invention, the asynchronous parsing task is used to mark whether to perform parsing on the target SEGY file. After generating the hash value, the historical SEGY files corresponding to the hash value are sequentially matched from the memory cache and disk, and a matching result is generated, including successful matching and failed matching.

[0059] The memory cache and disk contain file attribute tables that record the attributes of SEGY files. Specifically, the historical SEGY files corresponding to the hash value of the target SEGY file are queried from the file attribute table. If the hash value of the target SEGY file exists in the file attribute table, the matching result is a successful match. If the hash value of the target SEGY file does not exist in the file attribute table, the matching result is a failed match.

[0060] In this embodiment of the invention, if the matching result is successful, the historical SEGY file is associated with the target SEGY file; if the matching result is unsuccessful, an asynchronous parsing task is generated.

[0061] For example, if a user uploads multiple files with different SEGYs at the same time, the system will determine that they are different files after hash calculation. If no files are found in the cache or on the disk, the system will distribute the parsing tasks to multiple threads. The size of the thread pool will be dynamically adjusted according to the server performance. If the thread pool has no resources, the files will be queued and wait.

[0062] For example, if a user uploads multiple files with the same SEGY at the same time, the system will determine that they are the same files after hash calculation. If the system does not find the files in the cache or on the disk, it will only dispatch one asynchronous parsing task. Other requests will enter a waiting state. Once the parsing task is completed, the data will be automatically associated to avoid repeated parsing of the same data and reduce resource consumption.

[0063] It's important to note that after a SEGY file is uploaded, the system first calculates its hash value and uses this value to determine if it's the first upload. If a matching historical SEGY file is found, it means the file wasn't uploaded for the first time. In this case, there's no need to generate an asynchronous parsing task; the system can directly retrieve the data corresponding to the historical SEGY file from the database. This quickly determines if the target SEGY file exists and avoids repeatedly parsing duplicate uploads. If the system determines the file is the first upload, it responds immediately, displaying a progress bar to show the progress recorded in the memory cache, allowing users to easily check the parsing status.

[0064] Step S3: If it is determined that the target SEGY file will be parsed, the target SEGY file will be parsed into header information and data channel information, and the parsing results will be saved in the database.

[0065] In this embodiment of the invention, the database includes at least one data table, which is used to represent the relationship between users, target SEGY files, file header information, and data channel information.

[0066] The database includes a file attribute table, a user project table, a file header information table, and a data track information table. The file attribute table records the hash value, storage URL, file parsing status, and identifier information corresponding to a SEGY file. The identifier information includes the file identifier, file header ID, and data track ID.

[0067] The user project table records the project attributes corresponding to user information. Project attributes include project name, project description, file identifier, and upload time. The user project table and the file attribute table are linked through the file identifier as a logical foreign key.

[0068] The file header information table records the text file header fields and binary file header fields after the SEGY file is parsed; the file header information table and the file attribute table are linked through the file header ID as a logical foreign key.

[0069] The data trace information table is used to record the trace header information and data information after the SEGY file is parsed. The trace header information includes the data trace ID, the source point number, and the sampling interval. The data information includes the data trace information, which is stored based on the BLOB type. The data trace information table and the file attribute table are linked through the data trace ID as a logical foreign key.

[0070] It's important to understand that this invention performs layered parsing of SEGY files, dividing the parsing into two main parts: a file header and a data path. This decouples the file, separating and storing the file's metadata (such as the file header and data path information) from the user's engineering information, ensuring a simple and highly scalable system design. Furthermore, it adopts an object-oriented design, using dedicated entity classes to encapsulate data, adhering to the SEGY standard for parsing, and performing byte-level processing. It uses low-level byte stream operations to directly read and process binary data, controlling byte order to ensure big-endian reading, and implementing various byte combination methods, supporting 2-byte, 4-byte, and 6-byte data conversions.

[0071] In data storage design, the core idea is to achieve high efficiency and versatility in SEG-Y file parsing and management. The key point is the separation of files and information, avoiding redundant storage through the design of logical foreign keys, and minimizing the system's overhead in processing duplicate files.

[0072] In this embodiment of the invention, the database also includes a user information table and a data channel mapping table. The user information table is used to record the user's identity identifier. The user information table and the user project table are associated through the identity identifier as a logical foreign key. The data channel mapping table is used to record the association between SEGY files and data channel information.

[0073] This invention employs a structured data disk storage design, decoupling SEGY files from users. At the file level, a file attribute table records attributes strongly related to the SEGY file, such as SHA256 hash value, UUID (used for object storage location), total number of data channels, and file parsing status. Parsing operations are only performed and this information recorded when the file does not exist in memory cache or disk storage. At the user level, after a user uploads a file, it is not directly bound to the parsing result of the SEGY file. Instead, a user-file association is established through a user project table. The user project table records user-related project attributes (such as project name, project description, source file name, upload time, etc.). Thus, even if multiple users upload the same file, it only needs to be parsed once.

[0074] Furthermore, in the storage process of SEGY files, a modular storage approach is adopted, where the file header information of the SEGY file is stored separately in a file header information table. Each field of the text and binary file headers of the SEGY file is refined, allowing the system to accurately parse and manage the header content. Additionally, the data channel information of the SEGY file is stored in a data channel information table to support efficient retrieval and analysis. Simultaneously, a data channel mapping table is used to establish a mapping relationship between SEGY files and data channels, ensuring a clear and easily expandable one-to-many storage logic.

[0075] Furthermore, in the database design of this invention, physical foreign keys are not used; instead, logical foreign keys are used to establish relationships between data in the tables. This design ensures the clarity of data relationships, avoids the coupling problems caused by physical foreign keys, and facilitates expansion and database sharding operations.

[0076] like Figure 2 As shown, the process of saving the parsing results in the database is as follows: First, the parsing status of the target SEGY file is obtained, including parsing in progress and parsing completed. When the parsing status is parsing in progress, the parsing status is stored in the memory cache. When the parsing status is parsing completed, the file status of the target SEGY file is obtained, and the parsing results are saved according to the file status of the target SEGY file, including file normal and file abnormal.

[0077] Furthermore, if the file status is normal, the parsed file header information and data channel information are saved to the database; if the file status is abnormal, abnormal information is generated.

[0078] In this embodiment of the invention, the target SEGY file uploaded using the original disk data storage is parsed using a memory cache, and after the user confirms the parsing result, the target SEGY file is stored in the disk structured database. In this way, frequently requested files that have not been written to disk do not enter disk storage, significantly reducing disk I / O write pressure.

[0079] By storing the parsing status in the cache and tracking the parsing status of the target SEGY file in real time, the system can dynamically adjust its storage strategy. This ensures that files undergoing parsing do not consume database resources, while parsed files are properly handled based on their status. When the file status is normal, the system can quickly save key parsing results (such as file header information and data channel information) to the database for subsequent efficient retrieval and analysis. When the file status is abnormal, the system can immediately generate exception information, allowing administrators to quickly locate and resolve issues, thus ensuring the stability and reliability of the entire data storage process.

[0080] In this embodiment of the invention, to improve performance, logical foreign keys are implemented through field values. Although no strict constraints are enforced, data consistency is achieved through the following means: verifying the primary key values ​​of related fields in the database using a preset program; related fields are used to characterize fields in the data table that have a relationship; adding indexes to the logical foreign keys in the data table; and deleting orphaned data in the data table; orphaned data is used to characterize data that has no relationship.

[0081] This makes the data relationships in the database clearer, while improving the speed and efficiency of data access. The pre-defined verification process can promptly detect and correct data inconsistencies, avoiding data redundancy and errors. Adding indexes can speed up data retrieval, making data reading and updating operations faster. Deleting orphaned data ensures the integrity and accuracy of the data in the database, avoiding interference from invalid data. These measures work together to make the data storage method in this embodiment of the invention more efficient and reliable.

[0082] like Figure 3 As shown, this embodiment of the invention also provides a data storage system based on SEGY files for executing the above-described data storage method based on SEGY files. The data storage system based on SEGY files includes: a data processing module 301, an asynchronous parsing module 302, and a data storage module 303.

[0083] The data processing module 301 is configured to: in response to receiving a target SEGY file, perform a hash calculation on the target SEGY file to obtain a hash value.

[0084] The asynchronous parsing module 302 is configured to determine whether to generate an asynchronous parsing task based on the hash value. The asynchronous parsing task is used to mark whether to perform parsing on the target SEGY file.

[0085] The data storage module 303 is configured to: when it is determined that parsing of a target SEGY file is to be performed, parse the target SEGY file into file header information and data channel information, and save the parsing results in a database. The database includes at least one data table, which is used to represent the relationship between the user, the target SEGY file, the file header information and the data channel information.

[0086] As can be seen from the above technical solutions, the present invention discloses a data storage method and system based on SEGY files. The data storage method includes: in response to receiving a target SEGY file, performing a hash calculation on the target SEGY file to obtain a hash value; determining whether to generate an asynchronous parsing task based on the hash value; if it is determined that the target SEGY file will be parsed, parsing the target SEGY file into file header information and data channel information, and saving the parsing results in the database.

[0087] This invention uses hash values ​​to quickly detect file uniqueness, significantly reducing the storage and parsing overhead caused by duplicate files. Decoupling user project information from files allows a single file to be associated with multiple users or multiple project descriptions. The structured information of the SEGY file, such as the header and data channels, is stored independently, resulting in clear parsing logic and flexible, efficient querying. The use of logical foreign keys enhances system design flexibility, supporting future feature expansion, such as data channel analysis and visualization, and fine-grained user permission management.

[0088] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data storage method based on SEGY files, characterized in that, include: In response to receiving a target SEGY file, a hash calculation is performed on the target SEGY file to obtain a hash value; Based on the hash value, it is determined whether to generate an asynchronous parsing task, which is used to mark whether to perform parsing on the target SEGY file; If it is determined that the target SEGY file will be parsed, the target SEGY file will be parsed into file header information and data channel information, and the parsing results will be stored in a database. The database includes at least one data table, which is used to represent the relationship between the user, the target SEGY file, the file header information and the data channel information.

2. The data storage method according to claim 1, characterized in that, Also includes: Obtain user information, wherein the user information is the identity identifier of the user who uploaded the target SEGY file; Establish the association between the user information and the target SEGY file.

3. The data storage method according to claim 2, wherein determining whether to generate an asynchronous parsing task based on the hash value includes: The historical SEGY files corresponding to the hash value are sequentially matched from the memory cache and the disk, and a matching result is generated, which includes successful matching and failed matching. If the matching result is successful, then the historical SEGY file is associated with the target SEGY file; If the matching result is a failure, then the asynchronous parsing task is generated.

4. The data storage method according to claim 3, characterized in that, The database includes a file attribute table, a user project table, a file header information table, and a data channel information table; The file attribute table is used to record the hash value, storage URL, file parsing status and identifier information corresponding to the target SEGY file. The identifier information includes the file identifier, file header ID and data channel ID. The user project table is used to record the project attributes corresponding to the user information. The project attributes include project name, project description, file identifier, and upload time. The user project table and the file attribute table are linked through the file identifier as a logical foreign key. The file header information table is used to record the text file header fields and binary file header fields after the target SEGY file is parsed; the file header information table and the file attribute table are associated through the file header ID as a logical foreign key; The data trace information table is used to record the trace header information and data information after parsing the target SEGY file. The trace header information includes the data trace ID, the source point number, and the sampling interval. The data information includes the data trace information, which is stored based on the BLOB type. The data trace information table and the file attribute table are associated through the data trace ID as a logical foreign key.

5. The data storage method according to claim 4, characterized in that, The step of sequentially matching the historical SEGY files corresponding to the hash value from the memory cache and disk, and generating matching results, includes: Query the historical SEGY files corresponding to the hash value of the target SEGY file from the file attribute table; If the hash value of the target SEGY file exists in the file attribute table, then the matching result is a successful match; If the hash value of the target SEGY file does not exist in the file attribute table, the matching result is a match failure.

6. The data storage method according to claim 4, characterized in that, The database includes a user information table and a data channel mapping table; The user information table is used to record the user's identity identifier; The user information table and the user project table are linked through the identity identifier as a logical foreign key; The data channel mapping table is used to record the association between the target SEGY file and the data channel information.

7. The data storage method according to claim 6, characterized in that, The process of storing the parsing results in the database includes: Obtain the parsing status of the target SEGY file, including parsing in progress and parsing completed; If the parsing status is "parsing in progress", the parsing status is stored in the memory cache; If the parsing status is "parsing complete", the file status of the target SEGY file is obtained, and the parsing result is saved according to the file status of the target SEGY file. The file status includes "file normal" and "file abnormal".

8. The data storage method according to claim 7, characterized in that, The step of saving the parsing results based on the file status of the target SEGY file includes: If the file status is normal, the parsed file header information and data channel information are saved to the database; If the file status is abnormal, then an error message is generated.

9. The data storage method according to claim 8, characterized in that, Also includes: The primary key value of the associated field in the database is verified using a preset procedure. The associated field is used to characterize the fields in the data table that have a relationship. Add an index to the logical foreign key in the data table; Delete isolated data from the data table, where isolated data represents data that has no relation to each other.

10. A data storage system based on SEGY files, used to execute the data storage method based on SEGY files as described in any one of claims 1 to 9, characterized in that, include: The data processing module is configured to: in response to receiving a target SEGY file, perform a hash calculation on the target SEGY file to obtain a hash value; The asynchronous parsing module is configured to: determine whether to generate an asynchronous parsing task based on the hash value, wherein the asynchronous parsing task is used to mark whether to perform parsing on the target SEGY file; The data storage module is configured to: when it is determined that parsing of the target SEGY file is to be performed, parse the target SEGY file into file header information and data channel information, and save the parsing results in a database, wherein the database includes at least one data table, the data table being used to represent the association between the user, the target SEGY file, the file header information and the data channel information.

Images