An index file processing method and device, an electronic device, and a storage medium

Abstract

This application discloses an index file processing method, apparatus, electronic device, and storage medium, characterized by comprising: acquiring multiple data file blocks and multiple index files corresponding to each data file block, wherein each data file block contains multiple data bodies, and the index file corresponding to each data file block contains the offset of each data body in the data file block in which it exists; merging the multiple data file blocks into a target data file block in a target join order; calculating the offset of each data body after joining based on the target join order and the index files corresponding to the multiple data file blocks; and saving the offset of each data body after joining to the target index file. This solves the problem of low index file merging efficiency in the prior art and improves the efficiency of index file merging.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an index file processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Earthquake data is typically enormous, reaching hundreds of gigabytes or even terabytes in size. The analysis and processing often employs a divide-and-conquer approach, breaking down large datasets into smaller, parallel datasets for analysis. Finding specific information within these smaller datasets, both before and after analysis, is extremely difficult. Furthermore, merging files also presents significant challenges.

[0003] Therefore, there is a need to provide an index file processing method to solve the problem of low index file merging efficiency in the existing technology and improve the index file merging efficiency. Summary of the Invention

[0004] The main objective of this invention is to provide an index file processing method, apparatus, electronic device, and storage medium to solve the problem of low index file merging efficiency in the prior art and improve the index file merging efficiency.

[0005] In a first aspect, the present invention provides an index file processing method, comprising:

[0006] Obtain multiple data file blocks and multiple index files that correspond one-to-one with them. Each data file block contains multiple data bodies, and the index file that corresponds one-to-one with each data file block contains the offset of each data body in the data file block in which it exists.

[0007] Merge multiple data file blocks into a single target data file block in the target join order;

[0008] The offset of each data body after linking is calculated based on the target linking order and the index file that corresponds one-to-one with multiple data file blocks;

[0009] The offset of each data body after the connection is saved to the target index file.

[0010] Optionally, the step of saving the offset of the concatenated data body to the target index file includes:

[0011] Query the target data body in the target data file block based on the target offset.

[0012] Optionally, the step of calculating the offset of each data body after linking based on the target linking order and the index file corresponding one-to-one with multiple data file blocks includes:

[0013] Get the overall offset of each index file before the position of each data body after the join in the target join order;

[0014] The sum of the overall offsets of each index file and the sum of the current offsets of each data body are used as the offset of each data body after concatenation.

[0015] Optionally, the plurality of data file blocks may store seismic gather data that corresponds one-to-one with them.

[0016] Optionally, the index file corresponding to each data file block contains the offset and length of each data body in the data file block in which it exists;

[0017] The step of calculating the offset of each data body after linking based on the target linking order and the index file that corresponds one-to-one with multiple data file blocks includes:

[0018] The offset and length of each data body after connection are calculated based on the target connection order and the index file that corresponds one-to-one with multiple data file blocks;

[0019] The step of saving the offset of each concatenated data body to the target index file includes:

[0020] The offset and length of each data body after the connection are saved to the target index file.

[0021] According to a second aspect of the present invention, an index file processing apparatus is provided, comprising:

[0022] Acquisition module: used to acquire multiple data file blocks and multiple index files that correspond one-to-one with them. Each data file block contains multiple data bodies, and the index file that corresponds one-to-one with each data file block contains the offset of each data body in the data file block in which it exists.

[0023] Merge module: Used to merge multiple data file blocks into a single target data file block in the target join order;

[0024] Calculation module: used to calculate the offset of each data body after linking based on the target linking order and the index file that corresponds one-to-one with multiple data file blocks;

[0025] Saving module: Used to save the offset of each data body after the connection to the target index file.

[0026] Optionally, the device further includes:

[0027] Query module: Used to query the target data body in the target data file block based on the target offset.

[0028] Optionally, the calculation module further includes a data body offset calculation unit: used to obtain the overall offset of each index file before the position of each data body in the target connection order after connection; and to use the sum of the overall offsets of each index file and the sum of the current offsets of each data body as the offset of each data body after connection.

[0029] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the index file processing method described in any of the first aspects above.

[0030] According to a fourth aspect of the present invention, the present invention provides a storage medium storing a program that, when executed by a computer, implements the index file processing method described in any of the first aspects above.

[0031] Compared with the prior art, one or more embodiments of the above solutions may have the following advantages or beneficial effects:

[0032] This invention merges data file blocks and index files in a sequentially connected manner. This solves the problem of low index file merging efficiency in existing technologies and improves the efficiency of index file merging. Attached Figure Description

[0033] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, for those who are not skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating an index file processing method provided in an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of an index file processing device provided in an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of a computer device for processing index files, provided as an embodiment of the present invention. Detailed Implementation

[0037] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples, so that the process of how the present invention uses technical means to solve technical problems and achieve corresponding technical effects can be fully understood and implemented accordingly. The embodiments of the present invention and the various features therein can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of the present invention.

[0038] Example 1

[0039] like Figure 1 As shown, an embodiment of the present invention provides an index file processing method, including the following steps S101 to S104:

[0040] Step S101: Obtain multiple data file blocks and multiple index files that correspond to them one by one.

[0041] For example, an earthquake data file consists of multiple data file blocks, each containing corresponding seismic gather data for analysis. Each data file block contains multiple data body files, and an index file corresponding to each data file block contains the offset of each data body within that block. For instance, if a data file block contains three data bodies, their offsets in the index file would be 0 bytes, 2000 bytes, and 5000 bytes. When accessing a data body, the index is first used to find its offset within the data file block, and then the data body is accessed using that offset. For example, when accessing a data body with an offset of 2000 bytes, bytes 2000 to 5000 of the data body within the data file block are read.

[0042] Step S102: Merge multiple data file blocks into a single target data file block in the target linking order.

[0043] For example, when there are three data file blocks, namely File1, File2, and File3, they can be merged into one data file block according to the target join order, which is File1, File2, and File3. This join-based merging of data file blocks does not place a significant burden on the server and is a very efficient method.

[0044] Step S103: Calculate the offset of each data body after connection based on the target connection order and the index file that corresponds one-to-one with the multiple data file blocks.

[0045] For example, to obtain the overall offset of each index file before the position of each data body in the target join order, for example, there are 3 data bodies in data file block File3 with offsets of 0 bytes, 2000 bytes and 6000 bytes respectively, the overall offset of the index file of data file block 1 is 10000 bytes and the overall offset of the index file of data file block 2 is 20000 bytes, then the overall offset of each index file before data file block File3 is 30000 bytes.

[0046] The sum of the overall offsets of each index file and the sum of the current offsets of each data body are used as the offset of each data body after concatenation. For example, the offset of the third data body in File3 above is 36,000 bytes in the merged index file.

[0047] Step S104: Save the offset of each concatenated data body to the target index file.

[0048] For example, the index file corresponding to each data file block contains the offset and length of each data body in the data file block. For instance, the File1 data file block includes three data bodies with offsets and lengths of 1000 bytes, 2000 bytes, and 5000 bytes respectively. Similarly, the File2 data file block includes three data bodies with offsets and lengths of 1000 bytes, 2000 bytes, and 4000 bytes respectively. The offsets and file lengths of each data body after merging the two data file blocks are as follows: Data body 1 has an offset of 0 bytes and a length of 1000 bytes; Data body 2 has an offset of 2000 bytes and a length of 2000 bytes; Data body 3 has an offset of 5000 bytes and a length of 1000 bytes; Data body 4 has an offset of 6000 bytes and a length of 1000 bytes; Data body 5 has an offset of 8000 bytes and a length of 1000 bytes; and Data body 6 has an offset of 10000 bytes and a length of 1000 bytes.

[0049] For example, query the target data body within a target data file block based on the target offset. For instance, a data file block contains three data bodies. In the index file, the offsets and lengths of these three data bodies are 0 bytes and 1000 bytes, 2000 bytes and 1000 bytes, and 5000 bytes and 1000 bytes, respectively. When accessing a data body, first find the offset and length of the data body within the data file block using the index, and then access the data body within the data file block using the offset and length. For example, when accessing a data body with an offset of 2000 bytes and a length of 1000 bytes, read the data body between bytes 2000 and 3000 of the data file block.

[0050] This invention significantly improves the efficiency of random data access by establishing an index for accessing seismic data files. Simultaneously, it enables rapid merging of multiple data files using the index data. The efficiency improvement is particularly noticeable for processing very large earthquake data, increasing the flexibility of post-processing of seismic data.

[0051] Example 2

[0052] like Figure 2 As shown, an embodiment of the present invention provides an index file processing apparatus, comprising:

[0053] Acquisition module: used to acquire multiple data file blocks and multiple index files that correspond one-to-one with them. Each data file block contains multiple data bodies, and the index file that corresponds one-to-one with each data file block contains the offset of each data body in the data file block in which it exists.

[0054] Merge module: Used to merge multiple data file blocks into a single target data file block in the target join order;

[0055] Calculation module: used to calculate the offset of each data body after linking based on the target linking order and the index file that corresponds one-to-one with multiple data file blocks;

[0056] Saving module: Used to save the offset of each data body after the connection to the target index file.

[0057] Optionally, the device further includes:

[0058] Query module: Used to query the target data body in the target data file block based on the target offset.

[0059] Optionally, the calculation module further includes a data body offset calculation unit: used to obtain the overall offset of each index file before the position of each data body in the target connection order after connection; and to use the sum of the overall offsets of each index file and the sum of the current offsets of each data body as the offset of each data body after connection.

[0060] Example 3

[0061] Figure 3 This application shows a schematic diagram of the structure of a computer device provided in an exemplary embodiment, the computer device comprising:

[0062] The processor 301 includes one or more processing cores. The processor 301 executes various functional applications and data processing by running software programs and modules.

[0063] The receiver 302 and transmitter 303 can be implemented as a communication component, which can be a communication chip. Optionally, this communication component can include signal transmission functionality. That is, the transmitter 303 can be used to transmit control signals to the image acquisition device and the scanning device, and the receiver 302 can be used to receive corresponding feedback commands.

[0064] The memory 304 is connected to the processor 301 via the bus 305.

[0065] The memory 304 can be used to store at least one instruction, and the processor 301 is used to execute the at least one instruction to implement steps S101 to S104 in the above control method embodiments.

[0066] Those skilled in the art will understand that Figure 3 This is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include network access devices, etc.

[0067] The processor 301 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0068] The memory 304 can be an internal storage unit of the computer device, such as a hard drive or RAM. The memory 304 can also be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 304 can include both internal and external storage units. The memory 304 is used to store the computer program and other programs and data required by the terminal device. The memory 304 can also be used to temporarily store data that has been output or will be output.

[0069] Example 4

[0070] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, which can be loaded and executed by a processor to implement the above-described index file processing method.

[0071] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0072] Example 5

[0073] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the index file processing methods described in the above embodiments.

[0074] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation.

[0075] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0076] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0077] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0078] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0079] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method of processing an index file, characterized by, include: Obtain multiple data file blocks and multiple index files that correspond one-to-one with them. Each data file block contains multiple data bodies, and the index file that corresponds one-to-one with each data file block contains the offset of each data body in the data file block in which it exists. Merge multiple data file blocks into a single target data file block in the target join order; The offset of each data body after linking is calculated based on the target linking order and the index file that corresponds one-to-one with multiple data file blocks; The offset of each data body after the connection is saved to the target index file.

2. The method of claim 1, wherein, After the step of saving the offset of the concatenated data body to the target index file, the following is included: Query the target data body in the target data file block based on the target offset.

3. The method of claim 1, wherein, The step of calculating the offset of each data body after linking based on the target linking order and the index file that corresponds one-to-one with multiple data file blocks includes: Get the overall offset of each index file before the position of each data body after the join in the target join order; The sum of the overall offsets of each index file and the sum of the current offsets of each data body are used as the offset of each data body after concatenation.

4. The method of claim 1, wherein, The multiple data file blocks store the seismic gather data that corresponds to each of them.

5. The method of claim 1, wherein, The index file corresponding to each data file block contains the offset and length of each data body in the data file block in which it exists; The step of calculating the offset of each data body after linking based on the target linking order and the index file that corresponds one-to-one with multiple data file blocks includes: The offset and length of each data body after connection are calculated based on the target connection order and the index file that corresponds one-to-one with multiple data file blocks; The step of saving the offset of each concatenated data body to the target index file includes: The offset and length of each data body after the connection are saved to the target index file.

6. An index file processing apparatus characterized by comprising: include: Acquisition module: used to acquire multiple data file blocks and multiple index files that correspond one-to-one with them. Each data file block contains multiple data bodies, and the index file that corresponds one-to-one with each data file block contains the offset of each data body in the data file block in which it exists. Merge module: Used to merge multiple data file blocks into a single target data file block in the target join order; Calculation module: used to calculate the offset of each data body after linking based on the target linking order and the index file that corresponds one-to-one with multiple data file blocks; Saving module: Used to save the offset of each data body after the connection to the target index file.

7. The apparatus of claim 6, wherein, The device further includes: Query module: Used to query the target data body in the target data file block based on the target offset.

8. The apparatus of claim 6, wherein, The calculation module also includes a data body offset calculation unit: used to obtain the overall offset of each index file before the position of each data body in the target connection order after connection; and to use the sum of the overall offset of each index file and the sum of the current offset of each data body as the offset of each data body after connection.

9. An electronic device, comprising: The system includes a memory and a processor, the memory being used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the index file processing method according to any one of claims 1 to 5.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by a computer, it implements the index file processing method according to any one of claims 1 to 5.

Images