Method for storing transaction records in subpackage mode

Abstract

The invention relates to the technical field of information, in particular to a method for storing transaction records in a subpackage mode. Comprising the following steps that transaction records are stored in a plurality of sub-files with fixed sizes, and the sub-files are named as file (0) to file (m) in sequence; receiving a read-write request, wherein the request comprises an index number of a target record, an offset address and the length of data needing to be read and written; according to the offset address and the fixed length of the single sub-file, calculating the number x of the sub-file where the target record is located and an initial address in the sub-file; whether the length of the data needing to be read and written exceeds the remaining readable and writable length of the current sub-file (x) or not is judged; if not, directly performing read-write operation on the sub-file (x); and if so, sequentially processing the read-write operation across the sub-files. The invention provides a storage method capable of effectively reducing backup data volume and improving read-write efficiency, so as to solve the problem of performance bottleneck caused by file expansion in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and particularly to a method for subcontracting and storing transaction records. Background Art

[0002] In existing file systems, transaction records are usually stored in the form of a single file, and read and write operations are implemented through record indexes of a fixed length. However, when using a file system with power-off recovery capabilities (such as littlefs), due to the copy-on-write mechanism that requires backing up the entire file, as the file size continues to grow, the backup operation time significantly increases, resulting in a sharp decline in the read and write efficiency of the file system. This problem is particularly prominent in large-scale transaction scenarios, seriously restricting the real-time performance and stability of the system. Summary of the Invention

[0003] The present invention provides a storage method that can effectively reduce the amount of backup data and improve the read and write efficiency, so as to solve the performance bottleneck problem caused by file expansion in the prior art.

[0004] The technical solution adopted by the present invention is as follows: A method for subcontracting and storing transaction records, comprising the following steps:

[0005] Step 1: Store the transaction records in multiple sub-files of a fixed size, and the sub-files are sequentially named file(0) to file(m), where m is a natural number;

[0006] Step 2: Receive a read / write request, where the request includes the index number, offset address, and data length to be read / written of the target record;

[0007] Step 3: Calculate the sub-file number x where the target record is located and the starting address within the sub-file according to the offset address and the fixed length of a single sub-file, where: x = offset address ÷ fixed length of a single sub-file, starting address = offset address % fixed length of a single sub-file;

[0008] Step 4: Determine whether the data length to be read / written exceeds the remaining readable / writable length of the current sub-file file(x);

[0009] Step 5: If not, directly perform read / write operations on the sub-file file(x);

[0010] Step 6: If it exceeds, sequentially process the read / write operations across sub-files, including the following steps:

[0011] S1: Read / write the remaining readable / writable length of the current sub-file file(x);

[0012] S2: Update the offset address to the current offset address plus the length of the data that has been read and written, and update the length of the data to be read and written to the remaining unread and unwritten length;

[0013] S3: Repeat steps three to six until all data has been read and written.

[0014] As a further improvement of the present invention, the fixed length of each sub-file in step one is determined according to the length of a single transaction record and a preset sub-packaging strategy.

[0015] As a further improvement of the present invention, in step three, when the index number is an input parameter, the offset address is obtained by multiplying the index number by the length of a single record.

[0016] As a further improvement of the present invention, when reading and writing across sub-files in step six, a circular overwrite strategy and a dynamic expansion strategy are adopted, specifically including: if the number of sub-files has reached the upper limit m, overwrite the oldest sub-file; if dynamic expansion is allowed, add a new sub-file file(m + 1) and update the value of m.

[0017] As a further improvement of the present invention, the storage path of the sub-files is dynamically allocated based on a hash algorithm and a load balancing strategy.

[0018] As a further improvement of the present invention, the method is applicable to an embedded file system, and the metadata of each sub-file is stored independently to support fast recovery.

[0019] As a further improvement of the present invention, before the read and write operations, the access efficiency of the sub-files is optimized by pre-allocating continuous disk space and memory buffers.

[0020] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0021] An electronic device includes a memory, a processor, and a computer program stored on the memory. When the processor executes the computer program, the above method is implemented.

[0022] Through sub-packaging storage, dynamic positioning, and intelligent expansion mechanisms, the present invention effectively solves the performance bottleneck problem caused by file expansion in traditional file systems, and significantly improves the backup efficiency, read and write speed, system stability, and resource utilization rate. The specific beneficial effects are as follows.

[0023] (1) By splitting transaction records into multiple sub-files of a fixed size for storage, in a file system with power-off recovery function (such as littlefs), the copy-on-write operation only needs to back up the currently modified sub-file, rather than the entire large file. Compared with the traditional single-file storage solution, the amount of data to be backed up is reduced to 1 / m (m is the total number of sub-files) of the original solution, significantly shortening the backup time, and thus significantly improving the read and write efficiency of the file system, especially in transaction scenarios with frequent updates, the performance improvement is more obvious.

[0024] (2) Through the dynamic calculation of the offset address and the fixed length of the sub-file (x = offset address ÷ sub-file length), combined with the segmented read and write strategy across sub-files, the present invention can seamlessly handle the read and write requirements of ultra-long records. This method avoids the positioning delay caused by file expansion in the traditional solution, and at the same time flexibly manages the storage space through the loop overwrite or dynamic expansion strategy, extending the service life of the system.

[0025] (3) The metadata of each sub-file of the present invention is stored independently, supporting fast recovery and fault isolation. In case of power-off or system exception, only some sub-files need to be recovered, significantly reducing the risk of data loss. Combining the pre-allocation of continuous disk space or memory buffer technology further optimizes the real-time response ability of the embedded file system.

[0026] (4) The length of the sub-file of the present invention can be dynamically adjusted according to the length of a single transaction record (such as according to a preset sub-packaging strategy), and the storage path is allocated based on the hash algorithm or load balancing strategy, which not only ensures the uniform distribution of data but also avoids the storage hotspot problem. In addition, the automatic conversion mechanism between the index number and the offset address (index number × length of a single record) simplifies the user operation and improves the usability of the system.

[0027] (5) The dynamic expansion strategy of the present invention allows new sub-files (such as file(m + 1)) to be added to meet the data growth demand, while the loop overwrite strategy can automatically reuse old files when the storage space is limited, avoiding waste of storage resources. This method is especially suitable for scenarios with limited storage resources but large data throughput, such as Internet of Things devices and financial transaction terminals. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flowchart of a method for storing transaction records by sub-packaging according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] The present invention provides a method for subcontracting and storing transaction records, comprising the following steps:

[0031] Step 1: Store the transaction records in multiple sub-files of a fixed size, and the sub-files are sequentially named as file(0) to file(m), where m is a natural number;

[0032] Step 2: Receive a read / write request, and the request includes the index number, offset address, and data length to be read / written of the target record;

[0033] Step 3: Calculate the sub-file number x where the target record is located and the starting address within the sub-file according to the offset address and the fixed length of a single sub-file, where: x = offset address ÷ fixed length of a single sub-file, and starting address = offset address % fixed length of a single sub-file;

[0034] Step 4: Determine whether the data length to be read / written exceeds the remaining readable / writable length of the current sub-file file(x);

[0035] Step 5: If not exceeded, directly perform read / write operations on the sub-file file(x);

[0036] Step 6: If exceeded, sequentially process read / write operations across sub-files, including the following steps:

[0037] S1: Read / write the remaining readable / writable length of the current sub-file file(x);

[0038] S2: Update the offset address to the current offset address plus the length that has been read / written, and update the data length to be read / written to the remaining unread / written length;

[0039] S3: Repeat steps 3 to 6 until all data is read / written.

[0040] In the present invention, the fixed length of each sub-file in step 1 is determined according to the length of a single transaction record and a preset subcontracting strategy.

[0041] In the present invention, when the index number is an input parameter in step 3, the offset address is obtained by multiplying the index number by the length of a single record.

[0042] In the present invention, a cyclic overwrite strategy and a dynamic expansion strategy are adopted during read / write operations across sub-files in step 6, specifically including: if the number of sub-files has reached the upper limit m, overwrite the oldest sub-file; if dynamic expansion is allowed, add a new sub-file file(m + 1) and update the value of m.

[0043] In the present invention, the storage path of the sub-files is dynamically allocated based on a hash algorithm and a load balancing strategy.

[0044] The method described in the present invention is applicable to an embedded file system, and the metadata of each sub-file is stored independently to support fast recovery.

[0045] Before the read and write operations in the present invention, the access efficiency of sub-files is optimized by pre-allocating continuous disk space and memory buffers.

[0046] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0047] An electronic device includes a memory, a processor, and a computer program stored on the memory. When the processor executes the computer program, the above method is implemented.

[0048] Embodiment:

[0049] The following combines specific application scenarios to detail the implementation manner of the present invention. This embodiment takes the storage of transaction records in a financial transaction terminal as an example to show the specific application of the method for storing transaction records in packets.

[0050] System Configuration and Environment

[0051] (1) Hardware environment: An embedded device based on an ARM Cortex-M7 processor, equipped with 64MB of NAND flash memory, supporting power-off recovery function; (2) File system: The littlefs file system is adopted, version 2.4.1; (3) Software configuration: The length of a single transaction record is 512 bytes, the preset fixed size of a sub-file is 1MB (i.e., a single file can store 2048 records), and the initial number of sub-files m = 5 (i.e., file(0) to file(4)).

[0052] Implementation Steps and Operation Processes

[0053] Step 1, Sub-file Initialization and Storage Allocation:

[0054] According to the length of a single transaction record (512 bytes) and the preset packet splitting strategy (a single file stores 2048 records), 5 sub-files (file(0) to file(4)) are initialized, and the fixed size of each sub-file is 1MB.

[0055] The storage path allocation adopts a hash algorithm. A hash value is generated according to the transaction timestamp, and the sub-files are evenly distributed in 3 physical partitions of the flash memory to avoid storage hotspots.

[0056] Step 2, Receive Read and Write Requests and Calculate Positions:

[0057] Example request: It is required to read the record with the index number 5000, and the data length is 1024 bytes (i.e., 2 consecutive records).

[0058] Conversion to offset address through index number: Offset address = 5000 × 512 bytes = 2,560,000 bytes.

[0059] Calculate the target sub-file number and starting address: ① Sub-file number x = 2,560,000 ÷ 1,048,576 (1MB) ≈ 2 (take the integer part), that is, the target sub-file is file(2); ② Starting address = 2,560,000 % 1,048,576 = 462,848 bytes.

[0060] Step 3, Cross-sub-file read and write processing:

[0061] The length of the data to be read is 1024 bytes. Check the remaining readable and writable length of file(2): 1,048,576 - 462,848 = 585,728 bytes.

[0062] Since 585,728 bytes > 1024 bytes, directly read 1024 bytes of data starting from 462,848 bytes in file(2), without cross-file operation.

[0063] Cross-file scenario example: If it is necessary to read an offset address of 1,000,000 bytes and the data length is 1MB + 512 bytes (exceeding the single-file capacity): ① First, read the remaining length of file(0) (1,048,576 - 1,000,000 = 48,576 bytes); ② Update the remaining data length to 1,048,576 + 512 - 48,576 = 1,000,512 bytes; ③ Update the offset address to the starting position of file(1) (0 bytes), and continue to read 1MB of data from file(1); ④ The remaining 512 bytes are filled by bytes 0 to 511 of file(2) to complete all data read and write.

[0064] Step 4, Dynamic expansion and circular overwrite strategy:

[0065] Dynamic expansion: When the number of sub-files reaches the upper limit m = 5, if expansion is allowed, add file(5) and update m = 6, and write subsequent data to the new file.

[0066] Circular overwrite: If expansion is prohibited, overwrite the oldest file(0), mark it as reusable, write subsequent data to file(0) and reset its metadata.

[0067] Metadata management and recovery mechanism

[0068] The metadata of each sub-file (such as file number, storage location, last modification time) is independently stored in a dedicated area of the flash memory, and CRC check is used to ensure integrity.

[0069] After the system loses power abnormally, it only needs to scan the metadata of each sub-file, quickly locate the damaged file (such as file(2)), and recover the sub-file from the backup copy, without having to reconstruct the entire transaction record library.

[0070] Performance verification and effect comparison

[0071] Index Traditional single-file storage Sub-packet storage of the present invention Backup time-consuming (1GB file) 1200ms 240ms (only backup one sub-file) Reading latency (cross-file operation) 15ms 5ms (pre-allocated buffer optimization) Storage space utilization rate Fixed allocation, easy to fragment Dynamic expansion / circular overwrite, utilization rate increased by 40% Fault recovery time Full-file scan (300ms) Only scan metadata (50ms)

[0072] Application scenario expansion

[0073] (1) Internet of Things devices: In the scenario of high-frequency writing of sensor data, the number of sub-files is limited to 10 through the cyclic overwrite strategy, and 1000 pieces of data are stored in each file to ensure the long-term availability of storage resources.

[0074] (2) Blockchain nodes: Split the transaction block into multiple sub-files for storage, and combine hash path allocation to achieve distributed storage and fast verification.

[0075] This embodiment verifies the remarkable effects of the present invention in reducing the amount of backup data, improving read and write efficiency, and enhancing system stability through specific parameter settings, cross-file read and write processes, and performance comparison. Especially in resource-constrained embedded environments, through sub-packet storage and intelligent expansion mechanisms, the performance bottleneck problem of traditional file systems is effectively solved, providing an efficient and reliable storage solution for high-concurrency transaction scenarios.

[0076] In summary, a method for storing transaction records in sub-packets according to the present invention ensures high data availability and efficient operation of the system. Specifically, the independent storage of metadata not only simplifies the data recovery process, but also improves the speed and accuracy of data recovery. In the event of system anomalies or power outages and other emergencies, the system can quickly locate and recover damaged sub-files by scanning the metadata, avoiding the time consumption and resource waste caused by reconstructing the entire transaction record library in traditional solutions. In addition, the CRC check mechanism of metadata further enhances the integrity and reliability of data, providing a strong guarantee for the secure storage of transaction records.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for subcontracting and storing transaction records, characterized in that, It includes the following steps: Step 1: Store the transaction records in multiple sub-files of a fixed size, and the sub-files are sequentially named as file(0) to file(m), where m is a natural number; Step 2: Receive a read / write request, and the request includes the index number of the target record, the offset address, and the length of the data to be read / written; Step 3: Calculate the sub-file number x where the target record is located and the starting address within the sub-file according to the offset address and the fixed length of a single sub-file, where: x = offset address ÷ fixed length of a single sub-file, starting address = offset address % fixed length of a single sub-file; Step 4: Determine whether the length of the data to be read / written exceeds the remaining readable / writable length of the current sub-file file(x); Step 5: If it does not exceed, directly perform read / write operations on the sub-file file(x); Step 6: If it exceeds, sequentially process the read / write operations across sub-files, including the following steps: S1: Read / write the remaining readable / writable length of the current sub-file file(x); S2: Update the offset address to the current offset address plus the length of the data that has been read / written, and update the length of the data to be read / written to the remaining unread / written length; S3: Repeat steps 3 to 6 until all data is read / written.

2. The method for subcontracting and storing transaction records according to claim 1, wherein The fixed length of each sub-file in step 1 is determined according to the length of a single transaction record and a preset sub-packaging strategy.

3. A method for subcontracting and storing transaction records according to claim 1, characterized in that, In step 3, when the index number is an input parameter, the offset address is obtained by multiplying the index number by the length of a single record.

4. A method for subcontracting and storing transaction records according to claim 1, characterized in that, In step 6, when reading / writing across sub-files, a circular overwrite strategy and a dynamic expansion strategy are adopted, specifically including: if the number of sub-files has reached the upper limit m, overwrite the oldest sub-file; if dynamic expansion is allowed, add a new sub-file file(m + 1) and update the value of m.

5. A method for subcontracting and storing transaction records according to claim 1, characterized in that, The storage path of the sub-files is dynamically allocated based on a hash algorithm and a load balancing strategy.

6. A method for subcontracting and storing transaction records according to claim 1, characterized in that, The method is applicable to an embedded file system, and the metadata of each sub-file is stored independently to support fast recovery.

7. A method for subcontracting storage of transaction records according to claim 1, characterized in that, Before read / write operations, optimize the access efficiency of sub-files by pre-allocating continuous disk space and memory buffers.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory. When the processor executes the computer program, it implements the method described in any one of claims 1 to 7.

Images