A method for optimizing the storage of historical financial transaction data

Abstract

This invention relates to the field of financial data storage optimization technology, and in particular to a method for optimizing the storage of historical financial transaction data. The method includes: obtaining the access count and modification frequency of a data packet to be classified; obtaining a data packet to be archived; calculating the initial archiving priority and expected archiving time of the data packet to be archived; determining the archiving time interval corresponding to the data packet to be archived; adjusting a preset archiving sequence number to the mapped archiving sequence number; obtaining an archiving time comparison result; and, upon obtaining the first archiving time comparison result, determining the category of the storage method and the storage operation corresponding to that category, and storing the data to be archived. This invention, by comprehensively evaluating access index values ​​and storage costs, and utilizing the archiving time comparison result for adaptive optimization decisions, achieves refined management and efficient storage of historical financial transaction data archiving, effectively improving storage resource utilization and system operating efficiency.

Description

Technical Field

[0001] This invention relates to the field of financial data storage optimization technology, and in particular to a method for optimizing the storage of historical financial transaction data. Background Technology

[0002] With the increasing digitalization of the financial industry, the volume of transaction data generated by banks, payment institutions, and other financial service platforms is exploding. Historical financial transaction data is not only massive in quantity but also complex in structure, involving multi-dimensional characteristics such as peak access ratio, access cycle coverage, transaction time, and risk level. This data is of significant value in storage analysis, risk management, customer analysis, and compliance supervision. However, existing financial transaction data storage solutions have significant shortcomings. A large amount of historical transaction data lacks tiered storage and optimized processing during archiving, resulting in low utilization of storage resources.

[0003] Existing methods often rely on fixed time windows or simple rules for archiving, without fully considering multi-dimensional characteristics such as peak access ratio, access cycle coverage, transaction time freshness, and cross-type access ratio. This can lead to delayed storage of high-load data, affecting data availability and audit efficiency.

[0004] Furthermore, archived data is generally stored using a single method, lacking intelligent optimization techniques tailored to data characteristics and historical access value, thus failing to balance storage costs and data availability.

[0005] Therefore, there is an urgent need for a method that can comprehensively consider access characteristic parameters and storage characteristic parameters to scientifically evaluate, hierarchically archive, and optimize the storage of historical financial transaction data, so as to improve the utilization of storage resources, ensure that high-load data is stored first, and meet the requirements of financial data compliance and auditing.

[0006] Chinese Patent Publication No. CN118244994A discloses a historical data storage method and apparatus based on cloud computing, comprising: collecting historical financial datasets and extracting features to obtain transaction feature data; constructing a target cloud resource allocation strategy and generating a data storage priority table based on the transaction feature data; collecting account transaction data of target users and matching the account transaction data with cloud services through the target cloud resource allocation strategy to obtain multiple initial data storage locations; optimizing the layout of the multiple initial data storage locations based on the data storage priority table to obtain data storage layout information, and storing the account transaction data based on the data storage layout information; obtaining data query requests from target users and performing data queries based on the data storage layout information, and returning the target query results. It is evident that existing financial data storage optimization technologies typically employ only a single storage strategy, failing to flexibly adjust storage methods according to transaction and storage characteristics, easily leading to storage latency and low resource utilization. Furthermore, they lack support for customizing storage content and compression ratios under different business environments, making it difficult to meet diverse financial business needs. Summary of the Invention

[0007] To address this, the present invention provides a method for optimizing the storage of historical financial transaction data. This method combines access characteristic parameters and storage characteristic parameters to enable flexible selection between direct storage and compressed storage operations, and allows for customizable storage content and compression ratios under different storage operations. This overcomes the problems of high-load data access latency, low storage resource utilization, and inability to adapt to diverse business needs caused by a single storage strategy in the prior art.

[0008] To achieve the above objectives, the present invention provides a method for optimizing the storage of historical financial transaction data, comprising:

[0009] Get the access count and modification frequency of several unarchived, unclassified data packets in the historical financial transaction dataset that will be stored within the current archiving time.

[0010] When the number of accesses exceeds a preset access threshold or the modification frequency exceeds a preset modification frequency threshold, the corresponding data packet to be classified is updated to a data packet to be archived. Any data packet to be archived includes a preset archive time, a preset archive sequence number, and data to be archived.

[0011] Based on the access characteristic parameters and storage characteristic parameters in the data to be archived, the initial archiving priority and the expected archiving time of the data packet to be archived are calculated.

[0012] Based on the sequence number adjustment rules, obtain the mapped archive sequence number corresponding to the initial archive priority, and adjust the preset archive sequence number to the mapped archive sequence number;

[0013] Obtain the archiving time interval between each archiving time interval, and determine the archiving time interval corresponding to the data packet to be archived based on the expected archiving time and the mapped archiving sequence number;

[0014] The estimated archiving time of the data packet to be archived is compared with the archiving time threshold to obtain the archiving time comparison result;

[0015] Upon obtaining the first archiving time comparison result, the storage method category and the storage operation corresponding to the storage method category are determined based on the storage characteristic parameters of the data packet to be archived corresponding to the first archiving time comparison result, and the data to be archived is stored.

[0016] Furthermore, the steps of calculating the initial archiving priority and expected archiving time of the data packet to be archived include:

[0017] The access metric score and the storage cost evaluation value are calculated based on the access characteristic parameters and the storage characteristic parameters, respectively.

[0018] The initial archiving priority of the data packet to be archived is obtained by weighting the calculated access metric score and storage cost evaluation value according to a preset weight;

[0019] Based on the functional mapping relationship between the storage cost assessment value and the expected archiving time interval, and in conjunction with the initial archiving priority, the expected archiving time interval of the data packet to be archived is calculated.

[0020] Furthermore, the step of calculating the access index score includes:

[0021] Obtain the peak access ratio, access cycle coverage, access timestamp, and cross-type access ratio from the access feature parameters;

[0022] The peak access ratio, access cycle coverage, access timestamp, and cross-type access ratio are quantified and standardized, and then input into the access indicator value evaluation function to obtain the corresponding access indicator score.

[0023] Furthermore, the access metric value evaluation function includes:

[0024] Calculate the access metric values ​​corresponding to the access cycle coverage, the access timestamp, and the cross-type access ratio after quantification and standardization.

[0025] The access metric score is obtained by weighting the access period coverage, the access timestamp, and the cross-type access ratio.

[0026] Furthermore, the step of calculating the storage cost assessment value includes:

[0027] Obtain the data size, data structure complexity, data encryption strength, and data verification strength from the storage feature parameters;

[0028] The data size, data structure complexity, data encryption strength, and data verification strength are quantified and standardized, and then input into the storage cost evaluation function to obtain the corresponding storage cost evaluation value.

[0029] Furthermore, the steps to obtain the archiving time comparison results include:

[0030] The estimated archiving time of the data packet to be archived is compared with the archiving time threshold to obtain the archiving time comparison.

[0031] If the estimated archiving time corresponding to any data packet to be archived is greater than or equal to the archiving time threshold, the first archiving time comparison result is obtained;

[0032] If the estimated archiving time for all data packets to be archived is less than the archiving time threshold, the second archiving time comparison result is obtained.

[0033] Furthermore, the steps for determining the category to which the storage method belongs and the storage operation corresponding to that category include:

[0034] When the first storage method is obtained, the data packet to be archived is directly stored at the corresponding archiving time;

[0035] When the second storage method is obtained, the compression ratio of the data to be archived and the compressed storage data to be archived corresponding to the compression ratio of the data to be archived are obtained based on the compression ratio calculation formula, and the compression storage operation is performed on the compressed storage data to be archived.

[0036] Furthermore, the steps for performing a direct storage operation include:

[0037] When the current archiving time arrives, each data packet to be archived is written to the storage medium according to the mapped archiving sequence number and at intervals of the archiving time interval.

[0038] Furthermore, the steps for obtaining the compression ratio of the archived data for compressed storage based on the compression ratio calculation formula include:

[0039] Based on the data encryption strength, the data structure complexity, and the data size, the compression ratio of the data to be archived is calculated using the compression ratio calculation formula.

[0040] Obtain the corresponding data of the same type under each periodic system job type in the data to be archived, sort the data of the same type according to the data size to obtain the sorting result, and determine the compression execution order according to the sorting result;

[0041] Determine whether there is data with a custom compression storage operation category in the data to be archived, and calculate the proportion of the custom compression portion to the total compression ratio;

[0042] When the percentage of the custom compression portion does not reach the preset compression ratio threshold, compression is performed on the largest portion of the data to be archived until the compression ratio reaches or exceeds the preset compression ratio threshold.

[0043] Furthermore, the steps for performing the compressed storage operation include:

[0044] The data to be archived is divided based on the periodic system job type to obtain several periodic job coverage datasets;

[0045] In the datasets covered by each cycle of work, the data to be archived of the same type are sorted according to the data size in the storage feature parameters;

[0046] The compression execution order of the datasets covered by each cycle job is determined based on the sorting results;

[0047] The compression ratio is matched with the archived compressed storage data corresponding to the compression ratio of the data to be archived, and the compression storage operation is performed.

[0048] Compared with existing technologies, the advantages of this invention lie in its ability to achieve dual-dimensional intelligent decision-making for optimizing the storage of historical financial transaction data by constructing an access indicator value evaluation function and a storage cost evaluation function. The method first performs refined quantification and weighted calculation on characteristic parameters such as peak access ratio, access cycle coverage, access timestamp, and cross-type access ratio, scientifically reflecting the contribution of each parameter to data value and ensuring the priority and urgency of high-load data in archiving. Furthermore, it utilizes an archiving time threshold to achieve real-time adaptive judgment of the timeliness of archiving operations, avoiding indiscriminate storage optimization for all data packets and effectively reducing unnecessary computational resources. When optimization is needed, the system dynamically selects between direct storage and compressed storage based on the storage cost assessment value. For compressed storage, an adaptive compression ratio calculation formula based on data characteristics is introduced. This formula combines data structure complexity, encryption strength, and data size to accurately predict compression benefits. By classifying transactions by category and sorting data in descending order of size, efficient execution of compression operations is ensured, achieving maximum space savings and reduced archiving time with minimal resource investment. Ultimately, this method significantly improves storage resource utilization, reduces the overall storage system's operation and maintenance costs, and ensures the real-time and accurate archiving of core transaction data.

[0049] Furthermore, by quantifying and standardizing the peak access ratio, access cycle coverage, access timestamp, and cross-type access ratio, and combining the concurrent transaction behavior and time characteristics in historical transaction data, an access indicator value evaluation function is established to evaluate each piece of historical transaction data; this scientifically reflects the impact of several parameters on storage value, improves storage resource utilization, optimizes archiving strategies, and enhances the system's management and scheduling efficiency for transaction data. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the method for optimizing the storage of historical financial transaction data according to an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram illustrating the process of calculating the initial archiving priority and expected archiving time of the data packet to be archived according to an embodiment of the present invention;

[0052] Figure 3 This is a logical decision diagram for obtaining the archiving time comparison results in an embodiment of the present invention;

[0053] Figure 4 This is a flowchart illustrating the process of determining the category of a storage method and the storage operation corresponding to that category, according to an embodiment of the present invention.

[0054] Figure 5 This is a schematic diagram illustrating the process of performing compressed storage operations according to an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0056] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0057] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0058] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0059] Please see Figure 1 As shown, it is a flowchart illustrating the method for optimizing the storage of historical financial transaction data according to an embodiment of the present invention. The present invention provides a method for optimizing the storage of historical financial transaction data, including:

[0060] Step S1: Obtain the access count and modification frequency of several unarchived data packets to be classified in the historical financial transaction dataset that will be stored within the current archiving time;

[0061] Step S2: When the number of accesses is greater than the preset access number threshold or the modification frequency is greater than the preset modification frequency threshold, the corresponding data packet to be classified is updated to a data packet to be archived. Any data packet to be archived includes a preset archive time, a preset archive sequence number, and data to be archived.

[0062] Step S3: Calculate the initial archiving priority and estimated archiving time of the data packet to be archived based on the access feature parameters and storage feature parameters in the data to be archived;

[0063] Step S4: Based on the sequence number adjustment rule, obtain the mapped archive sequence number corresponding to the initial archive priority, and adjust the preset archive sequence number to the mapped archive sequence number;

[0064] Step S5: Obtain the archiving time interval between each archiving time interval, and determine the archiving time interval corresponding to the data packet to be archived based on the expected archiving time and the mapped archiving sequence number;

[0065] Step S6: Compare the estimated archiving time of the data packet to be archived with the archiving time threshold to obtain the archiving time comparison result;

[0066] Step S7: When the first archiving time comparison result is obtained, the storage method category and the storage operation corresponding to the storage method category are determined according to the storage characteristic parameters of the data packet to be archived corresponding to the first archiving time comparison result, and the data to be archived is stored.

[0067] In this embodiment, the data packets to be classified are initial unarchived data, i.e., data in an active or changing period; the number of accesses is a measure of the query frequency of the data packets within a specific time window; the modification frequency is used to filter out high-load data packets and determine which data is forcibly included in the archiving process due to frequent changes; the historical financial transaction dataset is a collection of all completed or canceled financial transaction records accumulated by financial institutions or trading platforms within a specific period, and is the original data source to be archived and stored.

[0068] Access count is a measure of the total frequency with which a single data packet to be classified is queried, read, or referenced within a preset specific time window;

[0069] The preset access frequency threshold is A times the average access frequency of the dataset, with A ranging from [1.5, 3.0]; the preferred value is 2.0. The preset modification frequency threshold is B times the average modification frequency of the dataset, with A ranging from [1.0, 2.0], and the preferred value is 1.7.

[0070] The data package to be archived is the processing unit prepared for storage optimization; the preset archiving time is the theoretical archiving time initially set by the system based on the data generation time or the current archiving batch time; the preset archiving sequence number is the initial sequence number obtained by the data package to be archived according to its generation or collection order when it enters the storage optimization process; the data to be archived is the actual financial transaction record content contained in the data package to be archived; the access characteristic parameters are a set of parameters used to evaluate the value and importance of the transaction record itself, including peak access ratio, access cycle coverage, access timestamp, and cross-type access ratio; the storage characteristic parameters are a set of parameters used to evaluate the resources and complexity required for storage and subsequent access of the data to be archived, including data size, data structure complexity, data encryption strength, and data verification strength; the initial archiving priority is based on the access characteristic parameters and storage characteristics. The parameters are initially calculated to provide a quantitative indicator reflecting the urgency and importance of storing the data packets to be archived; the estimated archiving time is the time required for the data packets to actually complete the archiving operation under the current system load and storage policy, predicted based on the initial archiving priority and storage cost assessment; the mapped archiving sequence number is the archiving sequence number ultimately determining the writing order of the data packets, mapped from the initial archiving priority under the sequence number adjustment rule; the archiving time interval is the set time length between two adjacent archiving time intervals, used to refine and schedule the execution time of archiving operations; the archiving time interval is a reserved time period to accommodate one or a group of data packets to be archived to complete the writing operation; the archiving time threshold is a time critical value used to determine whether the data packets to be archived need to undergo storage optimization, such as compressed storage, reflecting the system's requirements for the real-time performance of storage operations.

[0071] By constructing access indicator evaluation functions and storage cost evaluation functions, a two-dimensional intelligent decision-making system for optimizing the storage of historical financial transaction data is achieved. The method first performs refined quantification and weighted calculation of characteristic parameters such as peak access ratio, access cycle coverage, access timestamp, and cross-type access ratio, scientifically reflecting the contribution of each parameter to data value and ensuring the priority and urgency of high-load data in archiving. It then utilizes archiving time thresholds to achieve real-time adaptive judgment of the timeliness of archiving operations, avoiding indiscriminate storage optimization of all data packets and effectively reducing unnecessary computational resource consumption. When optimization is deemed necessary, the system dynamically selects direct storage or compressed storage based on the storage cost evaluation value, and introduces an adaptive compression ratio calculation formula based on data characteristics for compressed storage. This formula combines data structure complexity, encryption strength, and data size to accurately predict compression benefits. By classifying transactions by category and sorting data by size in descending order, efficient execution of compression operations is ensured, achieving maximum space savings and reduced archiving time with minimal resource investment. Ultimately, the method significantly improves the utilization rate of storage resources, reduces the overall operation and maintenance costs of the storage system, and ensures the real-time performance and accuracy of core transaction data archiving.

[0072] See Figure 2 As shown, it is a schematic diagram of the process of calculating the initial archiving priority and expected archiving time of the data packet to be archived in an embodiment of the present invention;

[0073] Specifically, the steps for calculating the initial archiving priority and expected archiving time of the data packet to be archived include:

[0074] Step S3.1: Calculate the access metric score and storage cost assessment value based on the access characteristic parameters and the storage characteristic parameters, respectively;

[0075] Step S3.2: Calculate the initial archiving priority of the data packet to be archived by weighting the calculated access metric score and storage cost evaluation value according to a preset weight;

[0076] Step S3.3: Based on the functional mapping relationship between the storage cost assessment value and the expected archiving time interval, and in conjunction with the initial archiving priority, calculate the expected archiving time interval of the data packet to be archived.

[0077] Specifically, the steps for calculating the access metric score include:

[0078] Obtain the peak access ratio, access cycle coverage, access timestamp, and cross-type access ratio from the access feature parameters;

[0079] After quantification and standardization, the peak access ratio, the access cycle coverage, the access timestamp, and the cross-type access ratio are input into the access indicator value evaluation function to obtain the corresponding access indicator score.

[0080] In this embodiment, the peak access ratio includes the maximum peak access ratio, the minimum peak access ratio, and the current peak access ratio; wherein, the peak access ratio is the fluctuation of access activity within a specific time window, and is the ratio of the maximum instantaneous I / O to the average I / O request volume within the monitoring period;

[0081] The peak access ratio is normalized by the maximum and minimum to obtain the standard peak access ratio, which takes a value range of [0,1] after standardization.

[0082] The standard peak access ratio is calculated as the ratio of the difference between the current peak access ratio and the minimum peak access ratio to the difference between the maximum peak access ratio and the minimum peak access ratio.

[0083] Access cycle coverage is used to measure how often data packets are referenced by different types of periodic jobs or automated access flows in the system, reflecting the importance of data in automated processing within the system. Qualitative variables are converted into binary vectors through one-hot encoding.

[0084] Focusing on key periodic system access tasks, the following are examples: End-of-Day Clearing Access (DCA): Whether the data is accessed for daily batch data processing such as aggregation and verification; Weekly Data Verification Access (WVA): Whether the data is accessed for weekly data integrity or consistency verification; Monthly Archive Preprocessing Access (MPA): Whether the data is accessed for monthly data migration or pre-archiving jobs; Cross-Data Source Synchronization Access (CSA): Whether the data is used for cross-system data synchronization or mirroring; Storage Resource Allocation Calculation Access (SRC): Whether the data is used for jobs that calculate storage resources or billing; Encoding Mapping: Each category has a unique code in the system and is represented in vector form with fixed dimensions, for example, DCA is [1,0,0,0,0];

[0085] After encoding, the parameter dimensions are fixed and correspond one-to-one with the set of access cycle coverages registered in the system's business dictionary to ensure the comparability and independence of different access cycle coverages;

[0086] Within the current archiving period, retrieve the access timestamps corresponding to all transaction records from the historical financial transaction dataset that will be stored at the current archiving time;

[0087] The earliest and latest times in the access timestamps are determined and used as the boundaries of the time quantization interval. Each access timestamp is normalized according to its relative position within the time interval to obtain a transaction time weight parameter with a quantization value range of [0,1]. Among them, the transaction record closer to the current time has a corresponding quantization value of 1, and the transaction record farther from the current time has a corresponding quantization value of 0. The quantization value of the transaction record located between the two is linearly distributed according to its relative time position within the time interval.

[0088] Cross-type access ratio is used to characterize the breadth of target data accessed by different technical entities within the system, reflecting its universality and system dependence, and is quantified based on the time characteristics corresponding to the access timestamp and its concurrent transaction behavior;

[0089] Within the current archiving cycle, the system obtains the I / O activity timestamp information for each data packet to be archived; within this time interval, it counts the number of other data access requests that occur under the same application process identifier or the same storage node identifier, and obtains the concurrent access count corresponding to the data packet;

[0090] The system has a preset access security reference ratio, which is used to characterize the threshold of the number of concurrent accesses allowed for a single data packet in normal access mode;

[0091] The concurrent access ratio is calculated as the ratio of the number of concurrent accesses to the access security reference value. This ratio is then converted into an access risk level according to a non-linear hierarchical mapping rule, as follows:

[0092] When the concurrent access ratio is ≤0.5, the corresponding risk level is 1, the access isolation is high, the concurrency is less than half of the expected security threshold, and the access behavior is highly standardized.

[0093] If the concurrent access ratio is 0.5 < 1.0, the corresponding risk level is 2, the concurrency is at or below the safety threshold, the behavior is normal, and the system load is controllable.

[0094] 1.0 < concurrent access ratio ≤ 1.5, corresponding to risk level = 3, slightly exceeding the safety threshold, with some pressure and contention in access I / O;

[0095] A concurrent access ratio of 1.5 < 2.5 corresponds to a risk level of 4, which significantly exceeds the safety threshold and indicates a high risk of concurrent I / O conflicts and resource contention.

[0096] 2.5 < concurrent access ratio ≤ 4.0, corresponding to risk level = 5, extremely high concurrency, indicating that the data packet is a hot access target, which may cause I / O blocking;

[0097] A concurrent access ratio >4 corresponds to a risk level of 6, indicating an abnormal level of concurrency, which may involve malicious access or system anomalies.

[0098] The risk level is converted into a quantified risk value using a linear normalization method. The quantification formula is as follows:

[0099] Risk quantification value = (risk level - 1) / 5, with a value range of [0, 1].

[0100] By linearly normalizing the risk levels according to the interval [1,6], discrete risk levels can be mapped to a continuous quantitative interval [0,1]. This allows for a smooth transition of risk weights between different levels during subsequent evaluation of access index values, avoiding abrupt changes in the evaluation function results due to level jumps.

[0101] By quantifying and standardizing peak access ratio, access cycle coverage, access timestamp, and cross-type access ratio, and combining concurrent transaction behavior and time characteristics in historical transaction data, an access indicator value evaluation function is established to evaluate each piece of historical transaction data; this scientifically reflects the impact of several parameters on storage value, improves storage resource utilization, optimizes archiving strategies, and enhances the system's management and scheduling efficiency for transaction data.

[0102] Specifically, the access metric value evaluation function includes:

[0103] Calculate the standard access cycle coverage, the standard access timestamp, and the standard cross-type access ratio obtained after quantization and standardization processing;

[0104] The access metric score is obtained by weighting the access period coverage, the access timestamp, and the cross-type access ratio.

[0105] In this embodiment, the formula for calculating the access metric score is as follows: ;

[0106] in, The standard peak access ratio weight is a contribution coefficient to access intensity and is a preset weight.

[0107] The standard access peak ratio is derived from feature parameter quantization;

[0108] The access cycle coverage rate has a preset weight;

[0109] The access cycle coverage storage value component is calculated through the dot product of the encoded vectors;

[0110] The data timeliness weighting coefficient is a preset weight;

[0111] Data timeliness weighting comes from feature parameter quantification;

[0112] The risk quantification value weights are preset weights;

[0113] The risk quantification value for access behavior is derived from the quantification of feature parameters;

[0114] Access cycle coverage storage value component The calculation is performed by accessing the periodic coverage encoded vector. Its corresponding weight vector The result is obtained by performing a dot product calculation, i.e.: ;

[0115] Each element of the weight vector is assigned based on the coverage of each access cycle in historical data, i.e., the frequency of occurrence of system job types, to reflect its contribution to system resource consumption and storage value; the statistical range is limited to the time interval formed between the earliest and latest times in the access timestamps.

[0116] Within this time interval, calculate the total number of times each type of system automation task, such as DCA, WVA, and MPA, accesses data packets. ;

[0117] The total number of accesses is normalized and mapped to the [0,1] interval to obtain the weight vector corresponding to the coverage encoding vector of each access cycle. ;

[0118] Assume there are five key system access job types: End-of-Day Clearing Access (DCA), Weekly Data Verification Access (WVA), Monthly Archive Preprocessing Access (MPA), Cross-Data Source Synchronization Access (CSA), and Storage Resource Allocation Calculation Access (SRC).

[0119] Statistical results: The total number of accesses for five types of jobs was calculated between the earliest and latest times. For example:

[0120] DCA access count: 1000; WVA access count: 5000; MPA access count: 500; CSA access count: 2000; SRC access count: 100;

[0121] The maximum occurrence count of 5000 corresponds to WVA, and the minimum occurrence count of 100 corresponds to SRC; the occurrence count is mapped to the [0,1] interval using minimum-maximum normalization, resulting in... The weights of each component of the vector;

[0122] ;

[0123] When the access cycle encoding vector C of a data packet to be archived is DCA coverage [1,0,0,0,0], the access cycle coverage storage value component The result is obtained through dot product calculation: ;

[0124] All the weight coefficients take values ​​in the range [0,1], and the sum of all the weight coefficients equals 1. ;

[0125] Preferred value ;

[0126] Weight the access cycle coverage and risk quantification value weight Setting it to a relatively high value, totaling 0.65, reflects the system's breadth of data dependency and the highest level of attention given to concurrent I / O risks; peak access ratio weight. Basic focus on instantaneous I / O pressure; data timeliness weighting coefficient A moderate level indicates that the timeliness of data is less important than system dependence and concurrency risks, but it is still a key consideration for optimizing storage.

[0127] By weighting the standard peak access ratio, access cycle coverage storage value component, transaction time weight, and risk quantification value, the data's business importance, risk compliance, historical frequency, and timeliness are comprehensively reflected. In particular, by setting the access cycle coverage weight and risk quantification value weight to relatively high values, the storage priority of core business types and high-risk transaction data is ensured. The dynamic weight allocation of access cycle coverage using historical frequency improves the adaptability and accuracy of the assessment.

[0128] Specifically, the step of calculating the storage cost assessment value includes:

[0129] Obtain the data size, data structure complexity, data encryption strength, and data verification strength from the storage feature parameters;

[0130] The data size, data structure complexity, data encryption strength, and data verification strength are quantified and standardized, and then input into the storage cost evaluation function to obtain the corresponding storage cost evaluation value.

[0131] In this embodiment, the data size is the actual storage capacity of the data packet to be archived, usually in bytes or megabytes.

[0132] Data structure complexity describes the degree of structuring of the data to be archived. For example, fully structured data is such as relational database tables, semi-structured data is such as JSON or XML, and unstructured data is such as images or documents.

[0133] Data encryption strength is the strength level of the encryption algorithm used on the data to be archived before storage, such as AES-128 or AES-256.

[0134] Data verification strength refers to the strength or complexity of the verification and redundancy backup mechanisms used to ensure data integrity and reliability.

[0135] The size of the acquired data, the complexity of the data structure, the data encryption strength, and the data verification strength need to be quantified and standardized.

[0136] Since the numerical differences in data size can be extremely large, we first need to consider the original data size. Perform logarithmic transformation Using the min-max standardization method, Mapping to the [0,1] interval yields the standardized data size. The value range is [0,1].

[0137] The data structure complexity is categorized into levels and assigned numerical values; for example, fully structured data is quantized to 0.2, semi-structured data to 0.5, and unstructured data to 0.8. Since the quantized values ​​are within the range [0,1], they can be directly used as the standardized data structure complexity. The value range is [0,1].

[0138] A numerical value is assigned based on the strength of the encryption algorithm; for example, ... Quantified to 0.5 The quantization is set to 0.8, while a stronger encryption algorithm is quantized to 1.0; this is directly used as the standardized data encryption strength. The value range is [0,1]. The higher the encryption strength, the greater the computational overhead during writing and reading, and the higher the storage cost evaluation value.

[0139] Values ​​are assigned based on the checksum and redundancy level; for example, the basic checksum is assigned a value based on the single checksum level. Quantization is set to 0.3, dual backup verification is quantized to 0.7, and multiple redundancy encoding verification is, for example... Quantized to 1.0; directly used as the validation strength of the standardized data. The value range is [0,1]; the higher the verification strength, the higher the cost of storing additional metadata and performing verification operations;

[0140] The data size, data structure complexity, data encryption strength, and prime number data verification strength, after quantification and standardization, are fed into a storage cost evaluation function as input variables. The function performs a weighted calculation on each input variable and finally outputs a storage cost evaluation value. The evaluation value is a value in the range [0,1], and the higher the value, the higher the storage and management cost of the data to be archived.

[0141] The storage cost evaluation function can be expressed in the following weighted summation form: ;

[0142] in This is the final storage cost assessment value;

[0143] The sum of all weights equals 1;

[0144] Select data size weights based on the storage medium's capacity bottleneck. For example, it can be set to 0.4; in scenarios where storage capacity is the main bottleneck, data size contributes the most to storage costs.

[0145] Select the data encryption strength weight based on the system's CPU resource constraints. For example, it can be set to 0.2; this is to reflect the significant consumption of computing resources by encryption and decryption operations.

[0146] Choose the data structure complexity weight based on the complexity of data querying and management. For example, set it to 0.2;

[0147] Select the data verification strength weight based on data reliability requirements and verification mechanism overhead. For example, set it to 0.2.

[0148] By weighting data size, data structure complexity, data encryption strength, and data verification strength, the system comprehensively reflects storage resource consumption and system load. In particular, based on the actual bottlenecks of the storage system, such as capacity bottlenecks, the weights of each item are dynamically adjusted to ensure that the evaluation value truly reflects the current storage cost of the system.

[0149] See Figure 3 As shown, it is a logical decision diagram for obtaining the archiving time comparison results in an embodiment of the present invention;

[0150] Specifically, the steps to obtain the archive time comparison results include:

[0151] The estimated archiving time of the data packet to be archived is compared with the archiving time threshold to obtain the archiving time comparison.

[0152] If the estimated archiving time corresponding to any data packet to be archived is greater than or equal to the archiving time threshold, the first archiving time comparison result is obtained;

[0153] If the estimated archiving time for all data packets to be archived is less than the archiving time threshold, the second archiving time comparison result is obtained.

[0154] In this embodiment, the archiving time threshold is a preset time threshold used to measure whether the data packet archiving operation meets the real-time requirements; the ideal range of the archiving time threshold depends on the storage system performance and business requirements, and is usually between 500 milliseconds and 3 seconds; in this embodiment, the archiving time threshold is selected as 1.5 seconds.

[0155] By setting a preset archiving time threshold, the system enables real-time adaptive judgment of the timeliness of archiving tasks; it avoids indiscriminate storage optimization for all data packets, triggering complex optimization processes only when the expected archiving time exceeds the threshold; it significantly reduces unnecessary consumption of computing resources and improves the overall efficiency and resource utilization of the archiving process.

[0156] See Figure 4 As shown, it is a flowchart illustrating the process of determining the category of the storage method and the storage operation corresponding to the category of the storage method in an embodiment of the present invention;

[0157] Specifically, the steps for determining the category of the storage method and the storage operation corresponding to that category include:

[0158] Step S7.1: When the first storage method is obtained, perform a direct storage operation on the data packet to be archived at the corresponding archiving time;

[0159] Step S7.2: When the second storage method is obtained, the compression ratio of the data to be archived and the compressed storage data to be archived corresponding to the compression ratio of the data to be archived are obtained based on the compression ratio calculation formula, and the compression storage operation is performed on the compressed storage data to be archived.

[0160] Specifically, the steps for performing a direct store operation include:

[0161] When the current archiving time arrives, each data packet to be archived is written to the storage medium according to the mapped archiving sequence number and at intervals of the archiving time interval.

[0162] In this embodiment, when the current archiving time arrives, the system calls the preset mapped archiving sequence number and loads each data packet to be archived into the write cache in the order of the mapped archiving sequence number. Then, in units of archiving time interval, each data packet to be archived is written into the corresponding archiving time interval in sequence. When each archiving time interval arrives, the system retrieves the data packet with the corresponding sequence number from the write cache and completes one archiving write operation.

[0163] By executing the storage steps sequentially, each data packet to be archived is written sequentially at fixed time intervals in direct storage mode, ensuring the consistency between the writing timing and the archiving sequence, and improving the controllability of the archiving process and the stability of data storage.

[0164] Specifically, the steps to obtain the compression ratio of the archived data for compressed storage based on the compression ratio calculation formula include:

[0165] Based on the data encryption strength, the data structure complexity, and the data size, the compression ratio of the data to be archived is calculated using the compression ratio calculation formula.

[0166] Obtain the corresponding data of the same type under each periodic system job type in the data to be archived, sort the data of the same type according to the data size to obtain the sorting result, and determine the compression execution order according to the sorting result;

[0167] Determine whether there is data with a custom compression storage operation category in the data to be archived, and calculate the proportion of the custom compression portion to the total compression ratio;

[0168] When the percentage of the custom compression portion does not reach the preset compression ratio threshold, compression is performed on the largest portion of the data to be archived until the compression ratio reaches or exceeds the preset compression ratio threshold.

[0169] In this embodiment, the formula for calculating the compression ratio is as follows: ;

[0170] Compression ratio : Indicates the proportion of the compressed data packet size to the original size, with a value range of [0,1];

[0171] Data structure complexity The standardized data structure complexity has a value range of [0,1].

[0172] Data encryption strength The encryption strength of the standardized data ranges from [0,1]; encryption will reduce compression efficiency.

[0173] Data size S': The standardized data size, with a value range of [0,1].

[0174] Compression efficiency coefficient This coefficient adjusts the degree to which the complexity of the data structure affects the compression effect; based on industry experience with data compression algorithms and data structure characteristics, its ideal value range is [0.4, 0.9]; in this embodiment, it is preferably selected as 0.7 based on historical test results of financial unstructured data.

[0175] Encryption affects weight This weight represents the degree of negative impact of encryption strength on compression efficiency; based on the sensitivity of the storage system CPU to encryption and decryption operations, its ideal value range is [0.1, 0.4]; in this embodiment, it is preferably selected as 0.2 based on the current system's CPU load model.

[0176] Data size affects weight This weight represents the degree of positive impact of data size on potential compression benefits; based on practical experience in large-scale data storage, its ideal value range is [0.1, 0.4]; in this embodiment, it is preferably selected as 0.2 based on the historical data distribution characteristics of the archiving system.

[0177] Obtain the corresponding data of the same type under each periodic system job type in the data to be archived; sort the data of the same type in descending order according to the data size to obtain the sorting result; determine the compression execution order according to the sorting result, that is, the data of the same type with the largest data size is compressed first, in order to achieve the highest overall compression efficiency with the fewest compression operations.

[0178] Determine whether there is data with a custom compression storage operation category in the data to be archived; if so, calculate the proportion of the custom compression portion to the total compression ratio;

[0179] The preset compression ratio threshold refers to the minimum overall compression ratio target that the system aims to achieve during compressed storage operations. The ideal value range depends on the resource constraints of the storage system and the archiving time requirements, and is between [0.1, 0.6]. In this embodiment, 0.3 is selected.

[0180] When the percentage of the custom compression portion does not reach the preset compression ratio threshold, compression is performed on the largest portion of the data to be archived until the overall compression ratio reaches or exceeds the preset compression ratio threshold.

[0181] By adaptively selecting storage methods such as direct storage or compressed storage based on the business importance of the data, risk compliance, and the actual cost and performance bottlenecks of storage resources; initiating fine-grained compression ratio calculations when archiving time exceeds a threshold, and prioritizing the processing of large-size data, storage goals are achieved efficiently; storage resource utilization and system operating efficiency are significantly improved, and the overall operation and maintenance costs of the storage system are reduced.

[0182] See Figure 5 As shown, it is a flowchart illustrating the compression and storage operation performed in an embodiment of the present invention;

[0183] Specifically, the steps for performing compressed storage operations include:

[0184] Step S7.2.1: Divide the data to be archived based on the periodic system job type to obtain several periodic job coverage datasets;

[0185] Step S7.2.2: In the datasets covered by each periodic job, sort the data to be archived of the same type according to the data size in the storage feature parameters;

[0186] Step S7.2.3: Determine the compression execution order of the datasets covered by each cycle job based on the sorting results;

[0187] Step S7.2.4: Match the compression ratio with the archived compressed storage data corresponding to the compression ratio of the data to be archived, and perform the compression storage operation.

[0188] In this embodiment, based on the periodic system job type, such as DCA, WVA, MPA, etc., the data to be archived is divided to obtain several periodic job coverage datasets;

[0189] In each of the periodic job coverage datasets, the data to be archived of the same type are sorted in descending order according to the data size in the storage feature parameters; the purpose of the sorting is to prioritize the processing of the largest data packet in order to achieve the highest compression efficiency.

[0190] Based on the sorting results, the compression execution order of the datasets covered by each periodic job is determined; the periodic job dataset with the largest data size and the largest data packet within that dataset will be compressed first.

[0191] Obtain the compression ratio α of the data to be archived obtained from the compression ratio calculation formula in the aforementioned steps, and the compressed storage data to be archived corresponding to the compression ratio; the compressed storage data to be archived is obtained by applying a specific compression algorithm, such as LZ4, to the original data to be archived.

[0192] The system writes the compressed and stored data to be archived into a predetermined archive storage medium according to a determined compression execution order;

[0193] The specific steps are as follows: compress and encode the data to be archived and compressed.

[0194] The encoded content, along with the compressed storage data to be archived and the necessary decompression metadata, is written to the target storage medium, such as a secondary high-capacity disk array.

[0195] By classifying and sorting similar transaction data in descending order, the compression operation is made efficient. Prioritizing the compression of the largest data of the same type allows for the highest space saving and archiving time reduction with minimal computational resources. The compression storage operation effectively transfers high-storage-cost data to high-density, low-cost storage media, significantly alleviating the resource pressure on high-performance storage. Furthermore, by carrying necessary decompression metadata, the accuracy and efficiency of future data access and decompression are guaranteed.

[0196] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0197] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing the storage of historical financial transaction data, characterized in that, include: Get the access count and modification frequency of several unarchived, unclassified data packets in the historical financial transaction dataset that will be stored within the current archiving time. When the number of accesses exceeds a preset access threshold or the modification frequency exceeds a preset modification frequency threshold, the corresponding data packet to be classified is updated to a data packet to be archived. Any data packet to be archived includes a preset archive time, a preset archive sequence number, and data to be archived. Based on the access characteristic parameters and storage characteristic parameters in the data to be archived, the initial archiving priority and the expected archiving time of the data packet to be archived are calculated. Based on the sequence number adjustment rules, obtain the mapped archive sequence number corresponding to the initial archive priority, and adjust the preset archive sequence number to the mapped archive sequence number; Obtain the archiving time interval between each archiving time interval, and determine the archiving time interval corresponding to the data packet to be archived based on the expected archiving time and the mapped archiving sequence number; The estimated archiving time of the data packet to be archived is compared with the archiving time threshold to obtain the archiving time comparison result; Upon obtaining the first archiving time comparison result, the storage method category and the storage operation corresponding to the storage method category are determined based on the storage characteristic parameters of the data packet to be archived corresponding to the first archiving time comparison result, and the data to be archived is stored. The steps for determining the category of the storage method and the storage operation corresponding to that category include: When the first storage method is obtained, the data packet to be archived is directly stored at the corresponding archiving time; When the second storage method is obtained, the compression ratio of the data to be archived and the compressed storage data to be archived corresponding to the compression ratio of the data to be archived are obtained based on the compression ratio calculation formula, and the compressed storage operation is performed on the compressed storage data to be archived. The formula for calculating the compression ratio is as follows: ； in, For compression ratio, For data structure complexity, S' represents the data encryption strength, and S' represents the data size. To compress the efficiency coefficient, To encrypt and affect weight, The weight is affected by the size of the data.

2. The method for optimizing the storage of historical financial transaction data according to claim 1, characterized in that, The steps for calculating the initial archiving priority and estimated archiving time of the data packet to be archived include: The access metric score and the storage cost evaluation value are calculated based on the access characteristic parameters and the storage characteristic parameters, respectively. The initial archiving priority of the data packet to be archived is obtained by weighting the calculated access metric score and storage cost evaluation value according to a preset weight; Based on the functional mapping relationship between the storage cost assessment value and the expected archiving time interval, and in conjunction with the initial archiving priority, the expected archiving time interval of the data packet to be archived is calculated.

3. The method for optimizing the storage of historical financial transaction data according to claim 2, characterized in that, The steps for calculating the access index score include: Obtain the peak access ratio, access cycle coverage, access timestamp, and cross-type access ratio from the access feature parameters; The peak access ratio, access cycle coverage, access timestamp, and cross-type access ratio are quantified and standardized, and then input into the access indicator value evaluation function to obtain the corresponding access indicator score.

4. The method for optimizing the storage of historical financial transaction data according to claim 3, characterized in that, The access metric value evaluation function includes: Calculate the access metric values ​​corresponding to the access cycle coverage, the access timestamp, and the cross-type access ratio after quantification and standardization. The access metric score is obtained by weighting the access period coverage, the access timestamp, and the cross-type access ratio.

5. The method for optimizing the storage of historical financial transaction data according to claim 4, characterized in that, The step of calculating the storage cost assessment value includes: Obtain the data size, data structure complexity, data encryption strength, and data verification strength from the storage feature parameters; The data size, data structure complexity, data encryption strength, and data verification strength are quantified and standardized, and then input into the storage cost evaluation function to obtain the corresponding storage cost evaluation value.

6. The method for optimizing the storage of historical financial transaction data according to claim 5, characterized in that, The steps to obtain the archive time comparison results include: The estimated archiving time of the data packet to be archived is compared with the archiving time threshold to obtain the archiving time comparison. If the estimated archiving time corresponding to any data packet to be archived is greater than or equal to the archiving time threshold, the first archiving time comparison result is obtained; If the estimated archiving time for all data packets to be archived is less than the archiving time threshold, the second archiving time comparison result is obtained.

7. The method for optimizing the storage of historical financial transaction data according to claim 6, characterized in that, The steps for performing a direct storage operation include: When the current archiving time arrives, each data packet to be archived is written to the storage medium according to the mapped archiving sequence number and at intervals of the archiving time interval.

8. The method for optimizing the storage of historical financial transaction data according to claim 7, characterized in that, The steps to obtain the compression ratio of the archived data for compressed storage based on the compression ratio calculation formula include: Based on the data encryption strength, the data structure complexity, and the data size, the compression ratio of the data to be archived is calculated using the compression ratio calculation formula. Obtain the corresponding data of the same type under each periodic system job type in the data to be archived, sort the data of the same type according to the data size to obtain the sorting result, and determine the compression execution order according to the sorting result; Determine whether there is data with a custom compression storage operation category in the data to be archived, and calculate the proportion of the custom compression portion to the total compression ratio; When the percentage of the custom compression portion does not reach the preset compression ratio threshold, compression is performed on the largest portion of the data to be archived until the compression ratio reaches or exceeds the preset compression ratio threshold.

9. The method for optimizing the storage of historical financial transaction data according to claim 8, characterized in that, The steps for performing a compressed storage operation include: The data to be archived is divided based on the periodic system job type to obtain several periodic job coverage datasets; In the datasets covered by each cycle of work, the data to be archived of the same type are sorted according to the data size in the storage feature parameters; The compression execution order of the datasets covered by each cycle job is determined based on the sorting results; The compression ratio is matched with the archived compressed storage data corresponding to the compression ratio of the data to be archived, and the compression storage operation is performed.

Images