A distributed-based financial data storage method and system

Abstract

The present application relates to the field of data processing, more particularly, the present application relates to a kind of based on distributed financial data storage method and system, method includes: obtaining the financial transaction data stream to be stored and is divided to obtain multiple data blocks;For single data block, the distribution variation degree of the data block is calculated, based on distribution variation degree, respectively determine the weighted probability for constructing compression encoding table, encryption round of encryption algorithm and replication factor in distributed storage system, and based on the three key parameters determined in sequence, data block is compressed, encrypted and stored;Compression, encryption and storage are carried out to all data blocks, thus complete the distributed security storage of the financial transaction data stream.The present application controls compression, encryption and storage link by unified index linkage, while improving the anti-statistical analysis ability of encrypted data, realizes the adaptive balance of storage security and resource cost.

Description

Technical Field

[0001] This invention relates to the field of data processing. More specifically, this invention relates to a distributed financial data storage method and system. Background Technology

[0002] With the rapid development of fintech and the increasing popularity of applications such as mobile payments and online banking, the volume of financial data is growing explosively. Traditional centralized storage architectures face problems such as storage bottlenecks, single points of failure, and insufficient scalability, making it difficult to meet the needs of massive, high-concurrency financial transactions.

[0003] Distributed storage technology, through multi-node collaboration, can theoretically improve system availability and reliability, but the multi-replica redundancy mechanism also brings significant storage cost pressure. Existing solutions typically adopt a "compress first, encrypt later, then store" approach, that is, compressing data using algorithms such as Huffman coding, then using encryption algorithms to protect data security, and finally performing distributed storage.

[0004] However, financial data is generally characterized by high numerical density and strong local correlations. After compression using static Huffman coding, the data distribution is often uneven, leaving residual local statistical patterns. When directly encrypted on this basis, these distribution characteristics are easily exploited by statistical analysis methods such as differential attacks, weakening the actual security of the encryption.

[0005] Therefore, how to improve the uniformity of data distribution during the compression stage to enhance the encryption resistance to attacks, and how to achieve an adaptive balance between security and cost during the storage stage, have become key issues that urgently need to be addressed in the field of financial distributed storage. Summary of the Invention

[0006] To address the aforementioned technical problems, the present invention provides solutions in the following aspects.

[0007] In the first aspect, a distributed financial data storage method includes:

[0008] The system acquires the financial transaction data stream to be stored and divides the data stream into continuous data blocks according to a preset size; the data stream is in byte sequence format.

[0009] For a single data block, the degree of distribution variation of the data block is calculated. Based on the degree of distribution variation, the weighted probability for constructing the compression encoding table, the encryption rounds of the encryption algorithm, and the replication factor in the distributed storage system are determined. Based on the determined weighted probability, the data block is adaptively compressed to obtain a compressed data block. Based on the determined encryption rounds, the compressed data block is encrypted to obtain an encrypted data block. Based on the determined replication factor, the encrypted data block is stored.

[0010] The compression, encryption, and storage are performed on all data blocks, thus completing the distributed secure storage of the financial transaction data stream;

[0011] The distribution variation of a data block is calculated based on the probability of each character within the current data block, the probability change between adjacent blocks, and the global probability statistics in the entire data stream.

[0012] Preferably, the degree of distribution change of the data block specifically includes: calculating the degree of distribution change of each character within the data block; for any character in the current data block, the calculation process for its degree of distribution change is as follows:

[0013] Calculate the probability of the character appearing in the current data block; obtain the mean and standard deviation of the probability of the character appearing in all data blocks; calculate the difference between the probability of the character appearing in the current data block and its probability of appearing in the previous data block, and obtain the probability difference value.

[0014] Obtain the maximum and minimum probability difference values ​​of the character across all data blocks;

[0015] The first factor is obtained by normalizing the absolute value of the probability difference and the maximum and minimum values; the second factor is obtained by bias-weighted calculation based on the mean and standard deviation of the probability of the character appearing in the current data block and the probability of the character appearing in all data blocks.

[0016] Multiplying the first factor by the second factor yields the degree of variation in the distribution of the character within the current data block.

[0017] Preferably, obtaining the weighted probability includes:

[0018] For any character in the current data block, obtain the historical probability of the character in historical financial data, and multiply the historical probability of the character by the degree of distribution change of the character to obtain the weighted probability of the character.

[0019] Preferably, the adaptive compression is:

[0020] Based on the weighted probabilities of all characters in the current data block, a Huffman tree is constructed using the Huffman coding algorithm, and the constructed Huffman tree is used to compress the current data block.

[0021] Preferably, obtaining the encryption round includes:

[0022] Calculate the average distribution variation of all characters within the current data block;

[0023] Calculate the product of the average distribution variation of all characters and the preset first adjustment range, and round down the result of the product. Calculate the difference between the preset base encryption round and the rounded result to obtain the encryption round.

[0024] Preferably, the encryption algorithm is the AES algorithm.

[0025] Preferably, obtaining the replication factor includes:

[0026] Calculate the average distribution change of all characters within the current data block; calculate the product of the average distribution change of all characters and the preset second adjustment range, and round down the result of the product; calculate the difference between the preset maximum replication factor and the rounded result to obtain the median value.

[0027] The larger of the intermediate value and the preset minimum replication factor is taken as the replication factor.

[0028] Preferably, the encrypted data block is stored to multiple data nodes according to the replication factor.

[0029] In a second aspect, a distributed financial data storage system includes a processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement any of the distributed financial data storage methods described above.

[0030] The beneficial effects of this invention are:

[0031] This invention utilizes a unified index of distribution variation to coordinate and regulate three key stages: compression, encryption, and storage, forming an adaptive closed-loop control: During compression, the Huffman tree is dynamically reconstructed to improve data uniformity and enhance encryption's resistance to attacks; during encryption, the encryption rounds are adaptively adjusted based on data randomness to achieve a precise balance between security and efficiency; and during storage, the replication factor is dynamically adjusted based on data characteristics to optimize storage costs and reliability. Ultimately, this achieves a synergistic improvement in the security, performance, and cost-effectiveness of financial data storage. Attached Figure Description

[0032] Figure 1 This is a flowchart of steps S1-S3 in a distributed financial data storage method according to an embodiment of the present invention.

[0033] Figure 2 This is a structural block diagram of a distributed financial data storage system according to an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0035] Reference Figure 1 A distributed financial data storage method includes steps S1-S3, as detailed below:

[0036] S1: Obtain the financial transaction data stream to be stored and divide the data stream into continuous data blocks according to a preset size.

[0037] Financial transaction data is typically characterized by high concurrency and diverse, heterogeneous sources, with variations in data format, encoding methods, and field structures. Directly processing the raw data stream can lead to inefficiencies or failures in subsequent statistical analysis and encoding / compression operations due to data inconsistency. Furthermore, financial data streams are usually continuously input; without block processing, dynamic perception and adaptive adjustment of local data characteristics are impossible. Block processing also facilitates parallel computing and distributed storage scheduling, improving the overall system throughput.

[0038] In one embodiment, Apache Kafka is used as the data acquisition platform to receive real-time data streams from multiple sources, including bank transaction systems, payment gateways, and securities trading platforms. These data streams are transmitted in byte sequence format. The received byte sequences are parsed and standardized, including:

[0039] A JSON parser is used to decode the byte sequence into a structured object, and data type conversions and format standardization are performed on key fields (such as transaction amount and timestamp) (e.g., converting the amount field to a floating-point number, standardizing the timestamp to UTC format, and filling in missing values). Invalid or incorrectly formatted records are then filtered out. Finally, the cleaned structured data is converted into a unified binary format.

[0040] After the above preprocessing is completed, the data stream is divided into multiple data blocks continuously and non-overlapping according to a preset window size (e.g., every 10 transaction records).

[0041] By real-time aggregation and standardized cleaning of multi-source heterogeneous data, structured data blocks with uniform format and reliable quality are generated. This eliminates the interference of source differences on subsequent processes and lays a solid foundation for accurately calculating data distribution characteristics. The block mechanism transforms continuous data streams into discrete, processable units, enabling the system to perform subsequent adaptive compression, encryption, and storage at the block level, significantly improving processing flexibility and system scalability.

[0042] S2: For a single data block, calculate the degree of distribution variation of that data block.

[0043] Financial data is not static; its statistical characteristics (such as the frequency of different amounts) fluctuate over time. Traditional static compression encoding cannot adapt to this dynamic change, resulting in compressed data retaining local statistical patterns and forming an uneven distribution. This unevenness exposes weaknesses and patterns during the encryption stage, making it vulnerable to exploitation by statistical analysis methods such as differential attacks. Therefore, it is necessary to introduce an indicator that can quantify the local dynamic characteristics of the data to perceive changes in data distribution in real time and drive adaptive adjustments to parameters in all subsequent processing stages.

[0044] In one embodiment, for any character in the current data block, the probability of the character appearing in the current data block is calculated; the mean and standard deviation of the probability of the character appearing in all data blocks are obtained; the difference between the probability of the character appearing in the current data block and its probability of appearing in the previous data block is calculated to obtain the probability difference value.

[0045] Next, the maximum and minimum values ​​of the probability difference for the character are obtained across all data blocks. Based on the absolute value of the probability difference calculated above, and the maximum and minimum values ​​of the probability difference for the character across all data blocks, normalization is performed to obtain the first factor. Based on the probability of the character appearing in the current data block, and the mean and standard deviation of the probability of the character appearing in all data blocks, a bias-weighted calculation is performed to obtain the second factor.

[0046] Finally, multiply the first factor by the second factor to obtain the degree of distribution variation of the character in the current data block.

[0047] The first factor mentioned above is expressed by the following relation:

[0048]

[0049] In the formula, As the first factor, It is the difference between the probability of a single character appearing in the current data block and the probability of that character appearing in the previous data block (i.e., the probability difference value mentioned above). This represents the minimum probability difference of a single character across all data blocks. This represents the maximum probability difference of a single character across all data blocks. This is the first hyperparameter, and its value can be 0.0001 to prevent the denominator from being 0.

[0050] The above introduces the historical fluctuation range of characters throughout the data stream. As a reference, the first factor maps the probability difference of a character in the current data block to a dimensionless relative scale related to the characteristics of the character itself. Specifically:

[0051] The numerator of the first factor calculates the net change in the current probability difference value exceeding the historical minimum probability difference value, eliminating the influence of differences in the basic change levels of different characters. The denominator of the first factor represents the total change potential of the character over the entire historical period. When the first factor is close to 1, it indicates that the current change of the character has reached or is close to the historical maximum volatility level of the character, which is a drastic change. When the first factor is close to 0, it indicates that the current change of the character is weak, close to the historical minimum volatility level.

[0052] The second factor mentioned above is expressed by the following relational formula:

[0053]

[0054] In the formula, As the second factor, It is an exponential function with the natural constant e as its base. This is the second hyperparameter, and its possible value is 0.001. This represents the probability of a single character appearing in the current data block. This represents the mean probability of a single character appearing across all data blocks. The standard deviation of the probability of a single character appearing in all data blocks. This is the first hyperparameter, and its value can be 0.0001 to prevent the denominator from being 0.

[0055] The above molecular part The absolute deviation of a character's current probability from its historical global average probability is measured. The denominator incorporates the standard deviation of the character's probability across all blocks as a stability weight. A larger value indicates that the character has historical volatility, weakening the significance of its current absolute deviation. When the value is small, it indicates that the character has high historical stability, and the same absolute deviation is amplified in its significance.

[0056] Furthermore, by using an exponential function, the aforementioned stability-weighted deviation values ​​are mapped to the [0, 1] interval. Its core function is to control the convergence speed of the function and prevent it from becoming too large due to excessive weighted bias. The second factor converges too quickly to a saturation value of 1, thus maintaining effective discrimination and gradient over a large input range and ensuring a sensitive response to different degrees of deviation.

[0057] In summary, the aforementioned distribution variation is a comprehensive quantitative indicator that captures both short-term fluctuations and long-term statistical anomalies in character frequency. This indicator can accurately identify stable and abrupt regions in the data stream, providing precise and real-time data-driven basis for subsequent encryption, compression, and storage. It is the core hub for achieving closed-loop adaptive control of the entire system.

[0058] S3: Based on the degree of distribution variation of all data blocks, determine the weighted probability used to construct the compression encoding table, the encryption rounds of the encryption algorithm, and the replication factor in the distributed storage system. Based on the determined execution parameters, compress, encrypt, and store each data block in sequence, thus completing the distributed secure storage of the financial transaction data stream.

[0059] The compression efficiency and security of Huffman coding highly depend on its accuracy in assessing data probabilities. Constructing the encoding table solely based on the static frequency of current data blocks ignores the dynamic history and trends of character occurrences, causing the encoding to fail to adapt to the evolution of the data stream, resulting in a predictable compressed output. Therefore, it is necessary to introduce the degree of distribution variation as a weight based on frequency, enabling the encoding table to dynamically adjust with data distribution and reducing the predictability of the compressed data.

[0060] In one embodiment, for any character in the current data block, the historical probability of the character in historical financial data is obtained, and the weighted probability of the character is obtained by multiplying the historical probability of the character by the degree of distribution change of the character.

[0061] Furthermore, using the weighted probability set of all characters, the Huffman coding algorithm is run to generate a dynamic Huffman tree and coding table for the current data block. This coding table is then used to compress the current data block to obtain a compressed data block.

[0062] By combining historical occurrence probabilities with the degree of distribution variation, the construction of the Huffman tree not only considers the prevalence of character occurrences but also enhances the coding distinguishability of drastically changing characters. This makes the compressed bitstream closer to a uniform random distribution in the time dimension, significantly reducing local statistical residues, providing more random input for subsequent encryption, and improving resistance to differential attacks.

[0063] Furthermore, all compressed data blocks obtained through the above compression process are encrypted.

[0064] Existing encryption algorithms such as AES (Advanced Encryption Standard) rely on the sufficiency of their round functions in spreading and obfuscating plaintext. For input data with uneven distribution and obvious patterns, standard rounds may not completely eliminate statistical characteristics, making them vulnerable to differential attacks. In financial data containing repeating or similar numerical sequences, the differential paths are short and have a high probability, making it easier for attackers to guess the key. Therefore, it is necessary to dynamically adjust the encryption rounds based on the data distribution characteristics to achieve a balance between security and efficiency.

[0065] In one embodiment, the average distribution change of all characters in the current data block is calculated, the product of the average distribution change of all characters in the current data block and the preset first adjustment range (e.g., 4) is calculated, the result of the product is rounded down, and the difference between the preset base encryption round (e.g., 14 rounds) and the rounded result is calculated to obtain the encryption round after the current data block is adjusted.

[0066] The aforementioned first adjustment range restricts the value range of the adjusted encryption rounds to the interval [10, 14]. Even when the mean of the distribution change is close to 1, the encryption rounds are still no less than 10 rounds. Usually, 10 rounds or more can effectively resist known differential and linear attacks.

[0067] Furthermore, using the AES encryption algorithm, with the adjusted encryption rounds calculated above as the actual encryption rounds, the compressed data block is encrypted to generate an encrypted data block.

[0068] Through the above operations, for low-randomness data with small distribution variations and obvious patterns, the number of encryption rounds will be automatically increased to enhance diffusion and obfuscation, thus resisting differential attacks; for high-randomness data, the number of rounds will be appropriately reduced to lower computational overhead. This adaptive mechanism optimizes encryption efficiency in financial transaction scenarios without compromising the overall security baseline.

[0069] Furthermore, all encrypted data blocks obtained through the above encryption process are stored.

[0070] However, it's important to note that while distributed storage systems typically employ multi-replica mechanisms to ensure data reliability, a fixed replication factor leads to high storage costs and significant synchronization latency. In financial data, the security and importance of different data blocks are not uniform: highly random data, after encryption, already possesses strong resistance to attacks, allowing for a moderate reduction in the number of replicas to save storage; low-random data, on the other hand, requires higher redundancy to mitigate risk. Therefore, the replication factor needs to be dynamically adjusted based on data distribution characteristics to achieve an adaptive balance between security and cost.

[0071] In one embodiment, the average distribution variation of all characters in the current data block is calculated; the average distribution variation of all characters in the current data block is multiplied by a preset second adjustment range (e.g., a value of 4), and the result of the product is rounded down. The difference between the preset maximum replication factor (e.g., 5) and the rounded result is calculated to obtain an intermediate value; the larger value between the calculated intermediate value and the preset minimum replication factor (e.g., 2) is taken as the replication factor.

[0072] The minimum replication factor is set to 2 to achieve a typical minimum setting for basic high availability. Two replicas ensure that data can still be read from the other replica if one node fails, meeting the minimum data durability requirements of financial systems. The maximum replication factor is set to 5 to support the system tolerating the failure of multiple nodes (up to 4) without data loss. The second adjustment increases the mean of the replication factor's variation with distribution from 0 to 1, while the theoretical value decreases linearly from 5 to 1. However, due to the limitation of the minimum replication factor, the actual effective range is [2, 5].

[0073] Furthermore, using the Hadoop Distributed File System, encrypted data blocks are stored as replicas of the same size as the replication factor on different data nodes.

[0074] By mapping the degree of distribution variation to a replication factor, intelligent scheduling of storage resources is achieved: reducing the number of replicas for highly random data reduces storage and network overhead; increasing replicas for low-random data improves disaster recovery capabilities. This significantly reduces overall storage costs and enhances the economy and scalability of distributed storage systems without affecting system availability.

[0075] Finally, the compression, encryption, and storage operations described above are repeated for each data block in a pipelined or parallel manner until the entire financial transaction data stream has been processed. The processing of each block is independent of each other, supporting accelerated execution through a distributed computing framework.

[0076] In summary, this invention achieves an adaptive control mechanism that links compression, encryption, and storage in three stages. The system can dynamically adjust processing strategies based on the statistical characteristics of the data itself, optimizing storage efficiency and computing resource utilization while ensuring data security. It is suitable for financial data storage scenarios with high concurrency and high security requirements.

[0077] This invention also provides a distributed financial data storage system. For example... Figure 2 As shown, the system includes a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, implement the distributed financial data storage method according to the first aspect of the present invention.

[0078] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0079] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A distributed financial data storage method, characterized in that, include: Acquire the financial transaction data stream to be stored, and divide the data stream into continuous data blocks according to a preset size; The data stream is in the form of a byte sequence; For a single data block, the degree of distribution variation of the data block is calculated. Based on the degree of distribution variation, the weighted probability for constructing the compression encoding table, the encryption rounds of the encryption algorithm, and the replication factor in the distributed storage system are determined. Based on the determined weighted probability, the data block is adaptively compressed to obtain a compressed data block. Based on the determined encryption rounds, the compressed data block is encrypted to obtain an encrypted data block. Based on the determined replication factor, the encrypted data block is stored. The compression, encryption, and storage are performed on all data blocks, thus completing the distributed secure storage of the financial transaction data stream; Specifically, the degree of distribution change in the data block includes: calculating the degree of distribution change for each character within the data block; the calculation process for the degree of distribution change for any character in the current data block is as follows: Calculate the probability of the character appearing in the current data block; obtain the mean and standard deviation of the probability of the character appearing in all data blocks; calculate the difference between the probability of the character appearing in the current data block and its probability of appearing in the previous data block, and obtain the probability difference value. Obtain the maximum and minimum probability difference values ​​of the character across all data blocks; The first factor is obtained by normalizing the absolute value of the probability difference and the maximum and minimum values; the second factor is obtained by bias-weighted calculation based on the mean and standard deviation of the probability of the character appearing in the current data block and the probability of the character appearing in all data blocks. Multiplying the first factor by the second factor yields the degree of variation in the distribution of the character within the current data block.

2. The distributed financial data storage method according to claim 1, characterized in that, The acquisition of the weighted probability includes: For any character in the current data block, obtain the historical probability of the character in historical financial data, and multiply the historical probability of the character by the degree of distribution change of the character to obtain the weighted probability of the character.

3. The distributed financial data storage method according to claim 2, characterized in that, The adaptive compression is as follows: Based on the weighted probabilities of all characters in the current data block, a Huffman tree is constructed using the Huffman coding algorithm, and the constructed Huffman tree is used to compress the current data block.

4. The distributed financial data storage method according to claim 1, characterized in that, The acquisition of the encryption round includes: Calculate the average distribution variation of all characters within the current data block; Calculate the product of the average distribution variation of all characters and the preset first adjustment range, and round down the result of the product. Calculate the difference between the preset base encryption round and the rounded result to obtain the encryption round.

5. A distributed financial data storage method according to claim 4, characterized in that, The encryption algorithm is AES.

6. The distributed financial data storage method according to claim 1, characterized in that, The acquisition of the replication factor includes: Calculate the average distribution change of all characters within the current data block; calculate the product of the average distribution change of all characters and the preset second adjustment range, and round down the result of the product; calculate the difference between the preset maximum replication factor and the rounded result to obtain the median value. The larger of the intermediate value and the preset minimum replication factor is taken as the replication factor.

7. A distributed financial data storage method according to claim 6, characterized in that, The encrypted data block is stored to multiple data nodes according to the replication factor.

8. A distributed financial data storage system, characterized in that, include: A processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement the distributed financial data storage method according to any one of claims 1-7.

Images