Fund value data record distributed receiving and storing method and system

Abstract

The invention discloses a fund value data record distributed receiving and storing method and system, and belongs to the field of distributed big data, the method comprises the following steps: recording and analyzing the time consumption of each election cycle, the election success frequency, the failure frequency and network delay data, and adjusting the election timeout time in real time; collecting data of response time, log synchronization rate and stability of the nodes, scoring the nodes and allocating priorities to optimize the election process; a batch replication and asynchronous replication mechanism is adopted, so that communication delay between nodes is reduced; and during the election period, the voting strategy is adjusted according to the distribution priority, the node with the high priority preferentially obtains votes, and the priority of each node is calculated and updated in real time for the next election period. According to the method, the consistency and synchronism of data in a distributed database cluster can be effectively improved, the number of election failures is reduced, the election efficiency and stability of a system are optimized, and safe and efficient storage of fund numerical data is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the field of distributed big data and automated parallel computing, and specifically relates to a distributed receiving and storing method and system for data records of financial values. Background Art

[0002] Fund value data records are stored in a distributed database and contain floating-point numbers, strings, and date and time data. Floating-point numbers are used to record numerical changes, strings are used to record textual information, and dates and times are used to record the time of data operations. Data records can be stored as SQL database tables or Python-readable CSV files for easy querying and processing. The fund value data recording system primarily handles large-scale data modifications, particularly those involving floating-point multiplication and division operations, which differs from bank asset data recording systems. Data is stored on multiple nodes to ensure high availability and data consistency. Fund value data records include tables that record values converted from different currencies. For example, database tables representing domestic currency units for foreign exchange bonds require data to change with exchange rates and, in some cases, with time. Furthermore, the values in these database tables often reference each other across tables. Updates to one table must be accurately updated in another. This creates significant data structure complexity, and inadequate response can lead to omissions and errors. This places high demands on the distributed database's storage management algorithms. Due to the extensive and frequent data operations in the fund value system, inter-node data synchronization and consistency challenges are even more complex. Data transmission and storage in a distributed environment are subject to factors such as network latency, partitioning, and data loss, leading to inconsistent and asynchronous data. Fund value data records are stored in a distributed database as a table containing floating-point data, string data, and date and time data. Floating-point data is used to record changes in fund values, string data is used to record textual information about users, and date and time data is used to record the time when additions, deletions, modifications, and queries are made to the data table. These fund value data records are database tables that can be queried using structured queries and numerical tables that can be read using programming languages.

[0003] The data records for fund values are similar to the tables in the distributed databases used for back-end numerical storage at banks in that they both store large amounts of numerical data. Similarly, bank asset data records are frequently added, deleted, modified, and queried, and are subject to frequent data modifications during storage and operation. However, the data in bank asset data records typically undergo numerical operations such as additions or subtractions, typically representing increases or decreases in asset account records. Meanwhile, data in fund value data records often undergo numerical operations such as multiplication and division, such as increases or decreases in value by a certain ratio. Therefore, distributed big data systems used by banks typically perform additions and subtractions on corresponding numerical records for a specific account, while distributed big data systems used for gold asset data records perform multiplication and division on a specific set of data items of the same data type simultaneously by a certain ratio. Both the bank asset data recording system and the fund value data recording system are distributed databases, consisting of multiple nodes (cloud servers). The tables within these distributed databases primarily record values, and a single value record isn't necessarily stored on a single node. Synchronously storing a single data record across multiple nodes is a fundamental function of distributed database clusters, designed to prevent data loss due to node malfunctions and to facilitate rollback recovery in the event of accidental data deletion. This is a fundamental feature of Hadoop architectures, among others. However, the fund value data recording system stores the same value record across a greater number of nodes, as reflected in its cluster parameters. Furthermore, a single change in the fund value data recording system requires a wider response because more nodes must respond to the data modification. Furthermore, sometimes, the data record is not the same, but belongs to the same category. Each record within the same category is stored across multiple nodes in the cluster. This results in a complex response within the cluster, frequently leading to data loss and inconsistent data modifications.

[0004] In a distributed environment, data transmission and storage may be affected by physical factors such as network latency, partitioning, and data loss, leading to inconsistent or unsynchronized data. A method can be developed to ensure data consistency and synchronization across different nodes, addressing these issues through improved data replication algorithms, data verification mechanisms, and other technical means. In the technical application scenario of distributed storage of financial data records, identifying and resolving data consistency and synchronization issues is a physical prerequisite for addressing data loss and inconsistent data modifications.

[0005] In fact, the data receiving and storage system based on blockchain and distributed storage optimization is used to receive data in real time from various data sources (such as the database of the fund management system, the data interface of the market data source), realize data storage based on blockchain and distributed databases (such as Cassandra, Hadoop, etc.), and ensure the consistency of data between different nodes through consensus algorithms (such as Raft or PBFT), such as the Raft configuration change method and device described in the patent document with publication number CN115051913B, and an improved PBFT consensus method described in the patent document with publication number CN113676541B.

[0006] Initially, data reception and storage systems based on blockchain and distributed storage optimization were based on a distributed database cluster system using the PBFT architecture. However, it was discovered that PBFT typically requires at least 3f+1 nodes, where f is the number of faulty nodes to tolerate, and achieves fault tolerance through three-phase message exchange. Initially, industry systems used the PBFT consensus algorithm, but its high node requirements and complex three-phase message exchange made it unsuitable for financial data recording systems. Subsequently, the architecture generally shifted to Raft, but Raft's simplicity resulted in insufficient data consistency and high availability, ultimately forcing the development of a modified proprietary system architecture. Summary of the Invention

[0007] The purpose of the present invention is to propose a distributed receiving and storing method and system for recording data of financial values to solve one or more technical problems existing in the prior art and at least provide a beneficial option or create conditions.

[0008] The present invention discloses a distributed receiving and storage method for recording fund value data, applicable to the node election and log replication process in a distributed database cluster. The method records and analyzes the duration of each election cycle, the number of election successes and failures, and network latency data to adjust the election timeout in real time. Data on node response time, log synchronization rate, and stability are collected to score and prioritize each node to optimize the election process. Batch replication and asynchronous replication mechanisms are used to reduce communication delays between nodes. During the election period, voting strategies are adjusted based on the assigned priorities, with nodes with higher priorities receiving votes first. The priority of each node is then calculated and updated in real time for use in the next election cycle. This method can effectively improve the consistency and synchronization of data in a distributed database cluster, reduce the number of election failures, optimize the system's election efficiency and stability, and ensure the secure and efficient storage of fund value data.

[0009] To achieve the above objectives, according to one aspect of the present invention, a method for distributed reception and storage of data records of fund values is provided. The method is run on a distributed database cluster, and nodes in the distributed database cluster run steps including role definition, election process, log replication, and fault recovery. The method includes:

[0010] By recording and analyzing data on the duration of each election cycle, election success, and network latency, election timeouts can be adjusted in real time.

[0011] Collect data on node response time, log synchronization rate, and stability, and assign priorities to each node to optimize the election process;

[0012] Then use batch replication and / or asynchronous replication to reduce communication latency between nodes;

[0013] Then, fault detection and fault recovery are performed on the nodes in the distributed database cluster.

[0014] Furthermore, the method also includes a batch log replication mechanism, specifically: batch replication is performed by merging multiple log entries to reduce the number of communications between nodes.

[0015] Furthermore, the method may also include an asynchronous replication mechanism, specifically: introducing an asynchronous replication thread between nodes to separate the log replication operation from the main thread.

[0016] Furthermore, by recording and analyzing the duration of each election cycle, election success, and network latency data, the election timeout is adjusted in real time. Specifically:

[0017] The inverse ratio between the duration of each election in the current election cycle and the average network latency during that election is used as the election delay factor. The success of each election is represented by a binary value. The delay factor is filtered by the binary value and then combined with the election timeout from the previous election cycle to limit the election timeout for the next election cycle.

[0018] Furthermore, data on node response time, log synchronization rate, and stability are collected, and each node is assigned a priority to optimize the election process. Specifically:

[0019] Assign priority to each node based on the node's response time, log synchronization rate, and stability data;

[0020] During the election process, nodes vote based on their assigned priorities, and nodes with higher priorities receive votes from other nodes first;

[0021] The scores of storage nodes are recorded through configuration files or management tools in the distributed database cluster to adapt to different network conditions and system requirements.

[0022] Furthermore, each node is assigned a priority based on its response time, log synchronization rate, and stability data. The specific method is as follows:

[0023] The combined values of the average response time density and the average response time order of each node, the log synchronization rate density and the log synchronization rate order, and the failure rate density and the failure rate order are used as the allocation priority of each node.

[0024] Furthermore, it also includes the transfer of allocation priorities during the election cycle and the adjustment of voting between nodes, specifically:

[0025] The allocation priority of each node is stored in the local storage space of each node and passed to other nodes during the election;

[0026] During the election process, nodes adjust their voting priorities based on the allocation priority. Nodes with higher allocation priorities receive votes first, and whether a node participates in the election is determined based on the allocation priority.

[0027] After a node is elected, the allocation priority of each node is calculated again in real time and passed to other nodes in the next election cycle.

[0028] Furthermore, performing fault detection and fault recovery on nodes in a distributed database cluster specifically includes:

[0029] If data loss occurs at a node during the transmission of assigned priorities or the node cannot receive priority information from other nodes, the node will retry sending the assigned priority data or obtain its backup data on assigned priorities from neighboring nodes; for nodes with delays, the assigned priority data of the previous cycle will be retained in the local cache of the node as a temporary emergency backup.

[0030] The present invention also provides a distributed data recording and storage system for fund values, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for distributed data recording and storage for fund values are implemented to control each node of the cluster. The distributed data recording and storage system for fund values can be run on computing devices such as desktop computers, laptop computers, mobile phones, PDAs, and cloud data centers. The executable systems may include, but are not limited to, processors, memories, and server clusters. The processor executes the computer program to run in the following system units:

[0031] The election adjustment unit is used to adjust the election timeout in real time by recording and analyzing the duration of each election cycle, election success, and network latency data;

[0032] The election optimization unit collects data on node response time, log synchronization rate, and stability, and assigns priority to each node to optimize the election process.

[0033] Record replication units for using batch replication and / or asynchronous replication to reduce communication latency between nodes;

[0034] The fault monitoring unit is used to detect and recover faults of nodes in the distributed database cluster.

[0035] The present invention provides a distributed data storage method and system for receiving and storing financial value records. The method records and analyzes the duration of each election cycle, the number of election successes and failures, and network latency data to adjust the election timeout in real time. It also collects node response time, log synchronization rate, and stability data, scores each node, and assigns a priority to optimize the election process. It utilizes batch replication and asynchronous replication mechanisms to reduce communication delays between nodes. During the election, the voting strategy is adjusted based on the assigned priority, with nodes with higher priorities receiving votes first. The priority of each node is then calculated and updated in real time for use in the next election cycle. Traditional distributed systems are prone to frequent election failures and frequent leader changes during elections due to factors such as network latency and node performance fluctuations, impacting system consistency and high availability. Existing technologies often evaluate node performance based on a single metric (such as response time), which fails to fully reflect the actual performance of the nodes, easily leading to irrational elections and the selection of unreliable leader nodes. Furthermore, traditional data replication mechanisms cannot effectively handle complex scenarios such as high-frequency multiplication and division operations, resulting in inconsistencies or delays in data transmission and storage, which are particularly pronounced in highly concurrent environments with financial value data. The method of the present invention dynamically adjusts the election timeout, allowing the system to adapt to changes in network status and node performance, reducing the probability of election failure, improving election efficiency, and ensuring the efficient selection of leader nodes. By combining a multi-dimensional comprehensive evaluation of response time, log synchronization rate, and failure rate, the accuracy of node priority assessment is improved, making election decisions more reasonable and reliable. The use of batch replication and asynchronous replication mechanisms reduces communication delays, optimizes data synchronization speed, effectively solves the problems of data inconsistency and asynchrony in distributed systems, and improves the security and efficiency of financial numerical data. The method of the present invention not only optimizes the election mechanism of a distributed database, but also significantly improves the overall performance and stability of the multi-node response database system of a distributed cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other features of the present invention will become more apparent through a detailed description of the embodiments shown in conjunction with the accompanying drawings. In the drawings of the present invention, the same reference numerals represent the same or similar elements. Obviously, the drawings described below are only some embodiments of the present invention. It is possible for a person skilled in the art to derive other drawings based on these drawings without inventive effort. In the drawings:

[0037] Figure 1 Shown is a flow chart of a method for distributed receiving and storing data records of fund values;

[0038] Figure 2 Shown is a system structure diagram of a distributed receiving and storing system for data recording of fund values. DETAILED DESCRIPTION

[0039] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other unless there is a conflict.

[0040] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0041] like Figure 1 The flowchart of the distributed receiving and storing method of data record of fund value according to the present invention is shown below. Figure 1 The following describes a distributed receiving and storing method and system for data records of fund values according to an embodiment of the present invention.

[0042] The present invention proposes a distributed receiving and storing method for data records of fund values, which specifically includes the following steps:

[0043] By recording and analyzing data on the duration of each election cycle, election success, and network latency, election timeouts can be adjusted in real time.

[0044] Collect data on node response time, log synchronization rate, and stability, and assign priorities to each node to optimize the election process;

[0045] Use batch replication and / or asynchronous replication to reduce communication latency between nodes;

[0046] Perform fault detection and recovery on nodes in a distributed database cluster.

[0047] In some embodiments, nodes in a distributed database cluster receive data modification instructions input from an API port, a server port, and / or a client input port. The nodes in the distributed database cluster execute steps including role definition, an election process, log replication, and fault recovery to execute the data modification instructions, modify and save the data pointed to by the data modification instructions, thereby achieving data consistency and high availability for the data records of the fund value in the distributed system. Specific implementation steps include:

[0048] During the process of defining node roles, three node roles are set in the cluster, including leader, follower, and candidate. Specifically, in the initial state, all nodes are in follower state, responsible for receiving the leader's heartbeat and log replication commands; when the follower does not receive the heartbeat signal within the specified time, it automatically becomes a candidate and initiates an election request; when the candidate obtains the majority of node votes, it becomes the leader, responsible for managing log replication and processing client requests.

[0049] During the election process, when a node is elected, a new leader can be elected through these steps: the candidate node initiates an election request and broadcasts a voting request to all nodes in the cluster; each node in the cluster responds to the received voting request and votes, and the candidate becomes the leader if it receives more than half of the votes; after the election is successful, the leader broadcasts heartbeat messages to other nodes to maintain its leadership position and cluster stability.

[0050] During the log replication process, the leader node manages the log replication process to ensure data consistency among all nodes in the cluster, which includes the following steps: First, the leader receives the log request from the client and appends the request as a log entry to its own log; the leader copies the log entry to the follower node and waits for the replication confirmation of the follower node; when more than half of the nodes confirm that the log entry has been submitted, the leader applies the log entry to the state machine and returns the result to the client.

[0051] Then, during the fault recovery process, when the leader node fails or fails, the system automatically performs fault recovery and maintains the consistency and availability of the cluster through re-election. Specifically: when the follower node detects that the leader has failed, it enters the candidate state and initiates a new round of elections; the newly elected leader inherits the uncommitted log entries and continues to perform log replication tasks to ensure data consistency.

[0052] However, when optimizing the election process, existing technologies are susceptible to network latency or the simultaneous timeout of a large number of nodes, leading to duplicate or invalid elections and impacting system performance. Fixed timeouts can be improved to a dynamic adjustment mechanism that adjusts the timeout in real time based on historical network latency and the number of election failures to reduce duplicate elections. Alternatively, nodes can be assigned priorities, with their leader priority determined based on their historical performance (such as response speed and log consistency), thus reducing election time.

[0053] During log replication optimization, existing technologies require synchronization with multiple followers, impacting overall performance. This approach allows for batch replication by merging multiple log entries, reducing the number of communications between leaders and followers and improving replication efficiency. By upgrading synchronous replication to an asynchronous replication mechanism, this mechanism reduces the time leaders wait for replication confirmation while ensuring eventual consistency.

[0054] During fault detection and recovery, the re-election and data recovery process after a leader failure can cause temporary system unavailability. This can be achieved by introducing a heartbeat message optimization mechanism, increasing message content, and pre-emptively predicting leader performance to avoid triggering elections only after a node actually fails. Alternatively, one or more "hot standby" leaders can be established to monitor the current leader status at all times and quickly take over responsibilities in the event of a primary leader failure, reducing system unavailability.

[0055] Furthermore, the method also includes a batch log replication mechanism, specifically: batch replication is performed by merging multiple log entries to reduce the number of communications between nodes.

[0056] In some embodiments, batch replication can be performed by merging multiple log entries to reduce the number of communications between the leader and followers, including the following steps: when the leader receives a client request, it temporarily stores the request in a batch queue; triggering the batch replication operation at a fixed time or triggering replication when the accumulated log entries reach a set threshold; and a log entry synchronization confirmation mechanism after batch replication to ensure the consistency of all log entries.

[0057] Furthermore, the method may also include an asynchronous replication mechanism, specifically: introducing an asynchronous replication thread between nodes to separate the log replication operation from the main thread.

[0058] In some embodiments, the log copy operation can be separated from the main thread by introducing an asynchronous replication thread on the Leader side to reduce the Leader waiting time, including the following steps: after the Leader receives the log entry, it delegates the log copy task to the asynchronous replication thread; the asynchronous replication thread is responsible for copying the log entry to the Follower node; a callback function or message queue is used to confirm the replication status of the log entry to ensure replication consistency.

[0059] Furthermore, by recording and analyzing the duration of each election cycle, election success, and network latency data, the election timeout is adjusted in real time. Specifically:

[0060] The inverse ratio between the duration of each election in the current election cycle and the average network latency during that election is used as the election delay factor. The success of each election is represented by a binary value. The delay factor is filtered by the binary value and then combined with the election timeout from the previous election cycle to limit the election timeout for the next election cycle.

[0061] In some embodiments, the election process in existing technologies, such as the Raft architecture, is susceptible to network latency or simultaneous timeouts of numerous nodes, leading to repeated or invalid elections and impacting system performance. Testing and analysis have shown that a dynamic timeout can be implemented. This improvement, based on the discovery of a fixed timeout mechanism, allows for real-time adjustment based on historical network latency and the number of election failures, reducing repeated elections. Priority elections can also be implemented, assigning nodes priorities and determining their leader priority based on their historical performance (e.g., response speed and log consistency), thus reducing election time. The core of this optimization algorithm is to dynamically adjust the election timeout to mitigate the impact of election failures and network latency, thereby optimizing the election process. During the current election cycle, the previous election cycle is used to adjust the election timeout for the next election cycle, limiting the system's permitted election timeout for the next election cycle. This is because if a distributed cluster consistently allows so many nodes to elect one another, node elections will consistently timeout. The election is a process of continuously selecting and replacing nodes. Sometimes, if a node is selected but there is still time allowed, the election will continue. The time limit is set and a node is finally selected.

[0062] Specifically, record historical data such as the duration, number of successes, number of failures, and network delays of each election cycle;

[0063] The inverse ratio of the duration of each election in the current election cycle to the average network latency during that election is used as the election delay factor. The success of each election is represented by a binary value. The election delay factor is filtered by the binary value and then combined with the election timeout of the previous election cycle. The resulting value is used as the election timeout for the next election cycle.

[0064] For example, real-time data update, constructing dataset H, where dataset H is updated after each election, including:

[0065] t_i: the time taken for the i-th election;

[0066] s_i: Whether the i-th election is successful, represented by a binary value. For example, a value greater than or equal to 1 indicates success, a value less than 0 or a value between 0 and 1 indicates failure.

[0067] d_i: average network delay during the i-th election;

[0068] n: the number of election data samples, the number of elections in each election cycle, where each election is i;

[0069] w_i: The weight of the i-th election, which can be used to represent a value related to factors such as the election success rate and network latency. For example, it can be used to represent the ratio of the average network latency d_i during the i-th election to the time taken for the i-th election t_i;

[0070] t_n: the duration of the current election, because each election cycle has n elections from the beginning to the end, and the current moment is the last n elections;

[0071] T_{e-1}: The election timeout of the previous election cycle. Relative to T_e, e-1 represents the previous election timeout.

[0072] T_e: election timeout of the current election cycle;

[0073] T_{e+1}: The election timeout for the next election cycle.

[0074] In one embodiment, the time delay factor of each election is multiplied by a value indicating whether the election is successful, for example, 1 indicates success and 0 indicates failure. If successful, the time delay factor of the election is saved as a constant multiple of 1. If failed, the time delay factor of the election is saved in decreasing values from 0 to 1. The time delay factor obtained by filtering the success of the election with a binary value is used as a characteristic time delay factor. The characteristic time delay factors corresponding to each election in each election cycle are accumulated and summed to obtain an accumulated characteristic time delay factor. The characteristic time delay factor is multiplied by the election timeout time of the previous election cycle to obtain the election timeout time of the next election cycle.

[0075] By calculating delay factors for each election and adjusting them based on the success or failure of the election, the election timeout dynamically adapts to the system's current network status and election efficiency. This dynamic adjustment is more flexible than a fixed timeout and helps improve the responsiveness and efficiency of the election process.

[0076] When an election succeeds, the system maintains the stability of the election strategy. When an election fails, the system adjusts the election timeout by decreasing it, quickly responding to changes in network status. This approach helps reduce the number of election failures and improves the overall performance and stability of the system.

[0077] Optimizing the election timeout directly affects the delay in node synchronization. Reasonable election cycle settings can reduce frequent changes in leaders, thereby maintaining data consistency and the normal operation of the cluster.

[0078] In distributed systems, frequent election timeouts and failures can lead to election instability, causing frequent leader changes and impacting cluster operation. Dynamically adjusting timeouts can reduce the probability of election failures and improve election stability.

[0079] Traditional fixed election timeouts struggle to cope with dynamic changes in network conditions, leading to unnecessary waiting and frequent timeouts. By evaluating the duration and latency of each election and adjusting the timeout, the system can respond more quickly and reduce election failure rates.

[0080] Network latency and different node states (such as response time and synchronization rate) can affect election results. Dynamically adjusting the election timeout allows the system to better cope with these changes and avoid election failures caused by unreasonable timeout settings.

[0081] This method dynamically adjusts system parameters (such as the election timeout) based on the results of each election. Real-time feedback is used to adjust control parameters, ensuring system stability in a dynamic environment. By converting the success or failure of each election into a quantifiable factor (i.e., a time delay factor), the system can adaptively optimize its behavior. This tuning mechanism allows the system to adjust future parameters based on historical performance, avoiding the drawbacks of over-reliance on fixed configurations.

[0082] Election success and failure are determined by adjusting the delay factor using binary value filtering, and the election cycle is appropriately distributed through weighted and cumulative calculations. This approach considers the probabilistic distribution of election results. In systems where a cluster contains one or more types of homogeneous data, each record stored in multiple nodes requires simultaneous multiplication and division, the potential for extreme overflows is significant. This approach effectively minimizes the negative impact of election failure on the system.

[0083] The delay factor for each election, combined with the timeout period of the previous cycle, can quickly respond to changes in election status and strike a balance between long-term trends and short-term fluctuations.

[0084] By reducing the delay factor of failed elections and saving it, we effectively prevent extreme situations (such as continuous failures) from having a dramatic impact on the timeout period, ensuring smooth adjustments to the election cycle. This design can reduce the negative impact of abnormal situations on the overall election process and maintain system stability.

[0085] Therefore, we developed this method, combining the election success rate with a time delay factor to dynamically adjust the election cycle timeout, addressing common issues of election instability and inappropriate timeout settings in distributed environments. Its adaptive adjustment and feedback control mechanisms enable the system to respond in real time to complex network state changes. This is particularly true for clusters with one or more types of homogeneous data, each stored on multiple nodes and requiring simultaneous multiplication and division. This effectively ensures data consistency and stability in distributed database clusters.

[0086] In the specific process of data acquisition, there are multiple elections in each election cycle. The election process involves the interaction of multiple nodes, including election requests, voting, confirmation and other steps. It is mainly collected through election log records, heartbeat message detection, and election result monitoring.

[0087] Record the time from the start to the end of each election, i.e., the election cycle duration. The election start time is recorded when the node becomes a candidate and initiates an election request. The end time is recorded when the election ends, i.e., when a new leader is elected or the election times out. The election cycle duration is from the start to the end of the election cycle.

[0088] The start time is recorded at the beginning of the election cycle, and the end time is recorded at the end of the election cycle, and the elapsed time is calculated immediately. The elapsed time data is stored in the node's local log file or in-memory data structure for subsequent analysis and optimization.

[0089] The number of election successes and failures is recorded and counted to assess the stability of the election process. When a node is successfully elected as a leader, a success counter is incremented; when the election process fails due to a timeout or insufficient votes, a failure counter is incremented. Success counts are recorded when a leader is elected and a confirmation message is broadcast. Failure counts are recorded when an election times out or a candidate fails to receive sufficient votes. This count data is stored in the node's election status record and updated during periodic analysis.

[0090] Monitor network latency between nodes to understand the impact of the network on the election process. Use heartbeat messages in the cluster to measure the time delay from the leader sending a heartbeat to the follower replying and confirming it. During the election, record the time difference between the election request message sent by the candidate and received by each node. Calculate the average latency periodically (such as every second or at the end of each election round). Heartbeat message latency is continuously collected within the heartbeat period (such as every 100ms). Election message latency is collected in real time during each election process, from the time the election request is sent to the time the response is confirmed. Latency data is stored in the node's temporary cache for real-time calculation and logging.

[0091] To minimize the impact on system performance, collected data can be temporarily stored in node memory and written to log files or a database in batches periodically (e.g., every 10 seconds). Distributed monitoring services such as Prometheus can be used to centrally collect and analyze performance data from each node, improving overall monitoring efficiency. Real-time monitoring and alerting mechanisms for data collection and analysis should also be implemented. If an excessive election failure rate or abnormally high latency is detected, tuning processes will be automatically triggered.

[0092] In response time collection, the average response time of each node is collected. Each Leader node sends a heartbeat message to the Follower node periodically (for example, every 100ms), and the Follower replies to confirm. By recording the time difference between sending and receiving the heartbeat message, the response time can be calculated. In addition, when processing client requests, the sending time and response time of each request are recorded, the response delay is calculated, and this data is fed back to the scoring module. Among them, each node maintains a response time log, recording all response time data in the recent period (such as the past minute); the current average response time is calculated using a weighted average or sliding window method to adapt to fluctuations in response time.

[0093] For log synchronization rate collection, collect the log synchronization rate of each node, that is, the number of log entries successfully synchronized per second. When the Leader copies the log entry to the Follower, record the start and end time of each log entry copy. Each node periodically counts the number of log synchronization entries completed per unit time (such as per second) and calculates the synchronization rate. Each node maintains a log synchronization counter, which increases by one when the log is successfully copied; the timer triggers the rate calculation once per second, reads the counter value as the current log synchronization rate, and then resets the counter.

[0094] The goal of collecting stability (failure rate) data is to measure the failure rate of each node, that is, the frequency or number of times the node is unavailable. Node status monitoring is used to monitor the online status of the node through heartbeat signals and status reports. If no heartbeat response is received within the specified time, it is counted as a failure. Each node maintains a fault log, recording the time of each failure or response timeout, and calculating the frequency of failures within a certain time window. The failure rate can be calculated by recording the ratio of the node's online time to the number of failures. Use a sliding window to update the failure rate, for example, use the past 10 minutes as a window to count the latest failures to prevent old data from interfering with the current status assessment.

[0095] Furthermore, data on node response time, log synchronization rate, and stability are collected, and each node is assigned a priority to optimize the election process. Specifically:

[0096] Assign priority to each node based on the node's response time, log synchronization rate, and stability data;

[0097] During the election process, nodes vote based on their assigned priorities, and nodes with higher priorities receive votes from other nodes first;

[0098] The scores of storage nodes are recorded through configuration files or management tools in the distributed database cluster to adapt to different network conditions and system requirements.

[0099] In some embodiments, during the node scoring initialization, when the system starts, the scoring system of each node is initialized and an initial scoring value is set for each node. The initial node score can be set to a fixed value (such as 100), or dynamically calculated based on historical performance (the initial research and development is 100, and later based on data statistics, it is found that setting an initial value of 50 will be more suitable for the actual cluster). In the scoring data collection, each node periodically collects performance data of other nodes, including response time, log synchronization rate, and stability (such as failure rate). Performance data is automatically collected through heartbeat messages between nodes or during log synchronization.

[0100] Collect the average response time (in seconds) of each node. The shorter the response time, the better the node performance.

[0101] Collect the log synchronization rate (log entries / second) of each node to reflect the efficiency of the node in processing synchronization;

[0102] The failure rate of each node (number of node unavailability / total running time) is collected to measure the stability of the node. The lower the failure rate, the higher the node score.

[0103] Get the average response time of each node, and use the sequence number obtained by sorting the average response time of all nodes from small to large as the average response time order of each node; the reason why the average response time is sorted from small to large instead of from large to small is that some nodes have faults and do not respond, so the maximum response time of some nodes can be stretched to infinity, causing the program to report an error; if the average response time is sorted from small to large, the response time that is too long at the end can be intercepted.

[0104] Get the log synchronization rate of each node, sort the log synchronization rates of all nodes from largest to smallest, and use the resulting sequence number as the log synchronization rate order of each node. Because the log synchronization rate reflects the efficiency of the node in processing synchronization, the higher the efficiency value, the better.

[0105] The failure rate of each node is obtained, and the failure rates of all nodes are sorted from small to large to obtain the serial number, which is used as the failure rate order of each node. This is because the smaller the failure rate, the better the performance of the node as a leader node.

[0106] By statistically analyzing the probability distribution of the average response time values of all nodes, the probability density of the average response time value of each node is calculated as the average response time density of each node;

[0107] By statistically analyzing the probability distribution of the log synchronization rate values of all nodes, the probability density of the log synchronization rate value of each node is calculated as the log synchronization rate density of each node;

[0108] By statistically analyzing the probability distribution of the failure rate values of all nodes, the probability density of the failure rate value of each node is calculated as the failure rate density of each node;

[0109] Preferably, the probability density function in the Scipy library is used for statistical operations. The reason for calculating the probability density of the response time, log synchronization rate, and stability data of each node in the distributed cluster is to screen which nodes have a high probability of performing normally or a low probability of performing normally. It is undesirable to have some nodes with poor performance showing excellent performance under low probability, and it is also undesirable to have some nodes with good performance showing poor performance under low probability.

[0110] The combined values of the average response time density and the average response time order of each node, the log synchronization rate density and the log synchronization rate order, and the failure rate density and the failure rate order are used as the allocation priority of each node.

[0111] In one embodiment, the product of the average response time density of each node and its average response time order is used to represent the performance of the average response time of each node and the degree to which its performance is common in the distributed cluster;

[0112] The product of each node's log synchronization rate density and its log synchronization rate order is used to indicate the performance of each node's log synchronization rate and its prevalence in the distributed cluster.

[0113] The product of each node's failure rate density and its failure rate order is used to indicate the performance of each node's failure rate and its prevalence in the distributed cluster.

[0114] A method of combining the combination of the average response time density and the average response time order, the combination of the log synchronization rate density and the log synchronization rate order, and the combination of the failure rate density and the failure rate order of each node can be used. In a continuous numerical distribution, the combination of the average response time density and the average response time order, the combination of the log synchronization rate density and the log synchronization rate order, and the combination of the failure rate density and the failure rate order can be integrated or the mathematical expectation value can be obtained; while in a discrete and finite numerical distribution, the combination of the average response time density and the average response time order, the combination of the log synchronization rate density and the log synchronization rate order, and the combination of the failure rate density and the failure rate order can be accumulated and summed, and the summed values corresponding to each node are used as the allocation priority of each node.

[0115] The smaller the allocation priority value, the higher the priority.

[0116] This method combines three key performance indicators (KPIs)—average response time, log synchronization rate, and failure rate—to evaluate each node's performance across multiple dimensions. Compared to single-metric evaluation, combining density and order provides a more comprehensive approach to evaluating node performance.

[0117] By combining density and order, we can effectively distinguish between typical and occasional node performance. This prevents occasional exceptional performance (such as efficient log synchronization) from being mistaken for typical node performance, while also identifying brief, uncommon failures of high-quality nodes. This adaptability improves the accuracy of priority assignments.

[0118] By calculating the probability density function, we can identify the rarity or commonness of node performance and prevent a few abnormal or extreme values from interfering with the overall system evaluation. This is very important for maintaining the overall stability of the system and selecting reliable leader nodes.

[0119] By combining the performance of each node with its probability of occurrence and calculating the mathematical expectation or integral, the resulting allocation priority is more reliable. Nodes with lower priorities are more likely to demonstrate stable and excellent performance, providing a reliable basis for elections.

[0120] Traditional election methods may prioritize a node due to temporary high performance (such as a brief high synchronization rate), but this high performance may not be long-term stable. A method combining density and order can identify this randomness and avoid misselection. In a data recording system for capital values, node performance is affected by multiple factors, such as network latency and node load fluctuations. This multi-dimensional evaluation method can more accurately reflect the actual performance of nodes in complex environments, preventing a single factor from misleading election results. In distributed systems with large fluctuations in node performance, stable leader elections are crucial for data consistency. This method, through in-depth analysis of node stability, ensures that the elected leader is more reliable and reduces data synchronization and consistency issues caused by election errors.

[0121] By using a probability density function (PDF) to assess the commonality of various node performance metrics, the system quantifies the probability of good and bad node performance occurring within the cluster. This probabilistic assessment provides statistical support for understanding node performance and helps identify long-term reliable nodes. The product of density and order is used for a comprehensive evaluation: the order reflects the degree of performance, while the density assesses the commonality of this performance. This combination ensures that node performance evaluation considers both actual performance and stability and normality.

[0122] In the case of continuous data, integration or expected value methods are used to further eliminate the influence of random performance and improve evaluation accuracy. This process allows for reasonable node priority even in environments with uneven data distribution. Combining probability density with order makes the evaluation results more robust to outliers, meaning they are less susceptible to interference from extreme data. This design is crucial in distributed systems, especially when dealing with large-scale data nodes, ensuring the stability of system decisions.

[0123] This priority allocation method, by combining the actual performance of nodes with the statistical commonality of their performance, provides a precise and flexible election mechanism for distributed systems. It significantly mitigates election interference caused by random node performance, improves election stability, and enhances system consistency. The underlying probabilistic statistics and weighted evaluation model provide a solid theoretical foundation, ensuring more reliable and forward-looking election decisions.

[0124] Furthermore, each node is assigned a priority based on its response time, log synchronization rate, and stability data. The specific method is as follows:

[0125] The combined values of the average response time density and the average response time order of each node, the log synchronization rate density and the log synchronization rate order, and the failure rate density and the failure rate order are used as the allocation priority of each node.

[0126] Furthermore, it also includes the transfer of allocation priorities during the election cycle and the adjustment of voting between nodes, specifically:

[0127] The allocation priority of each node is stored in the local storage space of each node and passed to other nodes during the election;

[0128] During the election process, nodes adjust their voting priorities based on the allocation priority. Nodes with higher allocation priorities receive votes first, and whether a node participates in the election is determined based on the allocation priority.

[0129] After a node is elected, the allocation priority of each node is calculated again in real time and passed to other nodes in the next election cycle.

[0130] Among them, priority transfer and voting adjustments during the election period include:

[0131] In some embodiments, when the election starts, each node packages its own assigned priority with other performance data and broadcasts it to other nodes in the cluster through an election request message; after each node receives the priority data of other nodes, it updates the local node priority list to ensure that each node has the latest priority information in the cluster during the election process.

[0132] In some embodiments, during the election process, a node adjusts its voting strategy based on the assigned priority received, casting votes for candidate nodes with smaller assigned priority values (higher priority). Based on the number of votes received and the node's score, the node assesses whether it is sufficiently competitive to participate in the election. Based on a calculated or preset threshold, if its own score is poor (lower priority), the node will give up the election opportunity and cast its vote for a more qualified node.

[0133] In some embodiments, after the elected leader node is successfully elected, the leader will broadcast an election success message and restart the priority evaluation in the next cycle.

[0134] The priority reassessment after the leader is elected includes:

[0135] In some embodiments, after a new leader is elected, a timer is started to periodically (e.g., every 10 seconds) update the assigned priority data of each node; based on the latest performance data, the process of data collection, analysis, and priority calculation is repeated to ensure the timeliness and accuracy of the data at the next election.

[0136] In some embodiments, before the next election cycle starts, the leader node will send the recalculated priority data to other nodes to ensure that the priority information of each node is updated before the election begins; after receiving the updated priority, other nodes will immediately update their priority database locally to prepare for the next election.

[0137] Among them, exception handling and priority adjustment include:

[0138] In some embodiments, if data loss occurs during the priority transmission process or the node cannot receive priority information from other nodes, the node will retry sending the priority data or obtain backup data from neighboring nodes; for nodes with too high latency, the priority data of the previous cycle may be retained in the local cache as a temporary emergency measure.

[0139] In some embodiments, based on feedback from the election process (e.g., election failure, excessive delay, etc.), the scoring calculation parameters and delivery mechanism are adjusted. Adjustments can be made by dynamically adjusting weights, increasing the frequency of density assessments, etc.

[0140] Furthermore, performing fault detection and fault recovery on nodes in a distributed database cluster specifically includes:

[0141] If data loss occurs at a node during the transmission of assigned priorities or the node cannot receive priority information from other nodes, the node will retry sending the assigned priority data or obtain its backup data on assigned priorities from neighboring nodes; for nodes with delays, the assigned priority data of the previous cycle will be retained in the local cache of the node as a temporary emergency backup.

[0142] The distributed receiving and storing system for recording data of fund values runs on any computing device such as a desktop computer, a laptop computer, a mobile phone, a PDA, or a cloud data center. The computing device includes: a processor, a memory, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps in the distributed receiving and storing method for recording data of fund values are implemented. The executable system may include, but is not limited to, a processor, a memory, and a server cluster.

[0143] The embodiment of the present invention provides a distributed receiving and storing system for recording data of fund values, such as Figure 2As shown, the distributed receiving and storing system for data records of fund values in this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the embodiment of the distributed receiving and storing method for data records of fund values are implemented to control the receiving, modifying, saving, and checking of data in a database cluster. The processor executes the computer program and runs it in the following system units:

[0144] The election adjustment unit is used to adjust the election timeout in real time by recording and analyzing the duration of each election cycle, election success, and network latency data;

[0145] The election optimization unit collects data on node response time, log synchronization rate, and stability, and assigns priority to each node to optimize the election process.

[0146] Record replication units for using batch replication and / or asynchronous replication to reduce communication latency between nodes;

[0147] The fault monitoring unit is used to detect and recover faults of nodes in the distributed database cluster.

[0148] Among them, preferably, all undefined variables in the present invention, if not clearly defined, can be manually set thresholds.

[0149] Among them, dimensionless numerical calculations are used between physical quantities of different units.

[0150] The distributed data recording and storage system for fund values can be run on computing devices such as desktop computers, laptops, mobile phones, PDAs, and cloud data centers. The distributed data recording and storage system for fund values includes, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the examples described are merely illustrative of the distributed data recording and storage method and system for fund values and do not constitute a limitation of the method and system for distributed data recording and storage of fund values. The method and system may include more or fewer components than the examples, or a combination of certain components, or different components. For example, the distributed data recording and storage system for fund values may also include input and output devices, network access devices, buses, and the like.

[0151] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete component gate circuits or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The processor serves as the control center of the distributed receiving and storage system for recording the value of funds, and utilizes various interfaces and lines to connect the various sub-regions of the distributed receiving and storage system for recording the value of funds.

[0152] The memory can be used to store the computer programs and / or modules. The processor implements the various functions of the method and system for recording and receiving distributed financial value data by running or executing the computer programs and / or modules stored in the memory and accessing the data stored in the memory. The memory can primarily include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data generated based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0153] The present invention discloses a distributed receiving and storage method for recording fund value data, applicable to the node election and log replication process in a distributed database cluster. The method records and analyzes the duration of each election cycle, the number of election successes and failures, and network latency data to adjust the election timeout in real time. Data on node response time, log synchronization rate, and stability are collected to score and prioritize each node to optimize the election process. Batch replication and asynchronous replication mechanisms are used to reduce communication delays between nodes. During the election period, voting strategies are adjusted based on the assigned priorities, with nodes with higher priorities receiving votes first. The priority of each node is then calculated and updated in real time for use in the next election cycle. This method can effectively improve the consistency and synchronization of data in a distributed database cluster, reduce the number of election failures, optimize the system's election efficiency and stability, and ensure the secure and efficient storage of fund value data.

[0154] Although the present invention has been described in considerable detail and with particularity with respect to several embodiments, it is not intended to limit the present invention to any of these details or embodiments or any particular embodiment, so as to effectively encompass the intended scope of the present invention. In addition, the present invention has been described above with respect to embodiments foreseen by the inventors for the purpose of providing a useful description, and those insubstantial modifications of the present invention that are not currently foreseen may still represent equivalent modifications of the present invention.

Claims

1. A distributed receiving and storing method for data records of fund values, wherein the method is run on a distributed database cluster, and the node operations in the distributed database cluster include the steps of role definition, election process, log replication, and fault recovery, and is characterized in that: The method includes: adjusting the election timeout in real time by recording and analyzing data on the duration of each election cycle, whether the election is successful, and network latency; collecting data on node response time, log synchronization rate, and stability, and assigning a priority to each node to optimize the election process; using batch replication and / or asynchronous replication to reduce communication latency between nodes; and then performing fault detection and fault recovery on nodes in the distributed database cluster.

2. The distributed receiving and storing method for data records of fund values according to claim 1, characterized in that: in, The method also includes a batch log replication mechanism, specifically: batch replication is performed by merging multiple log entries to reduce the number of communications between nodes.

3. The distributed receiving and storing method for data records of fund values according to claim 2, characterized in that: in, The method may also include an asynchronous replication mechanism, specifically: introducing an asynchronous replication thread between nodes to separate the log replication operation from the main thread.

4. The distributed receiving and storing method for data records of fund values according to claim 1, characterized in that: in, By recording and analyzing the duration of each election cycle, election success, and network latency, the election timeout is adjusted in real time. Specifically, the inverse ratio between the duration of each election and the average network latency during the current election cycle is used as the election time delay factor. The success of each election is represented by a binary value. This time delay factor is filtered by the binary value of the election success and then combined with the election timeout of the previous election cycle to limit the election timeout of the next election cycle.

5. The distributed receiving and storing method for data records of fund values according to claim 1 or 4, characterized in that: in, Collect data on node response time, log synchronization rate, and stability, and assign a priority to each node to optimize the election process. Specifically, each node is assigned a priority based on the node's response time, log synchronization rate, and stability data. During the election process, nodes vote based on their assigned priorities, and nodes with higher priorities receive votes from other nodes first; The allocation priority of the storage node is recorded through a configuration file or a management tool in the distributed database cluster.

6. The distributed receiving and storing method for data records of fund values according to claim 5, characterized in that: in, Priority is assigned to each node based on the node's response time, log synchronization rate, and stability data. The specific method is: the combination of the average response time density and the average response time order of each node, the combination of the log synchronization rate density and the log synchronization rate order, and the combination of the failure rate density and the failure rate order of each node are used as the allocation priority of each node.

7. The distributed receiving and storing method for data records of fund values according to claim 5, characterized in that: in, It also includes the transmission of allocation priorities during the election cycle and the adjustment of votes between nodes, specifically: The allocation priority of each node is stored in the local storage space of each node and passed to other nodes during the election; During the election process, nodes adjust their voting priorities based on the allocation priority. Nodes with higher allocation priorities receive votes first, and whether a node participates in the election is determined based on the allocation priority. After a node is elected, the allocation priority of each node is calculated again in real time and passed to other nodes in the next election cycle.

8. The distributed receiving and storing method for data records of fund values according to claim 5, characterized in that: in, Perform fault detection and recovery on nodes in a distributed database cluster, including: If data loss occurs at a node during the transmission of assigned priorities or the node cannot receive priority information from other nodes, the node will retry sending the assigned priority data or obtain its backup data on assigned priorities from neighboring nodes; for nodes with delays, the assigned priority data of the previous cycle will be retained in the local cache of the node as a temporary emergency backup.

9. A distributed receiving and storing system for recording data of fund values, characterized in that: The distributed reception and storage system for data recording of fund values runs in any computing device such as a desktop computer, a laptop computer, or a cloud data center. The computing device includes: a processor, a memory, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps in the distributed reception and storage method for data recording of fund values as described in any one of claims 1 to 4 are implemented.

Images