A financial big data distributed machine learning acceleration method based on dynamic delay compensation

Abstract

This invention provides a method for accelerating distributed machine learning in financial big data based on dynamic latency compensation, comprising the following steps: establishing a financial big data feature analysis model to extract time-series features, volatility features, and real-time features from the input financial big data; constructing a computing node state prediction model to obtain prediction results; combining the time-series features, volatility features, and real-time features with the prediction results of the computing node state prediction model to calculate a dynamic latency compensation value; and adjusting the data transmission and processing order of each computing node according to the dynamic latency compensation value and a reinforcement learning algorithm to accelerate distributed machine learning in financial big data. This invention calculates accurate dynamic latency compensation values ​​using the features and prediction results of financial big data, enabling data transmission and processing to better adapt to the system, reducing waiting time caused by latency, improving the efficiency of parallel computing, and thus enhancing training efficiency.

Description

Technical Field

[0001] This invention relates to the field of financial big data processing and machine learning technology, and in particular to a distributed machine learning acceleration method for financial big data based on dynamic latency compensation. Background Technology

[0002] In the field of financial big data processing, distributed machine learning significantly improves data processing efficiency by distributing massive amounts of data across multiple computing nodes for parallel processing. However, existing technologies still have many shortcomings. Traditional distributed machine learning latency compensation methods often employ fixed parameters or simple linear weighting strategies, such as setting a fixed compensation time based on historical average latency, or simply calculating dynamic latency compensation values ​​based on node load and transmission delay. These methods cannot fully adapt to the dynamic characteristics of financial data and complex network environments.

[0003] Financial data is characterized by its high degree of temporality, volatility, and real-time nature. For example, in high-frequency trading scenarios, market conditions change rapidly, making real-time data processing crucial; in risk assessment scenarios, the accuracy and completeness of data have a profound impact on the results. Existing latency compensation methods cannot accurately match the specific latency requirements of different financial data processing tasks, nor can they effectively cope with unexpected situations such as network congestion and node failures. Furthermore, existing methods lack the ability to predict future latency changes and cannot optimize data transmission and processing order in advance, leading to low model training efficiency.

[0004] In summary, the technical problem that this invention actually solves is how to improve the training efficiency of the model. Summary of the Invention

[0005] To overcome the shortcomings of low training efficiency in the existing technology, the present invention aims to provide a distributed machine learning acceleration method for financial big data based on dynamic latency compensation. This method calculates accurate dynamic latency compensation values ​​based on the features of financial big data and the prediction results of computing node states, enabling data transmission and processing to better adapt to the system state, thereby reducing waiting time caused by latency, improving the efficiency of parallel computing, and thus enhancing training efficiency.

[0006] This invention discloses a distributed machine learning acceleration method for financial big data based on dynamic latency compensation, comprising the following steps:

[0007] Establish a financial big data feature analysis model to extract time-series features from the input financial big data. Volatility characteristics and real-time characteristics ;

[0008] Construct a computing node status prediction model based on historical load information. Historical data transmission delay information and the current load information and data transmission delay information Long Short-Time Memory (LSTM) networks are used to predict the load change trend of each computing node within T time steps. and data transmission latency change trend ;

[0009] Time series features Volatility characteristics and real-time characteristics Combined with the prediction results of the computing node state prediction model, the dynamic delay compensation value of each computing node at the current moment is calculated using a computational formula. The calculation formula is:

[0010]

[0011] in, , and It is a nonlinear mapping function;

[0012] Based on dynamic delay compensation value Furthermore, it incorporates reinforcement learning algorithms to adjust the data transmission and processing order of each computing node, thereby accelerating distributed machine learning for financial big data.

[0013] The adjustment of the data transmission and processing order satisfies the formula:

[0014]

[0015] Where s represents the current system state. For a set of actions, This is the state-action value function.

[0016] Preferably, the financial big data feature analysis model uses convolutional neural networks to extract features from financial big data, thereby extracting local features of financial big data through different convolutional kernels and obtaining the temporal features of financial big data. Volatility characteristics and real-time characteristics .

[0017] Preferably, the nonlinear mapping function and Gaussian radial basis functions are used:

[0018]

[0019]

[0020] in, , , , These are function parameters.

[0021] Preferably, the reinforcement learning algorithm employs a deep Q-network, which fits the state-action value function by constructing a neural network. .

[0022] Preferably, in calculating the dynamic delay compensation value This also includes dynamic delay compensation values. Outlier detection and correction are performed, specifically as follows:

[0023] Set threshold range ,

[0024] like Then let ;

[0025] like Then let .

[0026] Preferably, after adjusting the data transmission and processing order of each computing node, a feedback adjustment mechanism is also included to dynamically adjust the financial big data analysis model, the computing node state prediction model, and the dynamic latency compensation value based on the training results of the machine learning model and the real-time system status. Parameters in the calculation formula.

[0027] Preferably, the feedback adjustment mechanism uses a particle swarm optimization algorithm to optimize the parameters.

[0028] Preferably, the machine learning model employs an ensemble learning method.

[0029] Preferably, the computing nodes include edge computing nodes, cloud server computing nodes, and local computing nodes to form a multi-level collaborative computing architecture.

[0030] Preferably, data transmission employs a content-addressable distributed storage system and incorporates blockchain technology for data transmission.

[0031] After adopting the above technical solution, compared with the prior art, the beneficial effect of the present invention is that it can calculate an accurate dynamic delay compensation value by using the characteristics of financial big data and the prediction results of computing node status, so that data transmission and processing can better adapt to the system status, thereby reducing the waiting time caused by delay, improving the efficiency of parallel computing, and thus improving training efficiency.

[0032] The financial big data feature analysis model employs a convolutional neural network, which effectively extracts the temporal, volatile, and real-time characteristics of financial big data, providing an accurate data foundation for subsequent calculations. The computing node state prediction model uses a long short-term memory network to accurately predict future load and data transmission latency trends of computing nodes. Simultaneously, the nonlinear mapping function uses a Gaussian radial basis function, and the feedback adjustment mechanism employs a particle swarm optimization algorithm. The combination of these models and algorithms ensures efficient system operation across all stages, from feature extraction and system state prediction to parameter optimization, further improving training efficiency.

[0033] The computing nodes employ a multi-level collaborative computing architecture consisting of edge computing nodes, cloud server computing nodes, and local computing nodes. Edge computing nodes can perform preliminary processing at the data source, reducing data transmission volume and latency; cloud server computing nodes provide powerful computing and storage capabilities to handle complex tasks; and local computing nodes meet the need for rapid local response. Furthermore, data transmission utilizes a content-addressed distributed storage system combined with blockchain technology. The distributed storage system enables efficient data storage and retrieval, while blockchain technology ensures the security and integrity of data transmission, reducing errors and latency during the data transmission process, further improving training efficiency.

[0034] The machine learning model employs an ensemble learning approach to enhance its generalization ability and prediction accuracy. Simultaneously, a feedback adjustment mechanism is established. Based on the training results of the machine learning model and the real-time system state, the parameters in the financial big data feature analysis model, the computing node state prediction model, and the dynamic latency compensation value calculation formula are dynamically adjusted using a particle swarm optimization algorithm. This allows the system to continuously adapt to changes in data and the environment, continuously optimize the model training process, and further improve training efficiency. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating the steps of a distributed machine learning acceleration method for financial big data based on dynamic latency compensation, as described in this invention. Detailed Implementation

[0036] The advantages of the present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments.

[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0038] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0039] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0040] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0041] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0042] In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the convenience of the description of the invention and have no specific meaning in themselves. Therefore, "module" and "part" can be used interchangeably.

[0043] This embodiment discloses a distributed machine learning acceleration method for financial big data based on dynamic latency compensation, including the following steps: establishing a financial big data feature analysis model to extract time-series features from the input financial big data. Volatility characteristics and real-time characteristics Construct a computing node status prediction model based on historical load information. Historical data transmission delay information and the current load information and data transmission delay information Long Short-Time Memory (LSTM) networks are used to predict the load change trend of each computing node within T time steps. and data transmission latency change trend ; Time series characteristics Volatility characteristics and real-time characteristics Combined with the prediction results of the computing node state prediction model, the dynamic delay compensation value of each computing node at the current moment is calculated using a computational formula. The calculation formula is: ,in, , and It is a nonlinear mapping function; based on the dynamic time delay compensation value Furthermore, reinforcement learning algorithms are used to adjust the data transmission and processing order of each computing node to accelerate distributed machine learning in the financial big data field; the adjustment of the data transmission and processing order satisfies the formula: Where s is the current system state, For a set of actions, This is the state-action value function.

[0044] In this embodiment, refer to Figure 1 As shown, a detailed description of a distributed machine learning acceleration method for financial big data based on dynamic latency compensation will be provided, specifically including the following steps:

[0045] S100: Establish a financial big data feature analysis model to extract time-series features from the input financial big data. Volatility characteristics and real-time characteristics In this step, a financial big data feature analysis model will be established, which forms the foundation of the entire methodology. Taking futures trading data as an example, a convolutional neural network (CNN) is used to process the data. Local features are extracted through different convolutional kernels. In this process, the CNN can capture the changing patterns of futures prices over different time periods, thereby obtaining the temporal characteristics of the futures trading data. Such as short-term price fluctuations and long-term trends; volatility characteristics. This refers to the intensity of price fluctuations; real-time characteristics. Features such as the frequency and timeliness of updates to the latest transaction prices reflect the essential characteristics of financial big data, providing crucial information for subsequent processing and calculations.

[0046] Step S200: Construct a computing node state prediction model based on historical load information. Historical data transmission delay information and the current load information and data transmission delay information Long Short-Time Memory (LSTM) networks are used to predict the load change trend of each computing node within T time steps. and data transmission latency change trend In this step, a compute node state prediction model will be constructed based on historical load information. Historical data transmission delay information and the current load information and data transmission delay information Using Long Short-Term Memory (LSTM) networks, the load change trend of each computing node within the next T time steps is predicted. and data transmission latency change trend For example, in some embodiments, in a financial big data processing cluster containing multiple computing nodes, a Long Short-Term Memory (LSTM) network can predict future load conditions and data transmission latency changes of computing nodes by learning information such as CPU utilization, memory usage, and round-trip time of data transmission for each computing node over a past period. The computing node state prediction model provides a prediction of the future system state for subsequent dynamic latency compensation calculations, and is an important prerequisite for achieving dynamic adjustment.

[0047] Step S300: In this step, the temporal features extracted in step S100 are... Volatility characteristics and real-time characteristics The prediction results of the node state prediction model in step S200 are combined and calculated using the formula. Calculate dynamic delay compensation value In this calculation formula, , , Let be the weighting coefficient, and By weighting the predicted trends of features and node states, the influence of each factor on the time delay step is reasonably allocated. Nonlinear mapping function. and (Using the Gaussian radial basis function (RBF)) further applies a nonlinear transformation to the trends in computing node load and data transmission latency, making the calculation results more consistent with reality. For example, in some embodiments, for a node processing foreign exchange transaction data, if... , , ,predict , By substituting the values ​​into the formula, the dynamic delay compensation value of the computing node can be calculated. This step organically combines characteristics and system prediction, achieving accurate quantification of the delay compensation value.

[0048] Step S400: In this step, based on the dynamic delay compensation value... This involves adjusting the data transmission and processing order of each computing node using reinforcement learning algorithms. A Deep Q-Network (DQN) is employed, with dynamic latency compensation values ​​used as a key element of the system state S as input to the reinforcement learning model. In the current system state, the DQN processes the action set... Choose to make the state-action value function The largest action determines the order of data transmission. For example, in some embodiments, in a distributed system containing multiple computing nodes, when a computing node calculates the dynamic latency compensation value... Then, the Deep Q Network (DQN) will determine the overall state of the current system (including the load of each compute node, data transmission latency, and the latency compensation value of that compute node). (etc.), selecting the optimal data transmission path and processing order to accelerate distributed machine learning in financial big data. This is achieved by using dynamic latency compensation values... When combined with reinforcement learning algorithms, the system can dynamically adjust the data processing flow according to the actual situation, effectively improving the efficiency of machine learning.

[0049] Furthermore, the financial big data feature analysis model employs convolutional neural networks to extract features from financial big data. By using different convolutional kernels, it extracts local features from financial big data and obtains the temporal features of the financial big data. Volatility characteristics and real-time characteristics .

[0050] This embodiment details the process of feature extraction using a Convolutional Neural Network (CNN) in a financial big data feature analysis model. CNNs possess powerful local feature extraction capabilities; by using convolutional kernels of different sizes and parameters, multi-level feature extraction can be performed on financial big data. For example, when processing foreign exchange transaction data, smaller convolutional kernels can extract short-term fluctuation features, while larger kernels can capture long-term trend features, thus accurately obtaining the time-series features of financial big data. Volatility characteristics and real-time characteristics This serves as the subsequent dynamic delay compensation value. Provides an accurate data foundation.

[0051] Furthermore, nonlinear mapping functions and Gaussian radial basis functions are used: , ,in, , , , These are function parameters.

[0052] In this embodiment, the nonlinear mapping function is defined. and Gaussian radial basis functions (RBF) are employed. RBF can perform nonlinear transformations on the data, better fitting the trends of computing node load and data transmission delay variations, as well as dynamic delay compensation values. The complex relationships between them. This is used to calculate the trend of node load changes. For example, middle, , The function can be trained and adjusted based on historical data, enabling it to reasonably adjust the dynamic latency compensation value according to different load change trends. This increases the contribution of [the system / mechanism] and improves the accuracy of delay compensation.

[0053] Furthermore, the reinforcement learning algorithm employs a deep Q-network to fit the state-action value function by constructing a neural network. .

[0054] In this embodiment, the reinforcement learning algorithm is described using a Deep Q-Network (DQN). The Deep Q-Network (DQN) fits the state-action value function by constructing a neural network. It can handle high-dimensional and complex system state spaces. In distributed machine learning scenarios for financial big data, the system state includes numerous factors, such as the load of each computing node, data transmission latency, and data characteristics. Deep Q-Networks (DQNs) utilize an experience replay mechanism to store the state, action, reward, and next state of each decision-making process in an experience pool, and randomly select samples for training. This effectively solves the data correlation problem, improves learning efficiency, and thus more accurately determines the data transmission and processing order, achieving the goal of acceleration.

[0055] Furthermore, in calculating the dynamic delay compensation value This also includes dynamic delay compensation values. Outlier detection and correction are performed by setting a threshold range. ,like Then let ;like Then let .

[0056] In this embodiment, the dynamic delay compensation value Outlier detection and correction are performed. In actual calculations, unreasonable dynamic delay compensation values ​​may occur due to data fluctuations, model errors, and other factors. Set threshold range ,For example , If calculated Then it will be corrected to ;like Then it will be corrected to To ensure dynamic delay compensation value Within reasonable limits, improve the stability and reliability of the system.

[0057] Furthermore, after adjusting the data transmission and processing order of each computing node, a feedback adjustment mechanism is also included to dynamically adjust the financial big data analysis model, the computing node state prediction model, and the dynamic latency compensation value based on the training results of the machine learning model and the real-time system status. Parameters in the calculation formula.

[0058] In this embodiment, a feedback adjustment mechanism will be established. After adjusting the data transmission and processing order of each computing node, the parameters in the financial big data feature analysis model, the computing node state prediction model, and the dynamic latency compensation value calculation formula will be dynamically adjusted based on the training results of the machine learning model (such as prediction accuracy, mean squared error, etc.) and the real-time system status (such as current computing node load, data transmission latency, etc.). For example, if the accuracy of model training is found to be low, it indicates that the current parameter settings may be inappropriate. The feedback adjustment mechanism will be used to optimize the relevant parameters to improve the training effect of the model.

[0059] Furthermore, the feedback adjustment mechanism uses a particle swarm optimization algorithm to optimize the parameters.

[0060] In this embodiment, the feedback mechanism employs a particle swarm optimization (PSO) algorithm to optimize the parameters. PSO simulates the foraging behavior of bird flocks, iteratively searching the solution space. In this embodiment, the parameters in the financial big data feature analysis model, the computing node state prediction model, and the dynamic latency compensation value calculation formula are used as particle positions. By continuously updating the particle positions, the optimal parameter combination is found, significantly improving the training efficiency and accuracy of the machine learning model.

[0061] Furthermore, the machine learning model employs an ensemble learning approach.

[0062] In this embodiment, the ensemble learning approach used in the machine learning model will be described in detail. This model integrates the prediction results of multiple sub-models of different types (such as random forests, gradient boosting trees, and neural networks). Different sub-models have different characteristics and advantages. Random forests can handle high-dimensional data and have good noise resistance; gradient boosting trees perform well in regression and classification problems; and neural networks excel at handling complex nonlinear relationships. By using the ensemble learning approach, the prediction results of these sub-models are merged, thereby improving the model's generalization ability and prediction accuracy, and better adapting to the complex characteristics of financial big data.

[0063] Furthermore, the computing nodes include edge computing nodes, cloud server computing nodes, and local computing nodes to form a multi-level collaborative computing architecture.

[0064] In this embodiment, the computing nodes are defined as edge computing nodes, cloud server computing nodes, and local computing nodes, forming a multi-level collaborative computing architecture. Edge computing nodes can perform preliminary processing at the source of data generation, reducing data transmission volume and latency. For example, edge computing nodes can be set up at bank ATMs to preprocess transaction data. Cloud server computing nodes have powerful computing and storage capabilities to handle complex machine learning tasks. Local computing nodes can meet the rapid response requirements of local data processing. The three work together to improve the overall performance and data processing efficiency of the system.

[0065] Furthermore, data transmission employs a content-addressable distributed storage system and incorporates blockchain technology for data transmission.

[0066] This embodiment will describe in detail the use of a content-addressed distributed storage system (DHT) for data transmission, combined with blockchain technology. The content-addressed distributed storage system (DHT) can efficiently store and retrieve data, quickly locating data through content addressing. Blockchain technology ensures the security and integrity of data transmission. In the process of big financial data transmission, every data transmission is recorded on the blockchain, preventing data tampering and forgery, and ensuring data credibility and security.

[0067] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for accelerating distributed machine learning of financial big data based on dynamic latency compensation, characterized in that, Includes the following steps: A financial big data feature analysis model is established to extract time sequence features from the input financial big data , volatility features , and real-time features ; Constructing a computing node state prediction model based on historical load information , historical data transmission delay information , and current load information and data transmission delay information , using a long short-term memory network to predict the load change trend and data transmission delay change trend of each computing node within T time steps The time series features Volatility characteristics and real-time characteristics Combined with the prediction results of the computing node state prediction model, the dynamic delay compensation value of each computing node at the current moment is calculated using a computational formula. The calculation formula is: in, , and It is a nonlinear mapping function; According to the dynamic delay compensation value Furthermore, the data transmission and processing order of each computing node is adjusted by combining reinforcement learning algorithms to accelerate the distributed machine learning of financial big data. In the current system state Below, from the action set The action that maximizes the state-action value function Q(s,a) is selected to determine the data transmission order O(i,t). The current system state includes the load of each computing node, the data transmission delay, and the delay compensation value of that computing node. The data transmission order satisfies the formula: Where s represents the current system state. For a set of actions, This is the state-action value function.

2. The distributed machine learning acceleration method for financial big data based on dynamic latency compensation according to claim 1, characterized in that, The financial big data feature analysis model employs a convolutional neural network to extract features from the financial big data. By using different convolutional kernels, it extracts local features from the financial big data to obtain its temporal features. Volatility characteristics and real-time characteristics .

3. The distributed machine learning acceleration method for financial big data based on dynamic latency compensation according to claim 1, characterized in that, The nonlinear mapping function and Gaussian radial basis functions are used: in, , , , These are function parameters.

4. The distributed machine learning acceleration method for financial big data based on dynamic latency compensation according to claim 1, characterized in that, The reinforcement learning algorithm employs a deep Q-network, which fits the state-action value function by constructing a neural network. .

5. The distributed machine learning acceleration method for financial big data based on dynamic latency compensation according to claim 1, characterized in that, Calculate the dynamic delay compensation value This also includes dynamic delay compensation values. Outlier detection and correction are performed, specifically as follows: Set threshold range , like Then let ; like Then let .

6. The distributed machine learning acceleration method for financial big data based on dynamic latency compensation according to claim 1, characterized in that, After adjusting the data transmission and processing order of each computing node, a feedback adjustment mechanism is also included to dynamically adjust the financial big data analysis model, the computing node state prediction model, and the dynamic latency compensation value based on the training results of the machine learning model and the real-time system status. Parameters in the calculation formula.

7. The distributed machine learning acceleration method for financial big data based on dynamic latency compensation according to claim 6, characterized in that, The feedback adjustment mechanism uses a particle swarm optimization algorithm to optimize the parameters.

8. The distributed machine learning acceleration method for financial big data based on dynamic latency compensation according to claim 6, characterized in that, The machine learning model employs an ensemble learning approach.

9. The distributed machine learning acceleration method for financial big data based on dynamic latency compensation according to claim 1, characterized in that, The computing nodes include edge computing nodes, cloud server computing nodes, and local computing nodes to form a multi-level collaborative computing architecture.

10. The distributed machine learning acceleration method for financial big data based on dynamic latency compensation according to claim 1, characterized in that, The data transmission employs a content-addressable distributed storage system and incorporates blockchain technology.

Images