Financial big data distributed machine learning acceleration method based on dynamic time delay compensation

Abstract

The invention provides a financial big data distributed machine learning acceleration method based on dynamic time delay compensation, and the method comprises the steps: building a financial big data feature analysis model, and extracting a time sequence feature # imgabs0 #, a volatility feature # imgabs1 # and a real-time feature # imgabs2 # from input financial big data; constructing a calculation node state prediction model, and obtaining a prediction result; and combining the time sequence characteristic # imgabs3 #, the volatility characteristic # imgabs4 # and the real-time characteristic # imgabs5 # with the prediction result of the computational node state prediction model to calculate a dynamic time delay compensation value # imgabs6 #, and adjusting the data transmission and processing sequence of each computational node according to the dynamic time delay compensation value # imgabs7 # in combination with a reinforcement learning algorithm to accelerate financial big data distributed machine learning. According to the method, the accurate dynamic time delay compensation value is calculated through the characteristics of the financial big data and the prediction result, so that data transmission and processing can better adapt to the system, the waiting time caused by time delay is shortened, the parallel computing efficiency is improved, and the training efficiency is further improved.

Description

Technical Field

[0001] The present invention relates to the field of financial big data processing and machine learning technology, and in particular to a financial big data distributed machine learning acceleration method based on dynamic delay compensation. Background Art

[0002] In the field of financial big data processing, distributed machine learning significantly improves data processing efficiency by distributing massive amounts of data across multiple computing nodes for parallel processing. However, existing technologies still have many shortcomings. Traditional distributed machine learning delay compensation methods often use fixed parameters or simple linear weighting strategies. For example, they set a fixed compensation time based on historical average delay, or simply calculate a dynamic delay compensation value based on node load and transmission delay. These methods cannot fully adapt to the dynamic characteristics of financial data and the complex network environment.

[0003] Financial data is highly time-series, volatile, and real-time. For example, in high-frequency trading scenarios, where market conditions fluctuate rapidly, real-time data processing is crucial. In risk assessment scenarios, data accuracy and completeness have a profound impact on the results. Existing delay compensation methods cannot precisely match the specific delay requirements of different financial data processing tasks, nor can they effectively address unexpected situations such as network congestion and node failures. Furthermore, existing methods lack the ability to predict future delay changes, preventing them from optimizing data transmission and processing sequences in advance, resulting in inefficient model training.

[0004] In summary, the technical problem actually solved by the present invention is how to improve the training efficiency of the model. Summary of the Invention

[0005] In order to overcome the low training efficiency of the model in the above-mentioned prior art, the purpose of the present invention is to provide a financial big data distributed machine learning acceleration method based on dynamic delay compensation, so as to calculate an accurate dynamic delay compensation value through the characteristics of financial big data and the prediction results of computing node status, so that data transmission and processing can better adapt to the system status, thereby reducing the waiting time caused by delay, improving the efficiency of parallel computing, and thus improving training efficiency.

[0006] The present invention discloses a distributed machine learning acceleration method for financial big data based on dynamic delay compensation, comprising the following steps:

[0007] Establish a financial big data feature analysis model to extract time series features from the input financial big data , volatility characteristics and real-time features ;

[0008] Build a computing node status prediction model based on historical load information , historical data transmission delay information And the current load information and data transmission delay information , using long short-term memory network to predict the load change trend of each computing node within T time steps and data transmission delay change trends ;

[0009] The time series features , volatility characteristics and real-time features Combined with the prediction results of the computing node state prediction model, the dynamic delay compensation value of each computing node at the current moment is calculated by the calculation formula , the calculation formula is:

[0010]

[0011] in, , and is a nonlinear mapping function;

[0012] According to the dynamic delay compensation value Combined with reinforcement learning algorithms, it adjusts the data transmission and processing order of each computing node to accelerate distributed machine learning of financial big data;

[0013] The adjustment of data transmission and processing order satisfies the formula:

[0014]

[0015] Where s is the current system state, is a set of actions, is the state-action value function.

[0016] Preferably, the financial big data feature analysis model uses a convolutional neural network to extract features from financial big data, so as to extract local features of financial big data through different convolution kernels and obtain the temporal features of financial big data. , volatility characteristics and real-time features .

[0017] Preferably, the nonlinear mapping function and Using Gaussian radial basis function:

[0018]

[0019]

[0020] in, 、 、 、 For function parameters.

[0021] Preferably, the reinforcement learning algorithm uses a deep Q network to fit the state-action value function by constructing a neural network .

[0022] Preferably, when calculating the dynamic delay compensation value Then also includes the dynamic delay compensation value Perform outlier detection and correction, specifically:

[0023] Set threshold range ,

[0024] like , then let ;

[0025] like , then let .

[0026] Preferably, after adjusting the data transmission and processing order of each computing node, a feedback adjustment mechanism is also established to dynamically adjust the financial big data analysis model, computing node status prediction model and dynamic delay compensation value according to the training results of the machine learning model and the real-time system status. Parameters in the calculation formula.

[0027] Preferably, the feedback adjustment mechanism uses a particle swarm optimization algorithm to optimize parameters.

[0028] Preferably, the machine learning model adopts an ensemble learning method.

[0029] Preferably, the computing nodes include edge computing nodes, cloud server computing nodes and local computing nodes to form a multi-level collaborative computing architecture.

[0030] Preferably, data transmission adopts a content-addressed distributed storage system and combines blockchain technology for data transmission.

[0031] After adopting the above technical solution, compared with the existing technology, the beneficial effect of the present invention is that it calculates accurate dynamic delay compensation values based on the characteristics of financial big data and the prediction results of computing node status, so that data transmission and processing can better adapt to the system status, thereby reducing the waiting time caused by delay, improving the efficiency of parallel computing, and thus improving training efficiency.

[0032] The financial big data feature analysis model utilizes a convolutional neural network, effectively extracting the temporal, volatility, and real-time characteristics of financial big data, providing an accurate data foundation for subsequent calculations. The compute node state prediction model employs a long short-term memory network to accurately predict future trends in compute node load and data transmission latency. Furthermore, the nonlinear mapping function utilizes a Gaussian radial basis function, and the feedback adjustment mechanism employs a particle swarm optimization algorithm. The combination of these models and algorithms, from feature extraction and system state prediction to parameter optimization, ensures efficient system operation and further improves training efficiency.

[0033] Computing nodes utilize a multi-level collaborative computing architecture consisting of edge computing nodes, cloud server computing nodes, and local computing nodes. Edge computing nodes perform preliminary processing at the source of data, reducing data transmission volume and latency. Cloud server computing nodes provide powerful computing and storage capabilities to handle complex tasks. Local computing nodes meet local needs for rapid response. Furthermore, data transmission utilizes a content-addressable distributed storage system combined with blockchain technology. The distributed storage system enables efficient data storage and retrieval, while blockchain technology ensures the security and integrity of data transmission, reducing errors and latency during data transmission, and further improving training efficiency.

[0034] The machine learning model uses an ensemble learning approach to improve its generalization and prediction accuracy. A feedback adjustment mechanism is also established. Based on the machine learning model's training results and real-time system status, a particle swarm optimization algorithm is used to dynamically adjust the parameters in the financial big data feature analysis model, the computing node status prediction model, and the dynamic delay compensation value calculation formula. This allows the system to continuously adapt to changes in data and the environment, continuously optimize the model training process, and further improve training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the steps of a distributed machine learning acceleration method for financial big data based on dynamic delay compensation of the present invention. DETAILED DESCRIPTION

[0036] The advantages of the present invention are further described below with reference to the accompanying drawings and specific embodiments.

[0037] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0038] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0039] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0040] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0041] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0042] In the following description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0043] This embodiment discloses a distributed machine learning acceleration method for financial big data based on dynamic delay compensation, which includes the following steps: establishing a financial big data feature analysis model to extract time series features from the input financial big data; , volatility characteristics and real-time features ;Build a computing node status prediction model based on historical load information , historical data transmission delay information And the current load information and data transmission delay information , using long short-term memory network to predict the load change trend of each computing node within T time steps and data transmission delay change trends ; The time series features , volatility characteristics and real-time features Combined with the prediction results of the computing node state prediction model, the dynamic delay compensation value of each computing node at the current moment is calculated by the calculation formula , the calculation formula is: ,in, , and is a nonlinear mapping function; according to the dynamic delay compensation value Combined with the reinforcement learning algorithm, the data transmission and processing order of each computing node is adjusted to accelerate the distributed machine learning of financial big data; the adjustment of data transmission and processing order satisfies the formula: , where s is the current system state, is a set of actions, is the state-action value function.

[0044] In this embodiment, see Figure 1 As shown in the figure, a financial big data distributed machine learning acceleration method based on dynamic delay compensation is described in detail, which specifically includes the following steps:

[0045] S100: Establish a financial big data feature analysis model to extract time series features from the input financial big data , volatility characteristics and real-time features In this step, a financial big data feature analysis model will be established, which is the basic environment of the entire method. Taking the processing of futures trading data as an example, a convolutional neural network (CNN) is used to process futures trading data, and local features are extracted through different convolution kernels. In this process, the convolutional neural network (CNN) can capture the changing pattern of futures prices in different time periods, thereby obtaining the time series characteristics of futures trading data. , such as short-term price fluctuation trends and long-term trends; volatility characteristics , that is, the severity of price fluctuations; real-time characteristics , such as the frequency and timeliness of the latest transaction prices. These characteristics reflect the essential characteristics of financial big data and provide a key basis for subsequent processing and calculation.

[0046] Step S200: Construct a computing node status prediction model based on historical load information , historical data transmission delay information And the current load information and data transmission delay information , using long short-term memory network to predict the load change trend of each computing node within T time steps and data transmission delay change trends In this step, a computing node status prediction model will be constructed based on historical load information. , historical data transmission delay information And the current load information and data transmission delay information , using the long short-term memory network (LSTM) to predict the load change trend of each computing node in the next T time steps and data transmission delay change trends For example, in some embodiments, in a financial big data processing cluster containing multiple computing nodes, a long short-term memory network (LSTM) can predict future computing node load and data transmission delay changes by learning information such as the CPU usage, memory utilization, and round-trip time of each computing node over a period of time. The computing node state prediction model provides a prediction of future system states for subsequent dynamic delay compensation calculations and is an important prerequisite for achieving dynamic adjustment.

[0047] Step S300: In this step, the time series features extracted in step S100 are , volatility characteristics and real-time features , and, the prediction results of the node state prediction model calculated in step S200 are combined, and the calculation formula is Calculate dynamic delay compensation value In this calculation formula, 、 、 is the weight coefficient, and By weighting the characteristics and node status prediction trends, the influence of each factor on the delay step is reasonably allocated. Nonlinear mapping function and (Using Gaussian Radial Basis Function (RBF)) further performs nonlinear transformation on the trend of changes in computing node load and data transmission delay, so that the calculation results are more in line with the actual situation. For example, in some embodiments, for a node that processes foreign exchange transaction data, if , , ,predict , , substituted into the calculation formula to calculate the dynamic delay compensation value for the computing node. This step organically combines characteristics and system prediction to achieve accurate quantification of the delay compensation value.

[0048] Step S400: In this step, according to the dynamic delay compensation value , combined with the reinforcement learning algorithm to adjust the data transmission and processing order of each computing node. In this way, the deep Q network (DQN) is used to input the dynamic delay compensation value as the key factor of the system state S into the reinforcement learning model. In the current system state, the deep Q network (DQN) Choose the state-action value function The largest action, thus determining the data transmission order For example, in some embodiments, in a distributed system comprising multiple computing nodes, when a computing node calculates a dynamic delay compensation value After that, the Deep Q Network (DQN) will calculate the delay compensation value of the computing node based on the overall status of the current system (including the load of each computing node, data transmission delay, and the delay compensation value of the computing node). etc.), select the optimal data transmission path and processing order to accelerate the distributed machine learning of financial big data. Combined with reinforcement learning algorithms, the system can dynamically adjust the data processing process according to actual conditions, effectively improving the efficiency of machine learning.

[0049] Furthermore, the financial big data feature analysis model uses convolutional neural networks to extract features from financial big data, extracting local features of financial big data through different convolution kernels and obtaining the temporal features of financial big data. , volatility characteristics and real-time features .

[0050] This example describes in detail the process of extracting features using a convolutional neural network (CNN) in a financial big data feature analysis model. CNNs have powerful local feature extraction capabilities and can perform multi-level feature extraction on financial big data using convolution kernels of varying sizes and parameters. For example, when processing foreign exchange trading data, a smaller convolution kernel can extract short-term fluctuations in the data, while a larger convolution kernel can capture long-term trends, thereby accurately capturing the temporal characteristics of financial big data. , volatility characteristics and real-time features , is the subsequent dynamic delay compensation value Provide an accurate data foundation.

[0051] Furthermore, the nonlinear mapping function and Using Gaussian radial basis function: , ,in, 、 、 、 For function parameters.

[0052] In this embodiment, the nonlinear mapping function is clarified. and Gaussian Radial Basis Function (RBF) is used. Gaussian Radial Basis Function (RBF) can perform nonlinear transformation on data, better fitting the changing trend of computing node load and data transmission delay and dynamic delay compensation value. To calculate the node load change trend For example, middle, 、 Training and adjustment can be performed based on historical data, so that the function can reasonably adjust the dynamic delay compensation value according to different load change trends. The contribution degree of the delay compensation is improved.

[0053] Furthermore, the reinforcement learning algorithm uses a deep Q network to fit the state-action value function by constructing a neural network .

[0054] In this embodiment, the reinforcement learning algorithm is described using the Deep Q Network (DQN). The Deep Q Network (DQN) fits the state-action value function by constructing a neural network. , capable of processing high-dimensional and complex system state spaces. In distributed machine learning scenarios for financial big data, system state is influenced by numerous factors, such as the load of each computing node, data transmission latency, and data characteristics. The Deep Q Network (DQN) utilizes an experience replay mechanism, storing the state, action, reward, and next state of each decision-making process in an experience pool. Training is then performed on randomly selected samples, effectively addressing data correlation issues and improving learning efficiency. This allows for more accurate determination of data transmission and processing sequences, achieving acceleration.

[0055] Furthermore, when calculating the dynamic delay compensation value Then also includes the dynamic delay compensation value Perform outlier detection and correction, specifically: setting the threshold range ,like , then let ;like , then let .

[0056] In this embodiment, the dynamic delay compensation value Perform outlier detection and correction. In actual calculations, unreasonable dynamic delay compensation values may occur due to data fluctuations, model errors, etc. , set the threshold range ,For example , , if calculated , then correct it to ;like , then correct it to , ensuring dynamic delay compensation value Within a reasonable range, improve the stability and reliability of the system.

[0057] Furthermore, after adjusting the data transmission and processing order of each computing node, it also includes establishing a feedback adjustment mechanism to dynamically adjust the financial big data analysis model, computing node status prediction model and dynamic delay compensation value according to the training results of the machine learning model and the real-time system status. Parameters in the calculation formula.

[0058] In this embodiment, a feedback adjustment mechanism is established. After adjusting the data transmission and processing order of each computing node, the parameters in the financial big data feature analysis model, the computing node status prediction model, and the dynamic delay compensation value calculation formula are dynamically adjusted based on the machine learning model training results (such as prediction accuracy and mean squared error) and real-time system status (such as current computing node load and data transmission delay). For example, if the model training accuracy is low, it indicates that the current parameter settings may be inappropriate. The feedback adjustment mechanism can then be used to optimize the relevant parameters to improve the model training effect.

[0059] Furthermore, the feedback adjustment mechanism uses particle swarm optimization algorithm to optimize the parameters.

[0060] This example demonstrates a feedback mechanism that uses the particle swarm optimization (PSO) algorithm to optimize parameters. PSO simulates the foraging behavior of a flock of birds, performing an iterative search within the solution space. In this example, the parameters in the financial big data feature analysis model, the computing node status prediction model, and the dynamic delay compensation value calculation formula are used as particle positions. By continuously updating the particle positions, the optimal parameter combination is found, significantly improving the training efficiency and accuracy of the machine learning model.

[0061] Furthermore, the machine learning model adopts an ensemble learning method.

[0062] This example describes a machine learning model using an ensemble learning approach. This model combines the prediction results of multiple sub-models (e.g., random forests, gradient boosted trees, and neural networks). Each sub-model has distinct characteristics and advantages. Random forests can handle high-dimensional data and are highly resistant to noise, gradient boosted trees excel at regression and classification, and neural networks are adept at handling complex nonlinear relationships. By combining the prediction results of these sub-models through ensemble learning, the model's generalization and prediction accuracy are improved, making it more adaptable to the complex nature of financial big data.

[0063] Furthermore, the computing nodes include edge computing nodes, cloud server computing nodes and local computing nodes to form a multi-level collaborative computing architecture.

[0064] This embodiment specifies that computing nodes include edge computing nodes, cloud server computing nodes, and local computing nodes, forming a multi-level collaborative computing architecture. Edge computing nodes can perform preliminary processing at the source of data, reducing data transmission volume and latency. For example, edge computing nodes can be deployed at bank ATMs to pre-process transaction data. Cloud server computing nodes possess powerful computing and storage capabilities for handling complex machine learning tasks, while local computing nodes can meet the rapid response requirements of local data processing. The three work together to improve the overall system performance and data processing efficiency.

[0065] Furthermore, data transmission adopts a content-addressed distributed storage system and combines blockchain technology for data transmission.

[0066] This example describes how data transmission uses a content-addressed distributed storage system (DHT) and integrates it with blockchain technology. The DHT efficiently stores and retrieves data, quickly locating data through content addressing. Blockchain technology ensures the security and integrity of data transmission. During the transmission of financial big data, every data transaction is recorded on the blockchain, preventing data tampering and forgery and ensuring data credibility and security.

[0067] It should be noted that the embodiments of the present invention have better practicability and do not impose any form of limitation on the present invention. Any technician familiar with the field may use the technical content disclosed above to change or modify it into an equivalent effective embodiment. However, any modification or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A distributed machine learning acceleration method for financial big data based on dynamic delay compensation, characterized in that: The following steps are involved: Establish a financial big data feature analysis model to extract time series features from the input financial big data , volatility characteristics and real-time features ; Build a computing node status prediction model based on historical load information , historical data transmission delay information And the current load information and data transmission delay information , using long short-term memory network to predict the load change trend of each computing node within T time steps and data transmission delay change trends ; The timing characteristics , volatility characteristics and real-time features Combined with the prediction results of the computing node state prediction model, the dynamic delay compensation value of each computing node at the current moment is calculated by the calculation formula , the calculation formula is: in, , and is a nonlinear mapping function; According to the dynamic delay compensation value and combining a reinforcement learning algorithm to adjust the data transmission and processing order of each computing node to accelerate the financial big data distributed machine learning; The data transmission and the adjustment of the processing order satisfy the formula: Where s is the current system state, is a set of actions, is the state-action value function.

2. The financial big data distributed machine learning acceleration method based on dynamic delay compensation according to claim 1 is characterized in that: The financial big data feature analysis model uses a convolutional neural network to extract features from the financial big data, extracting local features of the financial big data through different convolution kernels to obtain the temporal features of the financial big data. , volatility characteristics and real-time features .

3. The financial big data distributed machine learning acceleration method based on dynamic delay compensation according to claim 1 is characterized in that: The nonlinear mapping function and Using Gaussian radial basis function: in, 、 、 、 For function parameters.

4. The financial big data distributed machine learning acceleration method based on dynamic delay compensation according to claim 1 is characterized in that: The reinforcement learning algorithm uses a deep Q network to construct a neural network to fit the state-action value function .

5. The financial big data distributed machine learning acceleration method based on dynamic delay compensation according to claim 1 is characterized in that: In calculating the dynamic delay compensation value Then also includes the dynamic delay compensation value Perform outlier detection and correction, specifically: Set threshold range , like , then let ; like , then let .

6. The financial big data distributed machine learning acceleration method based on dynamic delay compensation according to claim 1 is characterized in that: After adjusting the data transmission and processing order of each computing node, it also includes establishing a feedback adjustment mechanism to dynamically adjust the financial big data analysis model, the computing node status prediction model and the dynamic delay compensation value according to the training results of the machine learning model and the real-time system status. Parameters in the calculation formula.

7. The financial big data distributed machine learning acceleration method based on dynamic delay compensation according to claim 6 is characterized in that: The feedback adjustment mechanism uses a particle swarm optimization algorithm to optimize parameters.

8. The financial big data distributed machine learning acceleration method based on dynamic delay compensation according to claim 6 is characterized in that: The machine learning model adopts an ensemble learning method.

9. The financial big data distributed machine learning acceleration method based on dynamic delay compensation according to claim 1 is characterized in that: The computing nodes include edge computing nodes, cloud server computing nodes and local computing nodes to form a multi-level collaborative computing architecture.

10. The financial big data distributed machine learning acceleration method based on dynamic delay compensation according to claim 1 is characterized in that: The data transmission adopts a distributed storage system based on content addressing and combines blockchain technology for data transmission.

Images