A throughput optimization method and system for big data streaming machine learning

Abstract

The application discloses a kind of throughput rate optimization method and system for big data streaming machine learning.The present system includes the flow rate monitoring module and sampling control module deployed on computing node, and parameter optimization module and data sampling module deployed on management node;Parameter optimization module is used to obtain the data type of streaming data and selects a data sampling strategy, sends data sampling strategy and its parameters to data sampling module;And according to new sampling rate, model training throughput rate optimization current data sampling strategy parameter setting or select new data sampling strategy and send processing result to data sampling module;Sampling control module is used to calculate new sampling rate according to flow rate and model training throughput rate and send it to parameter optimization module;Data sampling module is used to sample streaming data for model training of streaming machine learning task;Flow rate monitoring module is used to monitor the flow rate of data source and model training throughput rate.

Description

Technical Field

[0001] The present invention relates to a throughput optimization method for a streaming machine learning system, and in particular to a throughput optimization method and system for a big data streaming machine learning system, belonging to the field of computer software technology. Background Art

[0002] Machine learning is the primary implementation method for artificial intelligence (AI). Through machine learning, AI has been successfully implemented in areas such as public welfare, healthcare, education, and finance, driving profound changes across multiple industries. Traditionally, when applying machine learning algorithms, an optimal algorithm model is trained using the entire dataset and then used for online predictions. This approach is known as batch machine learning. Currently, with the increasing popularity of scenarios such as digital power grids, the scale of edge IoT sensors has increased dramatically, and streaming data has become a significant form of big data. Streaming machine learning, by incrementally updating algorithm models in real time, can adapt to the boundless and dynamic nature of streaming data. It has been applied to scenarios such as sensor data monitoring and real-time data analysis and processing in digital power grids. To support streaming machine learning applications and fast streaming machine learning tasks, a series of mature streaming machine learning systems, such as Flink ML, Alink, and River, have emerged.

[0003] In the context of big data, streaming data is characterized by high speed and dynamic changes. To cope with this high-speed streaming data, streaming machine learning systems must have high throughput rates to process incoming data in a timely manner, enabling machine learning algorithms to learn from the latest data. Furthermore, the dynamic nature of streaming data means that its flow rate can change with changes in upstream business. However, in an environment with fixed hardware conditions, the throughput limit of a streaming machine learning system is fixed. If the flow rate of streaming data exceeds this limit, a serious data backlog will occur in the system, preventing the machine learning algorithm from learning from the latest data and making the algorithm no longer real-time.

[0004] Existing methods for optimizing the throughput of streaming machine learning systems primarily focus on optimizing operators and hardware resources. One approach involves optimizing the specific implementation of various operators in the system to improve their execution efficiency, thereby increasing the overall throughput of the system. With this approach, the upper limit of the system's throughput remains fixed when the hardware configuration is fixed. Furthermore, optimizing the execution efficiency of operators is difficult to achieve and yields low returns. A second approach involves deploying the streaming machine learning system using a resource management platform such as Kubernetes. The resource management platform dynamically adjusts the system's available hardware resources based on the upstream data flow rate, enabling dynamic scaling of the system and, in turn, dynamically adjusting the system's throughput to achieve optimization. However, this approach requires redundant hardware resources and relies on external platforms.

[0005] In summary, the existing stream machine learning system throughput optimization method mainly focuses on optimizing the internal operator of the system or expanding or shrinking the available hardware resources from the outside, and there is still a lack of optimization method for dynamically realizing stream speed matching to improve the overall throughput of the stream machine learning system under the condition of fixed hardware resources. SUMMARY

[0006] In view of the problems in the prior art, the purpose of the present application is to provide a throughput optimization method and system for big data stream machine learning, which can dynamically adjust the data stream speed to match the system processing capacity and realize throughput optimization.

[0007] The technical solution of the present application:

[0008] A throughput optimization method for big data stream machine learning, comprising the following steps:

[0009] 1) Start the data sampling module and the stream speed monitoring module on the computing node, and start the parameter optimization module and the sampling control module on the management node;

[0010] 2) The sampling control module determines the initial sampling rate and sets the stream speed reporting time interval;

[0011] 3) The parameter optimization module obtains the data type of the stream data provided by the data source and selects a data sampling strategy; then determines the initial parameters in the data sampling strategy according to the initial sampling rate, and sends the data sampling strategy and its parameters to the data sampling module;

[0012] 4) The sampling control module notifies the stream machine learning system on each computing node to start executing the stream machine learning task;

[0013] 5) The data sampling module samples each batch of stream data generated by the current data source according to the current data sampling strategy and its parameters, and sends the sampling result to the stream machine learning system for model training of the stream machine learning task;

[0014] 6) After the stream machine learning task starts to execute, the stream speed monitoring module continuously monitors the stream speed of the data source and the model training throughput; then according to the set stream speed reporting time interval, the stream speed and the model training throughput are sent to the sampling control module, and the model training throughput is sent to the parameter optimization module;

[0015] 7) The sampling control module calculates a new sampling rate according to the stream speed of the data source and the model training throughput, and sends it to the parameter optimization module;

[0016] 8) The parameter optimization module optimizes the parameter settings of the current data sampling strategy or selects a new data sampling strategy based on the new sampling rate and model training throughput, and sends the processing results to the data sampling module;

[0017] 9) The data sampling module samples each batch of streaming data generated by the currently received data source according to the processing result of step 8);

[0018] 10) While the streaming machine learning task is running, continue to execute steps 5) to 9) to optimize the model training throughput.

[0019] Furthermore, the sampling control module obtains the model training throughput rate T and the flow rate S of the data source, and calculates a new sampling rate r = T / S*alpha; then sends the new sampling rate r to the data sampling module, and the data sampling module samples each batch of streaming data generated by the currently received data source according to the new sampling rate r; wherein alpha is used to match the input and output flow rates.

[0020] Furthermore, alpha=0.95 is selected.

[0021] Furthermore, the data sampling strategy library includes a random sampling strategy, and its sampling method is: for each data sample in the streaming data, generate a random number n uniformly distributed between [0,1]; if the random number n is less than the set maximum sampling ratio e, then the data sample is selected; if the random number n is greater than or equal to the set maximum sampling ratio e, then the data sample is discarded.

[0022] Furthermore, the data sampling strategy library includes a distance-based core set sampling strategy, and its sampling method is:

[0023] 3.1) Read a batch of data from the streaming data and initialize the core set S to be empty;

[0024] 3.2) Calculate the Euclidean distance between each sample in the batch of data and store it as a matrix for subsequent access;

[0025] 3.3) Traverse the batch of sample data in sequence. For each data sample X, first add it to the core set S. Then calculate the sum d of the closest distances from all samples in the batch of sample data outside the core set S to the samples in the core set S. Then remove the data sample X from the core set S.

[0026] 3.4) Add the data sample corresponding to the smallest d that appears during the traversal process to the core set S;

[0027] 3.5) Repeat steps 3.3) and 3.4) until the amount of data in the core set S reaches the set maximum data amount n.

[0028] Furthermore, the sampling strategies in the data sampling strategy library are expanded by generating new sampling strategies, including sampling logic and parameters, sampling mode, and acceptable data types; wherein the sampling logic is the sampling strategy specifically applied to the data, the parameters are the specific control parameters in the sampling logic, and the sampling mode includes streaming and micro-batch.

[0029] A throughput optimization system for big data streaming machine learning, characterized by comprising a data sampling strategy library, a flow rate monitoring module, and a sampling control module deployed on computing nodes, and a parameter optimization module and a data sampling module deployed on management nodes;

[0030] Data sampling strategy library, used to store data sampling strategies;

[0031] a parameter optimization module configured to obtain the data type of streaming data provided by the data source and select a data sampling strategy; then determine initial parameters in the data sampling strategy based on the initial sampling rate, and send the data sampling strategy and its parameters to the data sampling module; and optimize the parameter settings of the current data sampling strategy or select a new data sampling strategy based on the new sampling rate and model training throughput rate, and send the processing results to the data sampling module;

[0032] The sampling control module is used to determine the initial sampling rate and set the flow rate reporting interval, notify the streaming machine learning system on each computing node to start executing the streaming machine learning task, and calculate the new sampling rate based on the flow rate of the data source and the model training throughput rate and send it to the parameter optimization module;

[0033] The data sampling module is used to sample each batch of streaming data generated by the currently received data source according to the current data sampling strategy and its parameters, and send the sampling results to the streaming machine learning system for model training of the streaming machine learning task;

[0034] The flow rate monitoring module is used to continuously monitor the flow rate of the data source and the model training throughput after the streaming machine learning task starts to execute; then, according to the set flow rate reporting time interval, the flow rate and the model training throughput are sent to the sampling control module, and the model training throughput is sent to the parameter optimization module.

[0035] The present invention is directed to a throughput optimization system for a big data streaming machine learning system. Figure 1 As shown in the figure, it includes computing nodes for executing streaming machine learning tasks and management nodes for sampling control, including data sampling strategy library, parameter optimization module, data sampling module, flow rate monitoring module and sampling control module. The deployment method, main responsibilities, interaction mode and details of each module are as follows:

[0036] Data sampling strategy library: deployed on computing nodes, contains a set of strategies for sampling a portion of representative data from the original streaming data. The sampling strategy needs to guarantee the representativeness of the sampled data to the original streaming data as much as possible while reducing the amount of data, that is, the same query task (such as machine learning task) on the original streaming data and the sampled data should be as consistent as possible.

[0037] Parameter optimization module: deployed on the management node, receives the input of the flow rate monitoring module, data sampling module and data sampling strategy library, outputs the optimized data sampling strategy parameters and realizes dynamic adjustment.

[0038] Sampling control module: deployed on the management node, interacts with the flow rate monitoring module to receive the input of the flow rate monitoring module and set parameters; according to the received data flow rate information, calculates the new sampling rate and notifies the data sampling module, so as to achieve the balance of the data flow rate inside and outside the system.

[0039] Data sampling module: deployed on the computing node, receives the input of the sampling control module and the parameter optimization module, responsible for processing the upstream incoming original streaming data, sampling the streaming data, and sending it to the streaming machine learning system for model training of the streaming machine learning task.

[0040] Flow rate monitoring module: deployed on the computing node, receives the input of the sampling control module, responsible for monitoring the data source flow rate and model training throughput, and regularly reports to the sampling control module.

[0041] The specific steps of the data sampling strategy library are as follows:

[0042] 1. The data sampling strategy library currently contains two data sampling strategies with different parameters, characteristics and application scope, and the data sampling strategy library can support strategy expansion.

[0043] 2. Random sampling strategy: the sampling mode of this strategy is streaming sampling, the input parameter is the maximum sampling ratio e, and it is suitable for any type of data. Random sampling strategy has wide usage, but the sampling effect may be poor. The specific sampling logic is as follows:

[0044] 2.1 For each data sample in the original streaming data, generate a random number n uniformly distributed in [0, 1];

[0045] 2.2 If the random number n is less than the set maximum sampling ratio e, select the data sample;

[0046] 2.3 If the random number n is greater than or equal to the set maximum sampling ratio e, discard the data sample.

[0047] 3. Distance-based core set sampling strategy: This strategy works by selecting a core set S from the streaming data. S is a subset of the streaming data that reflects the characteristics of the streaming data. This strategy uses micro-batch sampling as its sampling mode. Its parameter is the maximum data size n in the core set. It accepts numerical data that can be used to calculate distances. This strategy has a limited range of acceptable data types, but it provides good sampling results. The specific sampling logic is as follows:

[0048] 3.1 Read a batch of data from the data stream and initialize the core set S to be empty;

[0049] 3.2 Calculate the Euclidean distance between each sample in the batch of data and store it as a matrix for subsequent access;

[0050] 3.3 Traverse the batch of sample data in sequence. For data sample X, first add it to S, then calculate the sum d of the closest distances from all samples outside S to the samples in S, and then remove X from S;

[0051] 3.4 Add the sample X corresponding to the smallest d that appears during the traversal process to S;

[0052] 3.5 Repeat steps 3.3 and 3.4 until the amount of data in S reaches the maximum amount of data n.

[0053] 4. Follow the steps below to expand the sampling strategy library:

[0054] 4.1 The new sampling strategy includes sampling logic and parameters, sampling mode, and acceptable data types;

[0055] 4.2 Sampling logic is the sampling strategy specifically applied to the data. For example, the sampling logic of the random sampling strategy described in step 2 is to select data samples based on random numbers;

[0056] 4.3 Parameters are specific control parameters in the sampling logic, for example, the parameter of the random sampling strategy described in step 2 is the sampling rate;

[0057] 4.4 Sampling modes include streaming and micro-batch;

[0058] 4.4.1 In streaming sampling mode, each time a piece of data is read, the sampling logic described in step 4.2 is applied to it, and a decision is made as to whether to sample it and send it to the downstream streaming machine learning system;

[0059] 4.4.2 In batch sampling mode, data is first read continuously until the preset number is met, and then the sampling logic described in step 4.2 is applied to the batch data, and the sampling results are sent downstream;

[0060] 4.5 Acceptable data types are streaming data types that can be processed by the sampling strategy.

[0061] The specific steps of the parameter optimization module are as follows:

[0062] 1. Interact with the data sampling module to obtain the current batch streaming data type;

[0063] 2. Interact with the sampling control module to obtain the sampling rate of the current and historical batches;

[0064] 3. Interact with the flow rate monitoring module to obtain the model training throughput rate of historical batch streaming data;

[0065] 4. If historical model training throughput exists, determine the throughput trend. If the throughput continues to decrease during initialization or within the last several monitoring windows, reselect the data sampling strategy according to step 5. Otherwise, only change the parameters of the data sampling strategy according to step 6.

[0066] 5. When initializing or reselecting the data sampling strategy, interact with the data sampling strategy library, use the data type to filter the available data sampling strategies, and randomly select an available sampling strategy;

[0067] 6. Determine the parameters of the data sampling strategy based on the sampling rate and the maximum data inflow rate that the downstream streaming machine learning system can accept. For example, when using a core set-based sampling strategy, the maximum amount of data in the core set, n, is calculated using the sampling rate. If historical parameters already exist, change the parameters within a specified range of the current parameter value (e.g., plus or minus 10%). Otherwise, randomly select within the range.

[0068] 7. Return the selected data sampling strategy and parameters to the data sampling module.

[0069] The specific steps of the data sampling module are as follows:

[0070] 1. Receive input from the sampling control module and obtain the latest data sampling rate;

[0071] 2. Receive input from the parameter optimization module and obtain the sampling strategy and parameters of the current batch data;

[0072] 3. Sample the batch data according to the sampling strategy, and the output data meets the sampling rate requirements.

[0073] The specific steps of the flow rate monitoring module are as follows:

[0074] 1. Execute the initialization process, receive input from the sampling control module, and obtain flow rate reporting interval information from the sampling control module;

[0075] 2. Monitor the data source, record the amount of data flowing into the system within a fixed time interval, calculate the data source flow rate, and then report it to the sampling control module;

[0076] 3. Monitor the model training, record the amount of data used for model training within a fixed time interval, calculate the model training throughput, and then report to the sampling control module.

[0077] The specific steps of the sampling control module are as follows:

[0078] 1. Send flow rate reporting interval information to the flow rate monitoring module and start the flow rate monitoring module;

[0079] 2. Receive input from the flow rate monitoring module, obtain the data source flow rate S and the model training throughput T, and calculate the new sampling rate r = T / S * alpha. The parameter alpha is used to match the input and output flow rates to avoid overload during performance jitter. Here, alpha = 0.95;

[0080] 3. Send the new sampling rate r to the data sampling module, and the system starts sampling at the new sampling rate.

[0081] like Figure 2 As shown, a throughput optimization method and system for a big data streaming machine learning system, the specific steps are as follows:

[0082] 1. Start the data sampling module and flow rate monitoring module on the computing node, and start the parameter optimization module and sampling control module on the management node;

[0083] 2. The sampling control module determines the initial sampling rate and interacts with the flow rate monitoring module to set the flow rate reporting interval;

[0084] 3. The parameter optimization module obtains the data type of the first batch of streaming data and selects a data sampling strategy; it determines the initial parameters of the data sampling strategy based on the initial sampling rate of the sampling control module;

[0085] 4. The sampling control module notifies the streaming machine learning system on the computing node to start executing the streaming machine learning task;

[0086] 5. The data sampling module interacts with the parameter optimization module to obtain the current data sampling strategy and parameters, receive streaming data generated by the data source, perform sampling, and output the sampling results to the machine learning system for model training of the streaming machine learning task;

[0087] 6. The flow rate monitoring module continuously monitors the flow rate of the data source for use by the sampling control module;

[0088] 7. The sampling control module continuously receives the flow rate of the data source and the model training throughput sent by the flow rate monitoring module, and calculates the new sampling rate;

[0089] 8. The parameter optimization module receives data such as new sampling rate and model training throughput rate, optimizes parameter settings or selects new data sampling strategies;

[0090] 9. The data sampling module performs sampling according to the new sampling rate and data sampling strategy;

[0091] 10. While the streaming machine learning task is running, continue to execute steps 5 to 9 to dynamically set the sampling rate and optimize parameters to optimize the model training throughput.

[0092] The present invention proposes a throughput optimization method and system for a big data streaming machine learning system. Compared with the existing technology, the present invention has the following advantages:

[0093] 1. The present invention can increase the throughput limit of streaming machine learning systems without increasing the total amount of available software and hardware resources;

[0094] 2. The present invention can dynamically adjust the overall throughput of the streaming machine learning system without the help of an external resource management system;

[0095] 3. The present invention automatically strikes a balance between the throughput rate and model accuracy of the streaming machine learning system by dynamically adjusting the data sampling rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 It is a deployment diagram of the method and system of the present invention.

[0097] Figure 2 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0098] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0099] The throughput optimization method for streaming machine learning systems proposed in this paper can be applied to streaming machine learning systems such as Flink ML. Taking a representative streaming machine learning task as an example, its input data is of numerical type and the feature dimension is 1. The specific steps of the present invention are as follows:

[0100] 1. Start the data sampling module and flow rate monitoring module on the computing node, and start the parameter optimization module and sampling control module on the management node;

[0101] 2. Since the input data is numerical and the feature dimension is 1, the parameter optimization module selects the matching data sampling strategy as distance-based core set sampling;

[0102] 3. The sampling control module determines the data batch size for each sampling to be 5, the initial sampling rate to be 0.6, starts the flow rate monitoring module and sets the flow rate reporting interval to 1 second;

[0103] 4. The sampling control module notifies the machine learning system that initialization is complete and begins executing the streaming machine learning task;

[0104] 5. After the streaming machine learning task begins, the data sampling module reads five data points, A, B, C, D, and E. Following the core set sampling strategy, the core set S is initialized to be empty, with the parameter n = 3, the maximum number of data in the core set, and the Euclidean distance between each pair of A, B, C, D, and E is calculated.

[0105] 6. The flow rate monitoring module continuously monitors the data flow rate. If 5 data are detected, the flow rate monitoring module increases the amount of data flowing in this interval by 5.

[0106] 7. The data sampling module adds A to S, calculates the sum d1 of the closest distances from B, C, D, and E to the elements in S, and removes A from S; adds B to S, calculates the sum d2 of the closest distances from A, C, D, and E to the elements in S, and removes B from S; adds C to S, calculates the sum d3 of the closest distances from A, B, D, and E to the elements in S, and removes C from S; adds D to S, calculates the sum d4 of the closest distances from A, B, C, and E to the elements in S, and removes D from S; adds E to S, calculates the sum d5 of the closest distances from A, B, C, and D to the elements in S, and removes E from S; after comparing d1 to d5, it is found that d5 is the smallest, so E is added to S;

[0107] 8. The data sampling module repeats step 7 and eventually adds E and D to S. When the amount of data in S reaches the maximum amount of data determined by the parameters, the sampling is completed and the elements in S are output to the downstream streaming machine learning system for model training of the machine learning task.

[0108] 9. The downstream machine learning task reads data E and D for model training, and the flow rate monitoring module increases the amount of model training data in this interval by 2;

[0109] 10. The 1-second flow rate reporting interval of the flow rate monitoring module expires. The flow rate monitoring module calculates that the data source flow rate is 5 messages per second and the model training throughput is 2 messages per second. The flow rate monitoring module reports the data source flow rate and model training throughput to the sampling control module and clears the local statistical data.

[0110] 11. The sampling control module receives the data source flow rate and the model training throughput rate, calculates the new sampling rate as 0.38, and sends the new data sampling rate to the data sampling module;

[0111] 12. The parameter optimization module receives new sampling rate, model throughput, and other data, maintains the data sampling strategy as core set-based sampling, and adjusts the maximum data size n to 2;

[0112] 13. The data sampling module receives the new sampling rate and data sampling strategy;

[0113] 14. While the streaming machine learning task is running, continue to execute steps 5 to 13 to dynamically set the sampling rate and optimize the parameters, thereby optimizing the throughput of the machine learning task.

[0114] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A throughput optimization method for big data streaming machine learning, comprising the following steps: 1) Start the data sampling module and flow rate monitoring module on the computing node, and start the parameter optimization module and sampling control module on the management node; 2) The sampling control module determines the initial sampling rate and sets the flow rate reporting time interval; 3) The parameter optimization module obtains the data type of the streaming data provided by the data source and selects a data sampling strategy from the data sampling strategy library; then, the module determines the initial parameters of the data sampling strategy based on the initial sampling rate, and sends the data sampling strategy and its parameters to the data sampling module; 4) The sampling control module notifies the streaming machine learning system on each computing node to start executing the streaming machine learning task; 5) The data sampling module samples each batch of streaming data generated by the currently received data source according to the current data sampling strategy and its parameters, and sends the sampling results to the streaming machine learning system for model training of the streaming machine learning task; 6) After the streaming machine learning task begins execution, the flow rate monitoring module continuously monitors the flow rate of the data source and the model training throughput rate; then, according to the set flow rate reporting time interval, the flow rate and the model training throughput rate are sent to the sampling control module and the model training throughput rate is sent to the parameter optimization module; 7) The sampling control module calculates a new sampling rate based on the flow rate of the data source and the model training throughput and sends it to the parameter optimization module; 8) The parameter optimization module optimizes the parameter settings of the current data sampling strategy or selects a new data sampling strategy based on the new sampling rate and model training throughput, and sends the processing results to the data sampling module; 9) The data sampling module samples each batch of streaming data generated by the currently received data source according to the processing result of step 8); 10) While the streaming machine learning task is running, continue to execute steps 5) to 9) to optimize the model training throughput.

2. The method according to claim 1, characterized in that The sampling control module obtains the model training throughput rate T and the data source flow rate S, calculates a new sampling rate r = T / S*alpha; then sends the new sampling rate r to the data sampling module, and the data sampling module samples each batch of streaming data generated by the currently received data source according to the new sampling rate r; Among them, alpha is used to match the input and output flow rates.

3. The method according to claim 2, characterized in that Select alpha=0.

95.

4. The method according to claim 1, 2 or 3, characterized in that: The data sampling strategy library includes a random sampling strategy, and its sampling method is: for each data sample in the streaming data, generate a random number n uniformly distributed between [0,1]; if the random number n is less than the set maximum sampling ratio e, then the data sample is selected; if the random number n is greater than or equal to the set maximum sampling ratio e, then the data sample is discarded.

5. The method according to claim 1, 2 or 3, characterized in that: The data sampling strategy library includes a distance-based core set sampling strategy, and its sampling method is: 3.1) Read a batch of data from the streaming data and initialize the core set S to be empty; 3.2) Calculate the Euclidean distance between each sample in the batch of data and store it as a matrix for subsequent access; 3.3) Traverse the batch of data one by one. For each data sample X, first add it to the core set S. Then calculate the sum d of the closest distances from all samples in the batch outside the core set S to the samples in the core set S. Then remove the data sample X from the core set S. 3.4) Add the data sample corresponding to the smallest d that appears during the traversal process to the core set S; 3.5) Repeat steps 3.3) and 3.4) until the amount of data in the core set S reaches the set maximum data amount n.

6. The method according to claim 1, characterized in that The sampling strategies in the data sampling strategy library are expanded by generating new sampling strategies, including sampling logic and parameters, sampling mode, and acceptable data types; wherein the sampling logic is the sampling strategy specifically applied to the data, the parameters are the specific control parameters in the sampling logic, and the sampling mode includes streaming and micro-batch.

7. A throughput optimization system for big data streaming machine learning, characterized by: It includes a data sampling strategy library, a flow rate monitoring module, and a data sampling module deployed on the computing nodes, as well as a parameter optimization module and a sampling control module deployed on the management nodes; Data sampling strategy library, used to store data sampling strategies; A parameter optimization module is used to obtain the data type of streaming data provided by the data source and select a data sampling strategy; Then, the initial parameters of the data sampling strategy are determined according to the initial sampling rate, and the data sampling strategy and its parameters are sent to the data sampling module; and according to the new sampling rate and model training throughput rate, the parameter settings of the current data sampling strategy are optimized or a new data sampling strategy is selected and the processing results are sent to the data sampling module; The sampling control module is used to determine the initial sampling rate and set the flow rate reporting interval, notify the streaming machine learning system on each computing node to start executing the streaming machine learning task, and calculate the new sampling rate based on the flow rate of the data source and the model training throughput rate and send it to the parameter optimization module; The data sampling module is used to sample each batch of streaming data generated by the currently received data source according to the current data sampling strategy and its parameters, and send the sampling results to the streaming machine learning system for model training of the streaming machine learning task; The flow rate monitoring module is used to continuously monitor the flow rate of the data source and the model training throughput after the streaming machine learning task starts to execute; then, according to the set flow rate reporting time interval, the flow rate and the model training throughput are sent to the sampling control module, and the model training throughput is sent to the parameter optimization module.

Images