A data processing system and method based on divisible load and reinforcement learning

Abstract

The present invention provides a data processing system and method based on divisible load and reinforcement learning, which relates to the technical field of data processing. The method specifically includes the following steps: using sensors to collect data, and the client receives the data collected by the sensors; the client sends test data to each server for detection and processes the data; dividing the amount of data processed by each server according to the data distribution rate, data processing rate, and data return rate; based on the real-time data distribution rate, data processing rate, and data return rate, deciding whether to re-plan and schedule the amount of data processed by each server based on the reinforcement learning Q-Learning algorithm. If re-scheduling is required, re-divide the data to be processed. The technical solution of the present invention overcomes the problem in the prior art that data cannot be processed efficiently and flexibly.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly relates to a data processing system and method based on divisible load and reinforcement learning. Background Art

[0002] Data preprocessing is an important step before analyzing and calculating data. Performing data preprocessing can improve the accuracy of the model, making the machine learning process more efficient and the results more accurate.

[0003] Existing data preprocessing technologies for sensor behavior recognition data process data through a local processor or a single remote server. For example, for the classification problem after converting time series data into images, in the preprocessing stage, it is necessary to use computing resources to check and discard missing values, then perform normalization processing, and finally use the sliding window technique to segment the data, and use the Gramian angular field to convert each group of time series data into an image.

[0004] The data preprocessing methods in the prior art have the following problems: First, in the case of a large amount of data to be processed, the computing power of a single computing resource is limited, so it is difficult to perform complex and large-scale data processing. Limited by the computing power, data processing may be difficult to complete within an acceptable time. Second, when multiple computing resources cooperate to process data, the resource utilization rate is low. Computing resources are usually applied for and used by each user independently without unified scheduling. When the computing power of one computing node is saturated, another computing node is likely to be idle, resulting in the underutilization of the performance of computing nodes in many cases and causing resource waste, which will affect the execution efficiency of tasks. Finally, the divisible load method in the prior art assumes that the transmission and computing rates of each server are the same as those in the initial test stage and will not change, but in actual situations, the transmission rate and data processing rate between the client and the server will change due to interference. These problems restrict the efficiency and flexibility of data preprocessing methods.

[0005] Therefore, there is a need for an efficient and flexible data processing system and method based on divisible load and reinforcement learning. Summary of the Invention

[0006] The main object of the present invention is to provide a data processing system and method based on divisible load and reinforcement learning to solve the problem that data cannot be processed efficiently and flexibly in the prior art.

[0007] To achieve the above object, the present invention provides a data processing system based on divisible load and reinforcement learning, including: a client and a server, and the client and the server transmit data to each other.

[0008] The present invention also provides a data processing method based on divisible load and reinforcement learning, which specifically includes the following steps:

[0009] S1. Use sensors to collect data, and the client receives the data collected by the sensors.

[0010] S2. The client sends a small piece of test data to each server for detection. After each server receives the test data to be processed, it processes the data, including: calculating the data distribution rate, data processing rate, and data feedback rate.

[0011] S3. Divide the data processing volume of each server according to the data distribution rate, data processing rate, and data feedback rate. Number the servers in descending order according to the data distribution rate, and the data sending order is in the order of the server numbers.

[0012] S4. The client distributes data to each server according to the data volume divided in step S3. The server processes the distributed data and calculates the latest data distribution rate, data processing rate, and data feedback rate.

[0013] S5. Based on the real-time data distribution rate, data processing rate, and data feedback rate, decide whether to re-plan and schedule the data processing volume of each server based on the Q-Learning algorithm of reinforcement learning. If re-scheduling is required, re-divide the data to be processed.

[0014] Further, step S2 specifically includes the following steps:

[0015] S2.1. Calculate the data distribution rate of the th server :

[0016] (1);

[0017] Wherein, is the data processing volume size of the th server, is the time-consuming of the data distribution phase.

[0018] S2.2. Calculate the data processing rate of the th server :

[0019] (2);

[0020] Wherein, is the time-consuming of the data processing phase.

[0021] S2.3. Calculate the Data transmission rate of a server :

[0022] (3);

[0023] Among them, is the time consumed in the data recovery stage.

[0024] Furthermore, step S3 specifically includes the following steps:

[0025] S3.1, the total amount of data to be processed by the client is , the client sequentially recovers the processing results of each server and fills up the receiving channel bandwidth of the client, then there is:

[0026] (4);

[0027] (5).

[0028] S3.2, substituting formula (5) into formula (4) gives:

[0029] (6).

[0030] After sorting out:

[0031] (7);

[0032] Denote , then:

[0033] (8);

[0034] It can be obtained that: , , and so on, then there is:

[0035] (9);

[0036] Denote , then:

[0037] (10).

[0038] Rearranging formula (4) again, we get:

[0039] (11).

[0040] Furthermore, step S4 is specifically:

[0041] The client transfers data to each server in chunks. The data load of the server consists of multiple data blocks. While each server is receiving data, it calls the second basic service module of the server to process each block of data. When the data processing situation reporting module of the server starts to process data, it monitors the data processing situation of the server and reports relevant information to the client. After the client obtains the relevant information, it displays the data distribution rate, data processing rate, and data return rate to the user on the interface of the client; among them, the relevant information includes: the transmission time of a certain data block, the processing time, and the time taken for the corresponding result to be recycled.

[0042] Furthermore, step S5 specifically includes the following steps:

[0043] S5.1, Specify actions , represents rescheduling, represents no rescheduling; specify status ; among them, is the real-time data processing speed, is the actual data return speed, is the expected data processing time, is the expected data return time.

[0044] S5.2, Define the latency of the data processing system ; Reward function .

[0045] S5.3, Use the gradient descent method to update the neural network weights:

[0046] (12);

[0047] (13);

[0048] Among them, represents using the neural network to execute the function of the Q-table, where is the model weight of the neural network, is the target Q value, the input of the neural network model is the state , is the discount factor; is the executed action and the new state observed after that; is the next action selected in the new state, is the neural network learning rate.

[0049] S5.4. Determine whether to re-plan and schedule the amount of data processed by each server based on the real-time data distribution rate, data processing rate, and data feedback rate using the Q-Learning algorithm of reinforcement learning. If re-scheduling is required, re-partition the data to be processed. When re-scheduling, the total amount of data remains unchanged, and the constraints are as follows:

[0050] (14);

[0051] (15);

[0052] Among them, is the total amount of data allocated to server after re-scheduling, is the time taken for the data processing stage of server after re-scheduling, is the time taken for the data recovery stage of server after re-scheduling.

[0053] S5.5. Combine formulas (11), (12), and (13), and substitute formula (15) into formula (14) to obtain:

[0054] (16);

[0055] Among them, is the amount of data that has been processed by server at the start of scheduling.

[0056] After arrangement:

[0057] (17);

[0058] (18);

[0059] Denote: , , then:

[0060] (19).

[0061] It can be obtained that: , , and so on, then there is:

[0062] (20);

[0063] (21);

[0064] Denote, , , then there is:

[0065] (22);

[0066] (23).

[0067] The present invention has the following beneficial effects:

[0068] According to the method provided by the present invention, the user only needs to simply adjust the input parameters of the existing application program, and sometimes even does not need to make any modification. While improving the data processing efficiency, the application development difficulty is greatly reduced.

[0069] The method provided by the present invention overcomes the problems that the ordinary scheduling algorithm has inflexible one-time allocation and does not consider the changes in the performance of the client and the server with time and the task volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. In the drawings:

[0071] Figure 1 Shows a flowchart of a data processing method based on divisible load and reinforcement learning.

[0072] Figure 2 Shows the topological structure diagram of the scheduling node of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0073] The following will clearly and completely describe the technical solutions of the present invention with reference to the drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0074] A data processing system based on divisible load and reinforcement learning includes: a client and a server, and the client and the server transmit data to each other.

[0075] The client is responsible for interacting with the user, accepting the tasks initiated by the user and sending requests to the server. The client includes:

[0076] Interface: Responsible for interacting with the user. The user submits the behavior recognition data obtained through the sensor through the interface, sends a data processing task to the server, and shows the total execution progress of the task to the user.

[0077] The first basic service module: provides some general services, such as basic processing services for byte arrays (byte array copying, truncating, searching for sub-byte arrays), Socket communication services, serialization and deserialization services, etc.

[0078] The real-time storage module for statistical information: real-time statistics on the status of some server-side data processing, such as the transfer time of a certain data block, the processing time, and the time taken for the corresponding result to be recycled, and stores this data in the database for viewing.

[0079] The data sending module: responsible for submitting the Gram angle quantization algorithm program file, the behavior recognition data captured by the sensor, the relevant parameters of the program file, etc. to the server.

[0080] The result recycling module: responsible for recycling the processing results to the client.

[0081] The data processing monitoring module: when the server starts to process data, it is responsible for monitoring the data processing situation of the server and reporting relevant information to the client.

[0082] The load data splitting module: based on the historical information of server-side data processing, splits the data according to the scheduling strategy to allocate to different server nodes.

[0083] The server is responsible for accepting tasks initiated by the client and performing data processing and feedback, including the following modules.

[0084] The data receiving module: responsible for receiving from the client the corresponding time-series sample data obtained by the sensor through the accelerometer, gyroscope, magnetometer, etc. and storing it.

[0085] The result sending module: when the client initiates a result recycling request, it is responsible for returning the processed result to the client.

[0086] The data processing reporting module: responsible for collecting in real-time the transfer time of the data block and the processing time of the data block and reporting it to the client in a timely manner.

[0087] The task scheduling module: responsible for scheduling the execution of the application submitted by the client, interacting with it to input appropriate parameters and data, and recording its execution situation.

[0088] The second basic service module: responsible for processing the received sensor data, converting these data into Gram angle quantization two-dimensional images through algorithms, and providing Socket communication services, serialization and deserialization services.

[0089] The communication between the client and the server is implemented as Socket communication. In order to allow the server to identify different requests from the client, a custom TCP message is used. The basic structure of the message body is shown in Table 1. The message start flag is "START" and the end flag is "FINISHED", and then different types of messages are customized based on this.

[0090] The monitoring request message is shown in Table 1:

[0091] Table 1 Monitoring request message structure

[0092]

[0093] Purpose: The client initiates a request to monitor the data processing status of the server. After the server recognizes the request, it starts the data processing status reporting thread to report the data processing status of the server to the client in real time.

[0094] The result recovery request message is shown in Table 2:

[0095] Table 2 Result recovery request message structure

[0096]

[0097] Purpose: The client initiates a result recovery request. After the server recognizes the request, it transmits the file specified in the request back to the client.

[0098] The server data processing information report response message is shown in Table 3:

[0099] Table 3 Server data processing information report response message structure

[0100]

[0101] Purpose: After the server processes a small piece of data, it encapsulates the processing information of this small piece of data into a DealInfo instance, serializes it into bytecode, and transmits it back to the client.

[0102] The data / file sending request message to be processed is shown in Table 4:

[0103] Table 4: Data / file sending request message structure to be processed

[0104]

[0105] Purpose: It is a message when the client sends data to be processed to the server. This request specifies which task will process the data through the task code, whether the data transmitted this time is a series of files through "DATA / FILES", whether the data this time is test data through "USEFUL / TEST", and during the transmission process, it can be known which block of data has been transmitted through the "data block number" and "UNITEND".

[0106] The file transfer request message is shown in Table 5:

[0107] Table 5 File transfer request message structure

[0108]

[0109] Purpose: When the client initiates this request, it transmits a file to the server. The types of files transmitted are "PROGRAM", "PARAM", "OTHER", and the file name can be specified.

[0110] The result recovery response message is shown in Table 6:

[0111] Table 6 Result recovery response message structure

[0112]

[0113] Purpose: When the server receives the result request message, it sends this message to the client to transmit the processing result.

[0114] The present invention also provides a data processing method based on divisible load and reinforcement learning, specifically including the following steps:

[0115] S1. Use sensors to collect data, and the client receives the data collected by the sensors. For example, in the machine learning training of fall detection, first, the sensors need to obtain corresponding time-series sample data through accelerometers, gyroscopes, magnetometers, etc. The time-series sample data needs to be converted into a Gram angle two-dimensional image through a conversion algorithm, so as to use the image for learning to complete the recognition of behaviors. Specifically, this conversion algorithm first reads the source file, performs the first processing on the values, interpolates to remove NaN values, then synthesizes the accelerations of the three axes into the acceleration magnitude, then uses a sliding window to upsample the processed data, converts the data of each window into a color image using the Gram angle field, and finally stitches the images of different sensors as the final result image.

[0116] S2. The client sends test data to each server for detection. After each server receives the test data to be processed by itself, it processes the data, including: calculating the data distribution rate, data processing rate, and data feedback rate.

[0117] S3. Divide the data processing amounts of each server according to the data distribution rate, data processing rate, and data return rate. Number the servers in descending order according to the data distribution rate, and the data sending order is carried out in sequence according to the server number order.

[0118] S4. The client distributes data to each server pair according to the data amounts divided in step S3. The server pair processes the distributed data and calculates the latest data distribution rate, data processing rate, and data return rate.

[0119] S5. According to the real-time data distribution rate, data processing rate, and data return rate, based on the reinforcement learning Q-Learning algorithm, decide whether to re-plan and schedule the data processing amounts of each server. If re-scheduling is required, re-divide the data to be processed.

[0120] Such as Figure 1 shown are the various stages of the data processing method provided by the present invention based on divisible load and reinforcement learning. First is the prediction stage (i.e., step S2 where the client sends test data to each server for detection). The client distributes small pieces of data to each server for processing and recovers the results, measures the data distribution and recovery delays from the client to the server, and the data processing speed of the server. Subsequently, it enters the first scheduling stage. According to the client performance data obtained from the previous measurement, calculate the server load distribution according to the divisible load theory (i.e., steps S2 and S3), and start distributing data. The period from the end of the prediction stage to the completion of the calculation is the dynamic scheduling stage. Execute tasks according to the load distribution calculated in the first scheduling stage, and continuously count the server performance. Whenever the data processing speed and data recovery speed are not as expected, Q-Learning decides whether to re-schedule. If re-scheduling is required, start dynamic scheduling and execute step S5.4 to determine the new load of each server to achieve the purpose of minimizing the overall delay. The end moment of the dynamic scheduling stage is the completion moment of the calculation. Figure 2 in 、 、……、 are each server, is the time taken for the data distribution stage of the nth server, is the time taken for the data processing stage of the nth server, is the time taken for the data recovery stage of the nth server.

[0121] Figure 2 is the scheduling topology structure of the method provided by the present invention. The client adaptively divides the data of the divisible tasks in the application field according to the divisible load theory, sends the processing program and data to each server on the cloud platform, and realizes the automatic parallelism of the divisible load.

[0122] Specifically, step S2 specifically includes the following steps:

[0123] S2.1, calculate the data distribution rate of the th server: :

[0124] (1);

[0125] Wherein, is the amount of data processed by the th server, is the time consumed in the data distribution phase.

[0126] It is expressed by the time used for unit data.

[0127] S2.2, calculate the data processing rate of the th server: :

[0128] (2);

[0129] Wherein, is the time consumed in the data processing phase. It is expressed by the time used for unit data.

[0130] S2.3, calculate the data feedback rate of the th server: :

[0131] (3);

[0132] Wherein, is the time consumed in the data recovery phase. It is expressed by the time used for unit data.

[0133] Specifically, step S3 specifically includes the following steps:

[0134] S3.1, the total amount of data to be processed by the client is , the client sequentially recovers the processing results of each server and fills the receiving channel bandwidth of the client, then there is:

[0135] (4);

[0136] (5).

[0137] S3.2, combine formulas (1), (2), and (3), and substitute formula (5) into formula (4) to obtain:

[0138] (6).

[0139] After rearrangement, we get:

[0140] (7);

[0141] Denote , then:

[0142] (8);

[0143] We can obtain: , , and so on, then we have:

[0144] (9);

[0145] Denote , then:

[0146] (10);

[0147] Rearranging formula (4) again, we get:

[0148] (11).

[0149] Specifically, step S4 is specifically as follows:

[0150] The client transfers data to each server in blocks. The data load of the server consists of multiple data blocks. After each server receives the data, it separately calls the second basic service module of the server to process each block of data. When the data processing situation reporting module of the server starts to process the data, it monitors the data processing situation of the server and reports the relevant information to the client. After the client obtains the relevant information, it displays the data distribution rate, data processing rate, and data return rate to the user on the interface of the client; among them, the relevant information includes: the transmission time of a certain data block, the processing time, and the time taken for the corresponding result to be recycled.

[0151] To ensure that when the transmission rate and data processing rate between the client and the server change due to interference, the receiving channel bandwidth of the client can still be fully utilized to process and transmit data as soon as possible, it is necessary to decide whether to reschedule when the rate is inconsistent with the expectation. If the Q-Learning inference believes that the delay is smaller after rescheduling, it will reschedule; otherwise, it will not reschedule.

[0152] Next, the reward function in Q-Learning, etc. is defined to encourage the decision to prefer behaviors with higher data processing and return rates.

[0153] Specifically, step S5 specifically includes the following steps:

[0154] S5.1, Prescribed actions , represents rescheduling, represents no rescheduling; Prescribed status ; where, is the real-time data processing speed, is the actual data transmission speed, is the expected data processing time, is the expected data transmission time.

[0155] S5.2, Define the latency of the data processing system ; Reward function . Define the reward function , so that the data processing speed and data transmission speed during the execution process are faster than the expected speed. In this way, the Q-table tends to select actions that can minimize the system latency as much as possible, and can reschedule when some server devices fail or their performance unexpectedly degrades, and does not reschedule when the task execution goes smoothly.

[0156] S5.3, Use the gradient descent method to update the neural network weights:

[0157] (12);

[0158] (13);

[0159] where, represents using the neural network to execute the function of the Q-table, where are the model weights of the neural network, and the input of the neural network model is the state , is the discount factor, indicating the importance of future rewards; is the action taken and the new state observed after that; is the next action selected in the new state; the output is the probability of action a.

[0160] The Q-Learning algorithm is divided into two processes: inference and training.

[0161] In the inference process, a Q-table is constructed with actions as the horizontal axis and states as the vertical axis to store Q-values, and the action with the largest Q-value is selected based on the current state , and the action at the th step depends on the current state of the system.

[0162] During the training process, the Q-Learning algorithm first initializes the Q-table, and then continuously selects an action based on the current state to obtain a new state. In this process, the Q-learning algorithm updates the Q-value based on the Bellman equation, and defines ;

[0163] ;

[0164] Since there are continuous parts in the values of the state space, a neural network is trained to replace , and the current state is used as the input, and the neural network gives the Q value of each action in the corresponding state. Therefore, in each step of training, the formula: is no longer used to update the Q table. Instead, the gradient descent method in S5.3 is used to update the weights of the neural network.

[0165] S5.4. According to the real-time data distribution rate, data processing rate, and data feedback rate, based on the reinforcement learning (Q-Learning) algorithm, it is decided whether to re-plan and schedule the data processing volume of each server. If re-scheduled, the data to be processed is re-partitioned. Since the data processing time is much longer than the data transmission time, the first scheduling occurs after all the client data has been uploaded. Each server device in the same cloud environment ignores the data transmission time between each other and only considers the time for the result to be sent back from the server to the client. Therefore, when re-scheduling, the total data volume remains unchanged, and the constraint conditions are as follows:

[0166] (14);

[0167] (15);

[0168] Among them, is the total data volume allocated to server after re-scheduling, is the data processing stage time consumption of server after re-scheduling, is the data recovery stage time consumption of server after re-scheduling.

[0169] S5.5. Combining formulas (11), (12), and (13), substituting formula (15) into formula (14) gives:

[0170] (16);

[0171] Among them, is the data volume that has been processed by server at the start of scheduling.

[0172] After arrangement:

[0173] (17);

[0174] (18).

[0175] Note: , , then:

[0176] (19);

[0177] It can be obtained that: , , and so on, then there is:

[0178] (20);

[0179] (21);

[0180] Note, , , then there is:

[0181] (22);

[0182] (23).

[0183] First, obtain , and then sequentially obtain , redistribute data among servers, and complete re - scheduling.

[0184] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions, or substitutions made by those skilled in the art within the scope of the essence of the present invention should also fall within the protection scope of the present invention.

Claims

1. A data processing method based on divisible load and reinforcement learning, applied to a data processing system based on divisible load and reinforcement learning, comprising: Client and server, the client and server transmit data to each other; The data processing method specifically comprises the following steps: S1, using sensors to collect data, and the client receives the data collected by the sensors; S2, the client sends test data to each server for detection. After receiving the test data to be processed, each server processes the data, including: calculating the data distribution rate, data processing rate and data return rate; S3, dividing the amount of data processed by each server according to the data distribution rate, data processing rate and data return rate, numbering the servers in order from fast to slow according to the data distribution rate, and sending data in the order of the server numbers; S4, the client distributes data to each server pair according to the data volume divided in step S3, and the server processes the distributed data and calculates the latest data distribution rate, data processing rate and data return rate; S5, based on the real-time data distribution rate, data processing rate and data return rate, decides whether to re-plan and schedule the processing data volume of each server based on the reinforcement learning Q-Learning algorithm. If re-scheduling is required, the data to be processed is re-divided; Step S2 specifically includes the following steps: S2.1, calculate the The data distribution rate of the server : （1）； in, For the The amount of data processed by the server. The data distribution phase is time-consuming; S2.2, calculate the The data processing rate of the server : （2）； in, The data processing stage is time-consuming; S2.3, calculate the The data return rate of the server : （3）； in, The data recovery phase takes time; Step S3 specifically includes the following steps: S3.1, the total amount of data to be processed on the client is , the client recycles the processing results of each server in turn and occupies the client's receiving channel bandwidth, then: （4）； （5）； S3.2, combining formula (1), (2), (3), and substituting formula (5) into formula (4), we obtain: （6）； Arranged: （7）； remember ,but: （8）； We can get: , , and so on, we have: （9）； remember ,but: （10）； Rearranging formula (4), we obtain: （11）； Step S4 is specifically as follows: The client transmits data to each server in blocks. The data load of the server is composed of multiple data blocks. After receiving the data, each server calls the second basic service module of the server to process each block of data. When the data processing status reporting module of the server starts to process the data, it monitors the data processing status of the server and reports the relevant information to the client. After the client obtains the relevant information, it displays the data distribution rate, data processing rate and data return rate to the user on the client interface; the relevant information includes: the transmission time of a data block, the processing time, and the time taken for the corresponding result to be recovered.

2. According to claim 1, a data processing method based on divisible load and reinforcement learning is characterized in that: Step S5 specifically includes the following steps: S5.1, Prescribed Actions , Reschedules. Indicates no rescheduling; specifies the state ;in, For real-time data processing speed, is the actual data transmission speed, is the expected data processing time, is the expected data return time; S5.2, Defining the latency of a data processing system ; Reward function ; S5.3, using the gradient descent method, update the neural network weights: （12）； （13）； in, represents the function of using a neural network to execute the Q table, where is the model weight of the neural network, is the target Q value, and the input of the neural network model is the state , is the discount factor; Is to perform an action The new state observed later; is the next action chosen in the new state, is the neural network learning rate; S5.4, based on the real-time data distribution rate, data processing rate and data return rate, decide whether to re-plan and schedule the processing data volume of each server based on the reinforcement learning Q-Learning algorithm. If re-scheduling is required, the data to be processed is re-divided; when re-scheduling, the total amount of data remains unchanged, and the constraints are as follows: （14）； （15）； in, Is the server after rescheduling The total amount of data allocated, For the rescheduled server The data processing phase is time-consuming. For the rescheduled server The data recovery phase is time-consuming; S5.5, combining formula (11), (12), (13), substituting formula (15) into formula (14) yields: （16）； in, The server at the start of scheduling The amount of data that has been processed; Arranged: （17）； （18）； remember: , ,but: （19）； We can get: , , and so on, we have: （20）； （21）； remember, , , then: （22）； （23）。

Images