Education big data super-concurrent processing system and method

Abstract

The invention relates to the field of big data super-concurrent processing, in particular to an education big data super-concurrent processing system and method. The method comprises the following steps: collecting and preprocessing original request data to obtain actual request traffic; using historical data in actual request traffic to train a weighted autoregression model, and predicting the traffic; dynamically adjusting the concurrency based on the obtained flow prediction value, calculating the scheduling priority of the data packets, and sorting and distributing the data packets; a dynamic back pressure control mechanism based on flow prediction is introduced, flow input is dynamically adjusted, and overload is prevented. The problems that a traditional scheduling algorithm and a traditional load control strategy lack enough accurate flow prediction, concurrency cannot be adjusted in time, and system resources are excessively distributed or insufficient are solved; the priority processing of the key task cannot be ensured because the flow fluctuation and the scheduling priority are not fully considered; the problems that flexibility and self-adaptive capacity are lacked, the flow cannot be effectively limited when the load is too high, and the overload risk of the system is increased are solved.

Description

Technical Field

[0001] The present invention relates to the field of large data super-concurrency processing, and in particular to a system and method for super-concurrency processing of educational large data. Background Art

[0002] With the continuous development of big data technology and artificial intelligence, education platforms are gradually adopting more accurate and intelligent solutions in traffic prediction and resource scheduling. In recent years, traffic prediction methods based on time series analysis and machine learning models have made significant progress, which can more effectively capture trends, seasonal fluctuations and sudden changes in the request traffic of education platforms. In addition, the development of intelligent scheduling algorithms and dynamic resource management technologies enables education platforms to adjust concurrency and task priorities in real time according to traffic predictions, thereby optimizing resource allocation and improving system performance. In order to cope with extremely high concurrency and complex load scenarios, the back pressure control mechanism is also being continuously optimized, and the stability and responsiveness of the education platform are further improved by dynamically adjusting the traffic input. The development of the above technologies not only enhances the prediction accuracy of the education platform, but also significantly improves the adaptability and operation efficiency in ultra-high concurrency situations.

[0003] However, the above scheduling algorithms and load control strategies have the following problems: First, the lack of sufficiently accurate traffic prediction makes it impossible to adjust the concurrency in time, which easily leads to over-allocation or under-allocation of system resources; second, the failure to fully consider traffic fluctuations and scheduling priorities makes it impossible to ensure the priority processing of critical tasks; in addition, the lack of flexibility and adaptability makes it impossible to effectively limit traffic when the load is too high, thereby increasing the risk of system overload, and it is urgently needed to be improved through a new algorithm and processing technology. Summary of the invention

[0004] The present invention provides a system and method for super-concurrency processing of educational big data, which solves the problems that traditional scheduling algorithms and load control strategies lack sufficiently accurate traffic prediction, resulting in the inability to adjust the concurrency in time, causing excessive or insufficient allocation of system resources; fail to fully consider traffic fluctuations and scheduling priorities, and cannot ensure priority processing of key tasks; lack flexibility and adaptability, and cannot effectively limit traffic when the load is too high, thereby increasing the risk of system overload.

[0005] The present invention provides a system and method for processing educational big data in a super-concurrent manner, which specifically includes the following technical solutions:

[0006] A method for super-concurrency processing of educational big data, comprising the following steps:

[0007] S1: Collect and preprocess the original request data of the education platform to obtain the actual request traffic; use the historical data in the actual request traffic to train the weighted autoregressive model; use the weighted autoregressive model to predict future traffic and obtain the traffic prediction value;

[0008] S2: Based on the traffic prediction value, dynamically adjust the concurrency to obtain the adjusted concurrency; based on the adjusted concurrency, calculate the scheduling priority of the data packet; based on the scheduling priority of the data packet, sort and allocate the data packets; introduce a dynamic back pressure control mechanism based on traffic prediction, dynamically adjust the traffic input, control the reception of data packets, and prevent overload.

[0009] Preferably, the S1 specifically includes:

[0010] The weighted autoregressive model introduces an exponential decay factor and a dynamic nonlinear adjustment factor through a weighted autoregressive formula to adjust the impact of historical data on future traffic prediction and obtain a traffic prediction value; the weighted autoregressive formula is as follows:

[0011]

[0012] in, is the traffic prediction value, which indicates the predicted request traffic at time point t+1; t is the index variable at the time point; p is the maximum lag period of the historical data used by the weighted autoregressive model when making predictions, indicating the amount of historical data used; is the index variable of the lag period; α is the regression weight coefficient; e is a natural constant; λ is the attenuation factor; is the exponential decay factor; It's time point is the actual request flow; β is the dynamic nonlinear adjustment coefficient; μ is the mean of the historical actual request flow; γ is the nonlinear adjustment factor; is the dynamic nonlinear adjustment factor.

[0013] Preferably, the S1 specifically includes:

[0014] The weighted error loss function is introduced and combined with the gradient descent method to iteratively optimize the parameters of the weighted autoregressive model.

[0015] Preferably, the S2 specifically includes:

[0016] Based on the traffic prediction value, a concurrency adjustment formula is defined, exponential smoothing adjustment items and adaptive smoothing items are introduced to adjust the concurrent resources and obtain the adjusted concurrency.

[0017] Preferably, the S2 specifically includes:

[0018] Based on the adjusted concurrency, a dynamic scheduling priority adjustment mechanism is introduced. The scheduling priority of the data packet is obtained by combining the processing time of the data packet and the task priority. The specific calculation formula of the scheduling priority of the data packet is:

[0019]

[0020] Among them, S i is the scheduling priority of data packet i; i is the index variable of the data packet; P i is the task priority of data packet i; is the processing time of data packet i; κ is the concurrency impact coefficient; η is the nonlinear adjustment coefficient; is a concurrent load balancing benchmark; C t is the concurrency at time point t, indicating the adjusted concurrency.

[0021] Preferably, the S2 specifically includes:

[0022] Monitor the current load situation in real time to obtain the real-time load volume; introduce a dynamic back pressure control mechanism based on flow prediction, combine the real-time load volume, flow prediction value and historical data, and dynamically adjust the flow input.

[0023] Preferably, the S2 specifically includes:

[0024] In the process of implementing the dynamic back pressure control mechanism based on flow prediction, the flow prediction value and load ratio are introduced to dynamically adjust the back pressure threshold and control the flow input; the back pressure threshold calculation formula is as follows:

[0025]

[0026] Where F is the back pressure threshold; L max is the maximum load; L t is the real-time load at time point t; is the load ratio; τ is the weight coefficient; is the coefficient that adjusts the effect of load changes; is the parameter of the exponential function; is the parameter used to control the exponential influence of the load ratio; δ is the feedback correction coefficient; L t-1 is the real-time load at time point t-1; is the traffic prediction value, which indicates the predicted request traffic at time point t+1; D t is the actual request flow at time point t.

[0027] An educational big data super-concurrency processing system includes the following parts:

[0028] Data collection and preprocessing module, traffic prediction model module, concurrency adjustment module, scheduling priority adjustment module, load assessment module, dynamic back pressure control module;

[0029] The data collection and preprocessing module collects and preprocesses the original request data of the education platform, obtains the actual request flow and outputs it; the data collection and preprocessing module is connected to the flow prediction model module by means of data transmission;

[0030] The traffic prediction model module receives the actual request traffic and uses the historical data in the actual request traffic to train the weighted autoregressive model; performs traffic prediction through the weighted autoregressive model to obtain and output the traffic prediction value; the traffic prediction model module is connected to the concurrency adjustment module and the dynamic back pressure control module through data transmission;

[0031] The concurrency adjustment module dynamically adjusts the concurrency based on the traffic prediction value, obtains the adjusted concurrency and outputs it; the concurrency adjustment module is connected to the scheduling priority adjustment module by means of data transmission;

[0032] The scheduling priority adjustment module dynamically adjusts the scheduling priority of the data packet based on the adjusted concurrency and outputs it; the scheduling priority adjustment module is connected to the load evaluation module by means of data transmission;

[0033] The load evaluation module sorts and distributes data packets based on the scheduling priority of the data packets; monitors the current load situation in real time, obtains the real-time load amount and transmits it; the load evaluation module is connected to the dynamic back pressure control module by means of data transmission;

[0034] The dynamic back pressure control module receives the real-time load and traffic prediction value, introduces a dynamic back pressure control mechanism based on traffic prediction, dynamically adjusts the traffic input, controls the reception of data packets, and prevents overload.

[0035] The beneficial effects of the technical solution of the present invention are:

[0036] 1. The present invention introduces a weighted autoregressive model and combines it with a weighted error loss function to effectively capture the trends, seasonal changes and short-term fluctuations in historical actual request traffic, thereby significantly improving the accuracy of traffic prediction. The weighted autoregressive model introduces dynamic weighting factors to make the impact of recent data on traffic prediction more significant, avoiding the excessive reliance of traditional autoregressive models on long-term data, thereby better coping with the volatility and suddenness of education platform traffic and enhancing the responsiveness and robustness of the weighted autoregressive model.

[0037] 2. Based on the traffic prediction value, the present invention can adjust the concurrency in real time to ensure the reasonable allocation of resources of the education platform during high concurrency periods, avoiding the risk of over-allocation or insufficient resources; at the same time, combined with the concurrency, a dynamic scheduling priority adjustment mechanism is introduced, which can give priority to key tasks (such as online exams, live interaction, etc.) according to the processing time of the data packet and the task priority, optimize resource allocation, and ensure the efficient operation of the education platform and the timely processing of tasks.

[0038] 3. The present invention introduces a dynamic back pressure control mechanism based on traffic prediction, which can flexibly adjust the traffic input according to the real-time load and traffic prediction value, ensuring that the traffic input is reasonably controlled when the real-time load is close to the processing upper limit, avoiding the crash of the education platform due to traffic surge, improving the load management capability of the education platform, and enhancing the adaptability and stability in ultra-high concurrency scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a structural diagram of an educational big data super-concurrent processing system according to the present invention;

[0040] Figure 2 This is a flow chart of a method for super-concurrent processing of educational big data described in the present invention. DETAILED DESCRIPTION

[0041] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0043] The specific scheme of the educational big data super-concurrent processing system and method provided by the present invention is described in detail below with reference to the accompanying drawings.

[0044] See attached Figure 1 , which shows a structural diagram of an educational big data super-concurrency processing system provided by an embodiment of the present invention, the system includes the following parts:

[0045] Data collection and preprocessing module, traffic prediction model module, concurrency adjustment module, scheduling priority adjustment module, load assessment module, dynamic back pressure control module;

[0046] The data collection and preprocessing module collects original request data from the log or database of the education platform and sorts the original request data in time series; preprocesses the original request data to obtain the actual request traffic and outputs it; the data collection and preprocessing module is connected to the traffic prediction model module by data transmission;

[0047] The traffic prediction model module receives the actual request traffic and uses the historical data in the actual request traffic to train the weighted autoregressive model; performs traffic prediction through the weighted autoregressive model to obtain and output the traffic prediction value; the traffic prediction model module is connected to the concurrency adjustment module and the dynamic back pressure control module through data transmission;

[0048] The concurrency adjustment module dynamically adjusts the concurrency based on the traffic prediction value, obtains the adjusted concurrency and outputs it; the concurrency adjustment module is connected to the scheduling priority adjustment module by means of data transmission;

[0049] The scheduling priority adjustment module dynamically adjusts the scheduling priority of the data packet based on the adjusted concurrency and outputs it; the scheduling priority adjustment module is connected to the load evaluation module by means of data transmission;

[0050] The load evaluation module sorts the data packets based on the scheduling priority. The data packets are the specific carriers of the tasks, and the tasks with high scheduling priority (such as online examinations, live broadcast interactions, etc.) are preferentially assigned to the computing nodes for processing; at the same time, the current load situation is monitored in real time to obtain the real-time load amount and transmit it; the load evaluation module is connected to the dynamic back pressure control module by means of data transmission;

[0051] The dynamic back pressure control module receives the real-time load and flow prediction value. When the real-time load is close to the upper limit of concurrent processing, it triggers the dynamic back pressure control mechanism based on flow prediction, dynamically adjusts the back pressure threshold, and implements flow control in advance when the real-time load is close to the processing upper limit to avoid overload.

[0052] See attached Figure 2 , which shows a flow chart of a method for super-concurrency processing of educational big data provided by an embodiment of the present invention, the method comprising the following steps:

[0053] S1: Collect and preprocess the original request data of the education platform to obtain the actual request traffic; use the historical data in the actual request traffic to train the weighted autoregressive model; use the weighted autoregressive model to predict future traffic and obtain the traffic prediction value;

[0054] The original request data is collected from the log or database of the education platform, and is statistically recorded every hour or every minute and expressed in the form of a time series. The time range of the original request data is set according to the needs of the education platform and traffic fluctuations.

[0055] Preprocessing the original request data, wherein the preprocessing process specifically includes removing outliers and missing data from the original request data and performing standardization processing to obtain actual request traffic; the actual request traffic is represented in the form of a time series for traffic prediction;

[0056] The weighted autoregressive model is trained using historical data from actual request traffic, and the weighted autoregressive model parameters are optimized using a weighted error loss function. The weighted autoregressive model is a prediction model based on time series, which is suitable for predicting traffic with periodic fluctuations such as education platforms, and can capture trends and seasonal changes in time series. Based on the traditional autoregressive model, the weighted autoregressive formula is used to adjust the impact of historical data on future traffic forecasts by introducing dynamic weighting factors, ensuring that the weight of recent data is larger and the impact of long-term data gradually decreases, thereby improving the accuracy of predictions for short-term fluctuations.

[0057] The weighted autoregressive formula predicts future traffic by introducing an exponential decay factor and a dynamic nonlinear adjustment factor to obtain a traffic forecast value, which not only improves the ability to respond to short-term fluctuations, but also enhances the prediction accuracy of abnormal traffic changes. It can significantly improve the prediction effect in a complex big data environment and improve the computational efficiency of the weighted autoregressive model. The weighted autoregressive formula is as follows:

[0058]

[0059] in, is the traffic prediction value, which indicates the predicted request traffic at the next time point t+1 predicted at time point t; t is the index variable of the time point; p is the maximum lag period of the historical data used by the weighted autoregressive model when making predictions, which indicates the amount of historical data used and is obtained through experimental tuning; is the index variable of the lag period; α is the regression weight coefficient, which is a parameter of the weighted autoregressive model and is used to indicate the weight of the impact of historical data on the current forecast; e is a natural constant; λ is the attenuation factor, which is a parameter of the weighted autoregressive model and is used to control the degree of impact of historical data on the current forecast. The larger the attenuation factor value, the smaller the impact of historical data. It is an exponential decay factor, which controls the contribution of historical data through decay, avoids over-reliance on long-term data, and ensures that the impact of long-term data on current traffic forecasts gradually decreases over time; It's time point The actual request flow; β is the dynamic nonlinear adjustment coefficient, which is a parameter of the weighted autoregressive model and is used to control the strength of the dynamic nonlinear adjustment factor; μ is the mean of the historical actual request flow, which is used as a comparison benchmark and is calculated using the historical data currently used for training; γ is the nonlinear adjustment factor, which is a parameter of the weighted autoregressive model and is used to control the nonlinear response degree that deviates from the mean and adjust the sensitivity to abnormal flow fluctuations. The larger the value of the nonlinear adjustment factor, the stronger the response to data that deviates from the mean; It is a dynamic nonlinear adjustment factor that enables the weighted autoregressive model to make greater adjustments when traffic fluctuations are large, thereby enhancing the ability to respond to sudden traffic fluctuations.

[0060] The regression weight coefficient, attenuation factor, dynamic nonlinear adjustment coefficient, and nonlinear adjustment factor are used as parameters of the weighted autoregressive model. They are initialized and set by expert experience before model training. During model training, the weighted error loss function is used in combination with the gradient descent method to iteratively optimize the parameters of the weighted autoregressive model to continuously improve the prediction accuracy. The gradient descent method is a prior art and will not be described in detail here.

[0061] The weighted error loss function measures the difference between the traffic prediction value and the actual request traffic by applying weighted and nonlinear adjustments to the prediction error, thereby guiding the update of the weighted autoregressive model parameters in the gradient descent optimization.

[0062] The weighted error loss function is defined as follows:

[0063]

[0064] Where L is the value of the loss function, which is used to measure the difference between the traffic prediction value and the actual request traffic. The smaller the value of the loss function, the more accurate the prediction. T is the total time steps of the training data, that is, the length of the historical data used for training. t = p means that the value of t starts from p, ensuring that the number of historical data is sufficient, that is, at least p historical data are needed for prediction. w t is a weighting factor, representing the prediction error at time point t The weight of the data is set according to the specific implementation scenario, such as the importance, periodicity, seasonality and other factors of the data; t+1 is the actual request flow at time point t+1; e -λ(t-p) is an exponential decay factor.

[0065] S2: Based on the traffic prediction value, dynamically adjust the concurrency to obtain the adjusted concurrency; based on the adjusted concurrency, calculate the scheduling priority of the data packet; based on the scheduling priority of the data packet, sort and allocate the data packets; introduce a dynamic back pressure control mechanism based on traffic prediction, dynamically adjust the traffic input, control the reception of data packets, and prevent overload.

[0066] Based on the traffic forecast value, possible traffic peaks and fluctuations can be identified in advance. Therefore, it is necessary to dynamically adjust the concurrency based on the traffic forecast value to ensure that resources can be effectively allocated when facing a large number of concurrent requests to avoid overload or insufficient resources. The concurrency refers to the maximum number of requests that can be processed at a specific time point. The adjustment of the concurrency helps to ensure the stability and performance of the education platform in high-concurrency scenarios and ensure the smoothness of the user experience.

[0067] The concurrency adjustment formula is as follows:

[0068]

[0069] Among them, C t is the concurrency at time point t, that is, the adjusted concurrency; min is the operation of taking the minimum value; C max is the maximum concurrency, that is, the upper limit of concurrent processing that can be carried; ζ is the concurrency adjustment coefficient, which is used to control the impact of traffic prediction value on concurrency (the unit is request traffic level), obtained through experimental tuning; is an exponential smoothing adjustment item, which prevents the concurrency adjustment from increasing or decreasing sharply due to excessive fluctuations in the traffic forecast value, thereby improving robustness; θ is an exponential adjustment factor, which is used to control the smoothness between the current traffic forecast value and the concurrency at the previous moment. The larger the exponential adjustment factor, the stronger the smoothing effect on the historical concurrency; C t-1 It is the concurrency at time point t-1, that is, the concurrency at the previous moment, ensuring that the previous historical concurrency is taken into account in the calculation process of the current concurrency to enhance smoothness; is the adaptive smoothing coefficient, with a value range of [0,1], which is used to adjust the impact of the concurrency at the previous moment to ensure a smooth transition of the concurrency; It is an adaptive smoothing term used to enhance the continuity of concurrent volume adjustment.

[0070] The concurrency adjustment formula introduces exponential smoothing adjustment items and adaptive smoothing items, which can more finely control the current concurrency and avoid sudden changes in concurrency when the traffic forecast value changes greatly, which affects stability. Through the combination of nonlinear exponential and adaptive smoothing, it has the ability to respond quickly to sudden traffic. In ultra-high concurrency scenarios, it can adaptively adjust concurrency resources to prevent overload or insufficient resource allocation.

[0071] The dynamic adjustment of concurrency limits the processing capacity, which affects the request flow of some data packets. Since data packets are the specific carriers of tasks, it is necessary to introduce a dynamic scheduling priority adjustment mechanism. Based on the concurrency, the scheduling priority of the data packet is adjusted in combination with the processing time of the data packet and the task priority to ensure that tasks with high scheduling priority can be processed first. The scheduling priority refers to the priority level that determines the order of task processing when resources are limited. A high scheduling priority value means that the task will be executed first in terms of resource allocation and processing order.

[0072] The calculation formula for the scheduling priority of the data packet is:

[0073]

[0074] Among them, S i is the scheduling priority of data packet i; i is the index variable of the data packet; P i is the task priority of data packet i, which is used to indicate the priority processing requirement of the task; It is a dynamic smoothing item, which is used to avoid the scheduling priority being too high or too low due to extreme packet length, making the scheduling process more balanced; is the processing time of data packet i, indicating the amount of computing resources required to complete data packet i; κ is the concurrency impact coefficient, which is used to control the impact of concurrency on scheduling priority and is obtained through experimental tuning; It is a nonlinear adjustment factor used to smooth the difference between the scheduling priority and concurrency of data packets; is the load balancing factor, which dynamically adjusts the scheduling priority of the data packet according to the ratio of the concurrency to the concurrency load balancing benchmark; η is the nonlinear adjustment coefficient, which is used to control the sensitivity of the adjustment formula of the scheduling priority of the data packet to the difference between the scheduling priority and the concurrency, and is obtained through experimental tuning; It is a concurrent load balancing benchmark and a historical average concurrency, which is obtained by averaging the concurrency over a period of time selected based on expert experience, such as 1 hour, 24 hours, or 1 week.

[0075] After obtaining the scheduling priority of the data packet, the data packets are sorted according to the scheduling priority of the data packet, and tasks with high scheduling priority (such as online examinations, live interaction, etc.) are preferentially assigned to the computing node for processing. However, priority scheduling alone cannot avoid the problem of overload under high load conditions. Therefore, it is also necessary to monitor the current load situation in real time to obtain the real-time load volume, introduce a dynamic back pressure control mechanism based on traffic prediction, and dynamically adjust the traffic input by combining the real-time load volume, traffic prediction value and historical data to ensure that the traffic can be effectively controlled when the load is too high to prevent overload and ensure the stable operation and user experience of the education platform.

[0076] The dynamic back pressure control mechanism based on traffic prediction dynamically adjusts the back pressure threshold by introducing traffic prediction value and load ratio change, adjusts traffic input when the real-time load is close to the upper limit of concurrent processing, controls the reception of data packets, ensures stable operation and avoids overload. The back pressure threshold is a threshold value used to limit traffic input when the real-time load is close to the upper limit of concurrent processing. By dynamically adjusting the back pressure threshold, traffic input can be controlled to prevent overload.

[0077] The back pressure threshold calculation formula is as follows:

[0078]

[0079] Where F is the calculated back pressure threshold; L max is the maximum load, which is set according to the specific implementation scenario; L t is the real-time load at time point t, that is, the current real-time load; is the load ratio. A high load ratio indicates that the load is close to full load. The back pressure threshold needs to be lowered to limit more traffic from entering. τ is the weight coefficient, which is used to balance the current actual request traffic D t and traffic forecast The role of back pressure threshold adjustment, the value range is [0, 1], obtained through experimental tuning; It is the coefficient that adjusts the effect of load ratio change, used to control the nonlinear adjustment effect of load ratio, and is obtained through experimental tuning; It is an exponential function, based on the ratio of the current real-time load to the maximum load, used to adjust the impact on the back pressure threshold; It is the parameter of the exponential function, which determines the speed at which the load ratio affects the back pressure threshold and is obtained through experimental tuning; is a parameter used to control the exponential influence of the load ratio, obtained through experimental tuning; δ is a feedback correction coefficient that helps dynamically adjust the back pressure threshold, obtained through experimental tuning; L t-1 It is the real-time load at time point t-1, that is, the load at the previous moment.

[0080] After calculating the back pressure threshold, the traffic input is dynamically adjusted according to the back pressure threshold to limit the entry of too many data packets to prevent excessive load and ensure that tasks with high scheduling priority can be processed first while avoiding resource exhaustion, thus achieving efficient ultra-concurrency processing of educational big data and ensuring the stable operation of the platform and smooth user experience.

[0081] In summary, a system and method for super-concurrent processing of educational big data has been completed.

[0082] The order of the embodiments of the invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0083] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0084] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for super-concurrency processing of educational big data, characterized in that: The following steps are involved: S1: Collect and pre-process the original request data of the education platform to obtain the actual request traffic; Use historical data from actual request traffic to train the weighted autoregressive model; Through the weighted autoregressive model, the future traffic is predicted to obtain the traffic prediction value; S2: Dynamically adjust the concurrency based on the traffic prediction value to obtain the adjusted concurrency. Based on the adjusted concurrency, the scheduling priority of the data packet is calculated; Based on the scheduling priority of the data packets, the data packets are sorted and allocated; a dynamic backpressure control mechanism based on traffic prediction is introduced to dynamically adjust the traffic input, control the reception of data packets, and prevent overload.

2. The method for super-concurrency processing of educational big data according to claim 1, characterized in that: The S1 specifically includes: The weighted autoregressive model introduces an exponential decay factor and a dynamic nonlinear adjustment factor through a weighted autoregressive formula to adjust the impact of historical data on future traffic prediction and obtain a traffic prediction value; the weighted autoregressive formula is as follows: in, is the traffic prediction value, which indicates the predicted request traffic at time point t+1; t is the index variable at the time point; p is the maximum lag period of the historical data used by the weighted autoregressive model when making predictions, indicating the amount of historical data used; is the index variable of the lag period; α is the regression weight coefficient; e is a natural constant; λ is the attenuation factor; is the exponential decay factor; It's time point is the actual request flow; β is the dynamic nonlinear adjustment coefficient; μ is the mean of the historical actual request flow; γ is the nonlinear adjustment factor; is the dynamic nonlinear adjustment factor.

3. The method for super-concurrency processing of educational big data according to claim 2, characterized in that: The S1 specifically includes: The weighted error loss function is introduced and combined with the gradient descent method to iteratively optimize the parameters of the weighted autoregressive model.

4. The method for super-concurrency processing of educational big data according to claim 1, characterized in that: The S2 specifically includes: Based on the traffic prediction value, a concurrent volume adjustment formula is defined, exponential smoothing adjustment items and adaptive smoothing items are introduced to adjust concurrent resources and obtain the adjusted concurrent volume.

5. The method for super-concurrency processing of educational big data according to claim 4, characterized in that: The S2 specifically includes: Based on the adjusted concurrency, a dynamic scheduling priority adjustment mechanism is introduced. The scheduling priority of the data packet is obtained by combining the processing time of the data packet and the task priority. The specific calculation formula of the scheduling priority of the data packet is: Among them, S i is the scheduling priority of data packet i; i is the index variable of the data packet; P i is the task priority of data packet i; is the processing time of data packet i; κ is the concurrency impact coefficient; η is the nonlinear adjustment coefficient; is a concurrent load balancing benchmark; C t is the concurrency at time point t, indicating the adjusted concurrency.

6. The method for super-concurrency processing of educational big data according to claim 5, characterized in that: The S2 specifically includes: Monitor the current load situation in real time to obtain the real-time load volume; introduce a dynamic back pressure control mechanism based on flow prediction, combine the real-time load volume, flow prediction value and historical data, and dynamically adjust the flow input.

7. The method for super-concurrency processing of educational big data according to claim 6, characterized in that: The S2 specifically includes: In the process of implementing the dynamic back pressure control mechanism based on flow prediction, the flow prediction value and load ratio are introduced to dynamically adjust the back pressure threshold and control the flow input; the back pressure threshold calculation formula is as follows: Where F is the back pressure threshold; L max is the maximum load; L t is the real-time load at time point t; is the load ratio; τ is the weight coefficient; is the coefficient that adjusts the effect of load changes; is the parameter of the exponential function; is the parameter used to control the exponential influence of the load ratio; δ is the feedback correction coefficient; L t-1 is the real-time load at time point t-1; is the traffic prediction value, which indicates the predicted request traffic at time point t+1; D t is the actual request traffic at time point t.

8. A system for super-concurrent processing of educational big data, applied to a method for super-concurrent processing of educational big data as claimed in claim 1, characterized in that: Includes the following parts: Data collection and preprocessing module, traffic prediction model module, concurrency adjustment module, scheduling priority adjustment module, load assessment module, dynamic back pressure control module; The data collection and preprocessing module collects and preprocesses the original request data of the education platform, obtains the actual request flow and outputs it; The data collection and preprocessing module is connected to the traffic prediction model module through data transmission; The traffic prediction model module receives the actual request traffic and uses the historical data in the actual request traffic to train the weighted autoregressive model; performs traffic prediction through the weighted autoregressive model to obtain the traffic prediction value and output it; The prediction model module is connected with the concurrency adjustment module and the dynamic back pressure control module through data transmission; The concurrency adjustment module dynamically adjusts the concurrency based on the traffic prediction value, obtains the adjusted concurrency and outputs it; The concurrency adjustment module is connected to the scheduling priority adjustment module by means of data transmission; The scheduling priority adjustment module dynamically adjusts the scheduling priority of the data packet based on the adjusted concurrency and outputs it; The scheduling priority adjustment module is connected to the load evaluation module by means of data transmission; The load evaluation module sorts and distributes data packets based on the scheduling priority of the data packets; monitors the current load situation in real time, obtains the real-time load amount and transmits it; The load assessment module is connected to the dynamic back pressure control module by means of data transmission; The dynamic back pressure control module receives the real-time load and flow prediction value, introduces a dynamic back pressure control mechanism based on flow prediction, dynamically adjusts the flow input, controls the reception of data packets, and prevents overload.

Images