Data processing method

Abstract

The invention relates to the technical field of data processing, and discloses a data processing method which comprises the following steps: setting a plurality of data processing categories, and generating a data processing model according to the data processing categories; acquiring real-time state data of each working node in the calculation cluster, and calculating a state coefficient of the corresponding working node according to the real-time state data; generating a data processing strategy of the to-be-processed large-scale data set according to the data processing model and the state coefficients of all the working nodes; according to the feedback time node, the working data packet of each working node is obtained, whether the adjustment instruction is generated or not is judged, the defects of the data processing strategy are found and adjusted in time, the node resource utilization rate is ensured, and the data processing efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data processing method. BACKGROUND

[0002] With the explosive growth of data scale, the traditional centralized data processing method faces problems such as low efficiency and poor scalability, and cannot meet the processing needs of large-scale data sets. Distributed computing model has become a key technology to solve this problem because it can integrate multi-node computing resources to realize parallel processing.

[0003] In the prior art, general processing logic is often used, which leads to processing bottlenecks in different data processing categories with different computing needs, and the real processing capacity at the working node is not accurately evaluated, which reduces the rationality of task allocation, cannot timely discover abnormal conditions in the processing process, and reduces the node resource utilization and data processing efficiency. SUMMARY

[0004] To solve the above technical problems, the present application provides a data processing method, which realizes differentiated processing of different data processing categories by constructing data processing models of all data processing categories, accurately calculates the state coefficient of each working node, and generates a data processing strategy in combination with the data processing model, improves the task allocation efficiency, sets a reasonable feedback time node to obtain a working data packet, timely discovers and adjusts the deficiencies of the data processing strategy, ensures the node resource utilization and improves the data processing efficiency.

[0005] In some embodiments of the present application, a data processing method is provided, comprising: setting a plurality of data processing categories, generating a data processing model according to the data processing category; collecting real-time state data of each working node in the computing cluster, calculating the state coefficient of the corresponding working node according to the real-time state data; generating a data processing strategy for a large-scale data set to be processed according to the data processing model and the state coefficient of all working nodes; obtaining a working data packet of each working node according to the feedback time node, and determining whether to generate an adjustment instruction.

[0006] In some embodiments of the present application, the data processing model is generated according to the data processing category, comprising: generating a plurality of computing demand indicators according to historical data processing parameters; setting a plurality of data processing categories according to all computing demand indicators; generating a split processing sub-model and a summary calculation sub-model for each data processing category based on historical data processing logs; generate a data processing sub-model of each data processing category according to the split processing sub-model and the summary calculation sub-model; generate a data processing model according to the data processing sub-models of all data processing categories.

[0007] In some embodiments of the present application, the state coefficient of the corresponding working node is calculated according to the real-time state data, including: a plurality of standard state data intervals are preset; the real-time state data of each working node is compared with the corresponding standard state data interval, if the real-time state data is less than the corresponding standard state data interval, the state label of the corresponding real-time state data is marked as a low state level, and a first data difference value is calculated; if the real-time state data is between the corresponding standard state data intervals, the state label of the corresponding real-time state data is marked as a medium state level, and a second data difference value is calculated; if the real-time state data is greater than the corresponding standard state data interval, the state label of the corresponding real-time state data is marked as a high state level, and a third data difference value is calculated.

[0008] In some embodiments of the present application, the state coefficient of the corresponding working node is calculated according to the real-time state data, further including: a plurality of state evaluation indexes are preset, the weight coefficient of each state evaluation index for each data processing category is set in combination with the calculation demand index of each data processing category, and a state evaluation index-weight mapping table of each data processing category is constructed; determine the associated state data of each state evaluation index according to the association relationship between the state evaluation index and the real-time state data of the same working node; calculate the state sub-coefficient of the corresponding state evaluation index according to the state level, the number of state levels and the corresponding data difference value of the associated state data of the same working node of each state evaluation index; calculate the state coefficient of each working node for each data processing category according to the state evaluation index-weight mapping table of each data processing category and the state sub-coefficient of each state evaluation index.

[0009] In some embodiments of the present application, the calculation formula of the state sub-coefficient of the state evaluation index is: ; wherein T1 is the state sub-coefficient of the state evaluation index, t1 is the first state conversion coefficient, n1 is the number of data of the associated state data in the state evaluation index which is in the low state level, n2 is the number of data of the associated state data in the state evaluation index which is in the medium state level, and n3 is the number of data of the associated state data in the state evaluation index which is in the high state level. The first data difference of the i-th associated state data of the state evaluation index. The weight coefficient for the i-th associated state data is... The second data difference of the c-th associated state data of the state evaluation index. The weight coefficient for the c-th associated state data is... The third data difference of the s-th associated state data of the state evaluation index. is the weight coefficient for the s-th associated state data.

[0010] In some embodiments of this application, a data processing strategy for generating a large-scale dataset to be processed is provided based on a data processing model and the state coefficients of all working nodes, including: Obtain the large-scale dataset to be processed and generate the expected computational requirements for the large-scale dataset; Based on the expected computational needs, define the expected processing categories and calculate the similarity between the expected processing categories and each data processing category; All data processing categories are sorted according to similarity, and the data processing category ranked first is set as the target processing category for the large-scale dataset to be processed. By inputting the state coefficients of the large-scale dataset and all working nodes for the target processing category into the data processing sub-model of the target processing category, we obtain the data splitting strategy, task allocation strategy, and summary calculation strategy for the large-scale dataset.

[0011] In some embodiments of this application, before obtaining the work data packets of each working node according to the feedback time node, the process includes: The tasks to be computed at each working node are determined based on the task allocation strategy; An attention coefficient is generated based on the state coefficient of each working node for the target processing category and the complexity of the task to be calculated; The compensation coefficient is set according to the urgency of the target processing category; The attention coefficient at each work node is adjusted according to the compensation coefficient to obtain the adjusted attention coefficient. Based on the relationship between the corrected attention coefficient and the preset attention coefficient range, a corresponding time interval is selected, and several feedback time nodes are generated at the corresponding work nodes according to the selected time interval.

[0012] In some embodiments of this application, a corresponding time interval is selected based on the relationship between the modified attention coefficient and a preset attention coefficient range, including: The first preset attention coefficient range, the second preset attention coefficient range, the third preset attention coefficient range, and the fourth preset attention coefficient range are preset. When the corrected attention coefficient is within the first preset attention coefficient range, the fourth preset time interval is set as the time interval between the feedback time nodes at the corresponding work node. When the corrected attention coefficient is within the second preset attention coefficient range, the third preset time interval is set as the time interval between the feedback time nodes at the corresponding work node. When the corrected attention coefficient is within the third preset attention coefficient range, the second preset time interval is set as the time interval between the feedback time nodes at the corresponding work node. When the corrected attention coefficient is within the fourth preset attention coefficient range, the first preset time interval is set as the time interval between the feedback time nodes at the corresponding work node.

[0013] In some embodiments of this application, the working data packets of each working node are obtained according to the feedback time node, and it is determined whether to generate an adjustment instruction, including: The working data package includes real-time status data and real-time working data at each feedback time point; Several performance evaluation indicators are pre-defined; The real-time work data in the work data package of each work node is evaluated based on several work evaluation indicators to obtain the work sub-coefficient of each work evaluation indicator, and the real-time work coefficient of the corresponding feedback time node is generated by combining the weight coefficient of the corresponding work evaluation indicator. The first correction coefficient for the real-time working coefficient at the current feedback time node is generated based on the changing trend and rate of change of the real-time working coefficient between the current feedback time node and the previous consecutive feedback time nodes. Generate the real-time status coefficient of each working node for the target processing category at the feedback time node; The second correction coefficient for the real-time state coefficient at the current feedback time node is generated based on the changing trend and rate of change of the real-time state coefficient between the current feedback time node and the previous consecutive feedback time nodes. Based on the real-time working coefficient, first correction coefficient, real-time status coefficient and second correction coefficient at the current feedback time node of the same working node, generate the comprehensive processing coefficient of the corresponding working node at the current feedback time node, and determine whether to generate an adjustment instruction based on the comprehensive processing coefficient; The formula for calculating the comprehensive processing coefficient is as follows: ; Where Z is the comprehensive processing coefficient, q1 is the weighting coefficient of the real-time working coefficient, q2 is the weighting coefficient of the real-time status coefficient, w1 is the first correction coefficient, w2 is the second correction coefficient, g is the real-time working coefficient, and T2 is the real-time status coefficient.

[0014] In some embodiments of this application, determining whether to generate an adjustment instruction based on the comprehensive processing coefficient includes: Pre-set the threshold for the comprehensive processing coefficient; When the overall processing coefficient is less than the overall processing coefficient threshold, an adjustment instruction is generated; No adjustment instruction is generated when the comprehensive processing coefficient is not less than the comprehensive processing coefficient threshold.

[0015] The data processing method of this application embodiment has the following advantages compared with the prior art: By constructing data processing models for all data processing categories, differentiated processing for different data processing categories can be achieved. The state coefficient of each working node can be accurately calculated, and data processing strategies can be generated in combination with the data processing models to improve task allocation efficiency. Reasonable feedback time nodes can be set to obtain working data packets, and shortcomings of data processing strategies can be identified and adjusted in a timely manner to ensure node resource utilization and improve data processing efficiency. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a data processing method in an embodiment of this application. Detailed Implementation

[0017] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0018] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0019] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0020] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0021] like Figure 1 As shown in the figure, a data processing method according to an embodiment of this application includes: Step S101: Define multiple data processing categories and generate a data processing model based on the data processing categories; Step S102: Collect real-time status data of each worker node in the computing cluster, and calculate the status coefficient of the corresponding worker node based on the real-time status data; Step S103: Generate a data processing strategy for the large-scale dataset to be processed based on the data processing model and the state coefficients of all working nodes; Step S104: Obtain the working data packets of each working node according to the feedback time node, and determine whether to generate an adjustment instruction.

[0022] In some embodiments of this application, a data processing model is generated based on the data processing category, including: Multiple computational requirement indicators are generated based on historical data processing parameters; Multiple data processing categories are set according to all computational requirements indicators; Based on historical data processing logs, generate a split processing sub-model and a summary calculation sub-model for each data processing category; Generate a data processing sub-model for each data processing category based on the split processing sub-model and the summary calculation sub-model; Generate a data processing model based on the data processing sub-models of all data processing categories.

[0023] In this embodiment, historical data processing parameters refer to the key configurations, input conditions, constraints, etc. used when performing data processing tasks in the past, such as: the range of data to be processed (time span, data volume), computing resource limitations (number of CPU cores, memory), output format requirements (report fields, aggregation granularity), performance indicators (processing time, accuracy), etc.

[0024] In this embodiment, the computational demand index refers to the quantifiable computational target that transforms historical data processing parameters, including data volume index, computation timeliness index, resource constraint index, and business result index.

[0025] In this embodiment, the data processing category refers to a complete data processing task under specific business objectives, data scope, and constraints. Each scenario is a combination of computational requirement indicators, corresponding to a specific business problem. The core of setting the data processing category is to filter and combine computational requirement indicators to obtain data processing objectives, output uses, etc.

[0026] In this embodiment, the core task of the splitting processing sub-model is to decompose large-scale data into data segments that can be processed in parallel and allocate them to working nodes. The core task of the aggregation calculation sub-model is to collect the intermediate results of the splitting stage and merge them into the final result according to the rules. The splitting rules, basic processing, core dimensions of the intermediate result aggregation key, merging logic, secondary calculation, and output of the final result are all different for different data processing categories.

[0027] In this embodiment, the splitting processing sub-models for different data processing categories are obtained by neural training based on historical splitting processing strategies and historical parallel processing operations performed on datasets with similar data processing targets. The summarization calculation sub-model is obtained by neural training based on historical result integration strategies for historical intermediate results output after historical splitting processing of similar data processing targets and corresponding historical processing logic.

[0028] In some embodiments of this application, the state coefficient of the corresponding working node is calculated based on real-time state data, including: Several standard state data ranges are pre-defined; The real-time status data of each working node is compared with the corresponding standard status data range. If the real-time status data is less than the corresponding standard status data range, the status label of the corresponding real-time status data is marked as a low status level, and the corresponding first data difference is calculated. If the real-time status data falls within the corresponding standard status data range, mark the status label of the corresponding real-time status data as the medium status level, and calculate the corresponding second data difference. If the real-time status data is greater than the corresponding standard status data range, mark the status label of the corresponding real-time status data as a high status level, and calculate the corresponding third data difference.

[0029] In this embodiment, the standard state data range refers to the state data range that meets the computational requirements of the data processing category. These ranges are set according to the specific requirements of the data processing task and the characteristics of the business scenario. For example, for tasks that require high computing performance, the standard state data range of the corresponding state data will be set relatively high to ensure that the working node has sufficient processing power; while for tasks with low computing requirements, the standard state data range will be relatively low to make full use of resources and avoid waste.

[0030] In this embodiment, the first data difference is the difference between the minimum value of the real-time state data and the standard state data interval, and the first data difference is always a value less than 0. The second data difference is the difference between the minimum value of the real-time state data and the standard state data interval, and the second data difference is a value not less than 0. The third data difference is the difference between the maximum value of the real-time state data and the standard state data interval, and the third data difference is a value greater than 0.

[0031] In some embodiments of this application, calculating the state coefficient of the corresponding working node based on real-time state data further includes: Several status evaluation indicators are pre-defined, and the weight coefficients of the status evaluation indicators for each data processing category are set in combination with the calculation requirement indicators of each data processing category. A status evaluation indicator-weight mapping table for each data processing category is constructed. Based on the correlation between the status evaluation index and the real-time status data of the same working node, determine the associated status data for each status evaluation index. The state sub-coefficient of each state evaluation index is calculated based on the state level, number of state levels, and corresponding data difference of the associated state data of the same working node for each state evaluation index. The state coefficient of each working node for each data processing category is calculated based on the state evaluation index-weight mapping table for each data processing category and the state sub-coefficient of each state evaluation index.

[0032] In this embodiment, the status evaluation indicators include, but are not limited to, key indicators such as the computing resource utilization rate, data transmission rate, task processing success rate, and system stability of the worker node. These indicators can comprehensively reflect the actual status and performance of the worker node when processing data. The computing resource utilization rate reflects the occupation of computing resources such as CPU and memory of the worker node. The data transmission rate reflects the data interaction efficiency between the worker node and the network or other nodes. The task processing success rate is directly related to the quality of the work node in completing data processing tasks, while the system stability determines whether the worker node can remain reliable and stable during long-term operation.

[0033] In this embodiment, the correlation between state evaluation indicators and real-time state data is determined based on the changing relationship between historical state data and state evaluation indicators. This correlation is derived through extensive historical data analysis and modeling, accurately reflecting the degree of influence of different real-time state data on each state evaluation indicator. The greater the degree of influence, the stronger the correlation between the real-time state data and the corresponding state evaluation indicator, and the greater its weight in calculating the state sub-coefficients. For example, if historical data analysis shows that fluctuations in computing resource utilization are highly correlated with a certain real-time state data (such as CPU utilization), then this real-time state data will be given a higher weight when calculating the state sub-coefficients related to computing resource utilization.

[0034] In this embodiment, when calculating the state coefficient of each working node for each data processing category based on the state evaluation index-weight mapping table for each data processing category and the state sub-coefficient of each state evaluation index, a weighted summation method is used. For example, if the state sub-coefficient of the working node's computing resource utilization rate is 0.8, its weight coefficient for the corresponding data processing category is 0.3; the state sub-coefficient of the data transmission rate is 0.7, its weight coefficient for the corresponding data processing category is 0.4; the state sub-coefficient of the task processing success rate is 0.9, its weight coefficient for the corresponding data processing category is 0.2; and the state sub-coefficient of the system stability is 0.85, its weight coefficient for the corresponding data processing category is 0.1, then the state coefficient of this working node for this log data analysis data processing category is: 0.8×0.3 + 0.7×0.4 + 0.9×0.2 + 0.85×0.1 = 0.785. Through this calculation method, the state of the working node under different data processing categories can be accurately quantified, providing strong support for the optimization and adjustment of the entire data processing method.

[0035] In this embodiment, the weighting coefficients for the state evaluation indicators are different for each data processing category. For example, for computationally intensive data processing categories, the weighting coefficient for computational resource utilization will be relatively high because these tasks have a large demand for computational resources. Conversely, for data processing categories with high data rate requirements, the weighting coefficient for data transmission rate will dominate, since fast and accurate data transmission is crucial for completing such tasks. By setting the weighting coefficients for the state evaluation indicators for each data processing category in this differentiated manner, the actual state of the working nodes under different data processing scenarios can be measured more accurately. This allows for the construction of a state evaluation indicator-weight mapping table for each data processing category, thereby accurately calculating the state coefficient of each working node for each data processing category.

[0036] In some embodiments of this application, the calculation formula for the state sub-coefficient of the state evaluation index is as follows: ; Where T1 is the state sub-coefficient of the state evaluation index, t1 is the first state transition coefficient, n1 is the number of related state data points of the state evaluation index at the low state level, n2 is the number of related state data points of the state evaluation index at the medium state level, and n3 is the number of related state data points of the state evaluation index at the high state level. The first data difference of the i-th associated state data of the state evaluation index. The weight coefficient for the i-th associated state data is... The second data difference of the c-th associated state data of the state evaluation index. The weight coefficient for the c-th associated state data is... The third data difference of the s-th associated state data of the state evaluation index. is the weight coefficient for the s-th associated state data.

[0037] In this embodiment, the first state transition coefficient refers to the value that converts the ratio of data quantities into a value with the same dimension as the state sub-coefficient. The smaller the ratio of data quantities, the larger the state sub-coefficient, and vice versa.

[0038] In this embodiment, the second state transition coefficient refers to the sum of the first difference, the second difference, and the third difference, converted into a value with the same dimension as the state sub-coefficient. When the sum is larger, the state sub-coefficient is larger, and vice versa.

[0039] In some embodiments of this application, a data processing strategy for generating a large-scale dataset to be processed is provided based on a data processing model and the state coefficients of all working nodes, including: Obtain the large-scale dataset to be processed and generate the expected computational requirements for the large-scale dataset; Based on the expected computational needs, define the expected processing categories and calculate the similarity between the expected processing categories and each data processing category; All data processing categories are sorted according to similarity, and the data processing category ranked first is set as the target processing category for the large-scale dataset to be processed. By inputting the state coefficients of the large-scale dataset and all working nodes for the target processing category into the data processing sub-model of the target processing category, we obtain the data splitting strategy, task allocation strategy, and summary calculation strategy for the large-scale dataset.

[0040] In this embodiment, similarity refers to the degree of fit between the expected processing category and each data processing category in terms of computational requirement indicators. This degree of fit is obtained by quantifying the similarity values ​​of each computational requirement indicator and summing them by weight.

[0041] In this embodiment, the data processing category ranked first is the data processing category with the highest similarity. This category best matches the expected computational requirements of the large-scale dataset to be processed, and can make the most of the advantages of the existing data processing model to achieve efficient and accurate data processing.

[0042] In this embodiment, after obtaining the target processing category, the large-scale dataset and the state coefficients of all working nodes for that target processing category are input into a pre-built data processing sub-model for that target processing category. This sub-model comprehensively considers data characteristics, working node states, and historical processing experience to generate an optimized data processing strategy for the current large-scale dataset. This strategy includes, but is not limited to, a data splitting strategy, i.e., how to reasonably divide the large-scale dataset into multiple smaller datasets that can be processed in parallel; a task allocation strategy, i.e., how to allocate the split datasets to the most suitable working nodes for processing based on the real-time status and data processing capabilities of the working nodes; and a summary calculation strategy, i.e., how to collect and integrate the processing results of each working node to form the final data processing output.

[0043] In some embodiments of this application, before obtaining the work data packets of each working node according to the feedback time node, the process includes: The tasks to be computed at each working node are determined based on the task allocation strategy; An attention coefficient is generated based on the state coefficient of each working node for the target processing category and the complexity of the task to be calculated; The compensation coefficient is set according to the urgency of the target processing category; The attention coefficient at each work node is adjusted according to the compensation coefficient to obtain the adjusted attention coefficient. Based on the relationship between the corrected attention coefficient and the preset attention coefficient range, a corresponding time interval is selected, and several feedback time nodes are generated at the corresponding work nodes according to the selected time interval.

[0044] In this embodiment, the greater the urgency of the target processing category, the larger the compensation coefficient, and vice versa. The range of the compensation coefficient is (0.8, 1.2).

[0045] In this embodiment, the larger the state coefficient at the working node, the stronger the computing power of the corresponding working node. When the computing power is stronger and the complexity of the corresponding task to be computed is lower, the attention coefficient is smaller, that is, the less likely timeout processing will occur. In this case, the time interval between feedback time nodes should be increased to avoid frequent acquisition of data packets and waste of resources. On the other hand, when the computing power is lower and the complexity is higher, the attention coefficient is larger, and the time interval between feedback time nodes should be shortened to monitor its working status in a timely manner and prevent problems such as processing timeout.

[0046] In this embodiment, by evaluating the attention coefficient at each working node and reasonably setting the feedback time node at the corresponding working node, the needs of system resource utilization and task monitoring are effectively balanced. This avoids resource waste caused by frequent data acquisition and ensures timely monitoring of high-risk or low-performance nodes.

[0047] In some embodiments of this application, a corresponding time interval is selected based on the relationship between the modified attention coefficient and a preset attention coefficient range, including: The first preset attention coefficient range, the second preset attention coefficient range, the third preset attention coefficient range, and the fourth preset attention coefficient range are preset. When the corrected attention coefficient is within the first preset attention coefficient range, the fourth preset time interval is set as the time interval between the feedback time nodes at the corresponding work node. When the corrected attention coefficient is within the second preset attention coefficient range, the third preset time interval is set as the time interval between the feedback time nodes at the corresponding work node. When the corrected attention coefficient is within the third preset attention coefficient range, the second preset time interval is set as the time interval between the feedback time nodes at the corresponding work node. When the corrected attention coefficient is within the fourth preset attention coefficient range, the first preset time interval is set as the time interval between the feedback time nodes at the corresponding work node.

[0048] In this implementation, the preset attention coefficient range and preset time interval are determined comprehensively based on factors such as the actual performance of work nodes in historical data processing tasks, the urgency of tasks, and resource utilization efficiency.

[0049] In this embodiment, the first preset attention coefficient interval < the second preset attention coefficient interval < the third preset attention coefficient interval < the fourth preset attention coefficient interval, and the first preset time interval < the second preset time interval < the third preset time interval < the fourth preset time interval.

[0050] In some embodiments of this application, the working data packets of each working node are obtained according to the feedback time node, and it is determined whether to generate an adjustment instruction, including: The working data package includes real-time status data and real-time working data at each feedback time point; Several performance evaluation indicators are pre-defined; The real-time work data in the work data package of each work node is evaluated based on several work evaluation indicators to obtain the work sub-coefficient of each work evaluation indicator, and the real-time work coefficient of the corresponding feedback time node is generated by combining the weight coefficient of the corresponding work evaluation indicator. The first correction coefficient for the real-time working coefficient at the current feedback time node is generated based on the changing trend and rate of change of the real-time working coefficient between the current feedback time node and the previous consecutive feedback time nodes. Generate the real-time status coefficient of each working node for the target processing category at the feedback time node; The second correction coefficient for the real-time state coefficient at the current feedback time node is generated based on the changing trend and rate of change of the real-time state coefficient between the current feedback time node and the previous consecutive feedback time nodes. Based on the real-time working coefficient, first correction coefficient, real-time status coefficient and second correction coefficient at the current feedback time node of the same working node, generate the comprehensive processing coefficient of the corresponding working node at the current feedback time node, and determine whether to generate an adjustment instruction based on the comprehensive processing coefficient; The formula for calculating the comprehensive processing coefficient is as follows: ; Where Z is the comprehensive processing coefficient, q1 is the weighting coefficient of the real-time working coefficient, q2 is the weighting coefficient of the real-time status coefficient, w1 is the first correction coefficient, w2 is the second correction coefficient, g is the real-time working coefficient, and T2 is the real-time status coefficient.

[0051] In this embodiment, q1 is 0.4 and q2 is 0.6.

[0052] In this embodiment, the work evaluation index refers to the evaluation of the work performance of each work node within the feedback time node, including but not limited to the efficiency and accuracy of the work node in processing tasks, while the status evaluation result reveals the hardware status, resource utilization and other conditions of the work node.

[0053] In this embodiment, all work evaluation indicators are mapped to corresponding preset values. For example, when the work evaluation indicator is an efficiency indicator, the larger the real-time work data is than the preset value, the larger the corresponding work sub-coefficient is, and vice versa.

[0054] In this embodiment, the generation of the first and second correction coefficients based on the trend and rate of change refers to analyzing the corresponding coefficient data sequence between the current feedback time node and several consecutive previous feedback time nodes, determining the amount of data change using a differential calculation method, and then calculating the rate of change per unit time using a sliding window algorithm. Specifically, the first correction coefficient smooths the slope of the real-time working coefficient using a weighted moving average method, with the weighting coefficient dynamically adjusted based on historical data fluctuation characteristics; the second correction coefficient uses an exponential decay model to suppress short-term fluctuations in the real-time state coefficient, with the decay factor preset according to the working node type. Both correction coefficients are noise-filtered using a Kalman filter algorithm to ensure the stability of the correction results. The final generated correction coefficients are all limited to the range [0,1], and a dynamic adjustment mechanism is triggered when the rate of change exceeds a preset threshold.

[0055] In this embodiment, a comprehensive processing coefficient is generated using real-time work coefficients and real-time status coefficients. Specifically, when the work evaluation result of a certain work node is low or the status evaluation result is abnormal, the comprehensive processing coefficient is low. The system can then generate adjustment instructions to optimize the task allocation and resource allocation for that work node, thereby improving overall data processing efficiency and quality. This also includes dynamically adjusting and optimizing the data processing model. By continuously learning and adapting to changes in the actual data processing scenario, the data processing model can more accurately generate data processing strategies, improving the processing effect of large-scale datasets.

[0056] In some embodiments of this application, determining whether to generate an adjustment instruction based on the comprehensive processing coefficient includes: Pre-set the threshold for the comprehensive processing coefficient; When the overall processing coefficient is less than the overall processing coefficient threshold, an adjustment instruction is generated; No adjustment instruction is generated when the comprehensive processing coefficient is not less than the comprehensive processing coefficient threshold.

[0057] In this embodiment, the setting of the comprehensive processing coefficient threshold needs to be combined with the data processing accuracy requirements and system resource consumption of the specific business scenario. For example, it can be set to 0.85 in the financial risk control scenario and 0.7 in the log analysis scenario.

[0058] In this embodiment, when the overall processing coefficient is lower than the threshold, the system will trigger a three-level adjustment mechanism: first, the task redistribution algorithm will be used to migrate some tasks of the high-load node to the idle node; second, the granularity parameters of the split processing sub-model will be dynamically adjusted; and finally, the merging logic of the summary calculation sub-model will be optimized. If the threshold requirement is not met for three consecutive feedback time nodes, the model reconstruction process will be started and the data processing sub-model will be retrained.

[0059] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the technical principles of this application, and these improvements and substitutions should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: a plurality of data processing categories are set, and a data processing model is generated according to the data processing categories; real-time state data of each working node in the computing cluster is collected, and a state coefficient of the corresponding working node is calculated according to the real-time state data; a data processing strategy of the large-scale data set to be processed is generated according to the data processing model and the state coefficients of all working nodes; work data packets of each working node are obtained according to the feedback time node, and it is judged whether to generate an adjustment instruction.

2. The data processing method of claim 1, wherein, The method for generating a data processing model according to a data processing category comprises the following steps: a plurality of computing demand indexes are generated according to historical data processing parameters; a plurality of data processing categories are set according to all computing demand indexes; a split processing sub-model and a summary calculation sub-model of each data processing category are generated based on historical data processing logs; a data processing sub-model of each data processing category is generated according to the split processing sub-model and the summary calculation sub-model; a data processing model is generated according to the data processing sub-models of all data processing categories.

3. The data processing method of claim 2, wherein, The method for calculating a state coefficient of a corresponding working node according to real-time state data comprises the following steps: a plurality of standard state data intervals are set in advance; the real-time state data of each working node is compared with the corresponding standard state data interval, if the real-time state data is less than the corresponding standard state data interval, the state label of the corresponding real-time state data is marked as a low state level, and a first data difference value is calculated; if the real-time state data is between the corresponding standard state data intervals, the state label of the corresponding real-time state data is marked as a medium state level, and a second data difference value is calculated; if the real-time state data is greater than the corresponding standard state data interval, the state label of the corresponding real-time state data is marked as a high state level, and a third data difference value is calculated.

4. The data processing method of claim 3, wherein, The method for calculating a state coefficient of a corresponding working node according to real-time state data further comprises the following steps: a plurality of state evaluation indexes are set in advance, the weight coefficients of the state evaluation indexes for each data processing category are set in combination with the computing demand indexes of each data processing category, and a state evaluation index-weight mapping table of each data processing category is constructed; the associated state data of each state evaluation index is determined according to the association between the state evaluation index and the real-time state data of the same working node; the state sub-coefficients of the corresponding state evaluation indexes are calculated according to the state level, the number of state levels and the corresponding data difference value of the associated state data of the same working node of each state evaluation index; the state coefficients of each working node for each data processing category are calculated according to the state evaluation index-weight mapping table of each data processing category and the state sub-coefficients of each state evaluation index.

5. The data processing method of claim 4, wherein the calculation formula of the state sub-coefficients of the state evaluation indexes is: The method for generating a data processing strategy of a large-scale data set to be processed according to a data processing model and state coefficients of all working nodes comprises the following steps: ； Wherein, T1 is a state sub-coefficient of a state evaluation index, t1 is a first state transition coefficient, n1 is a data quantity of associated state data in a low state level in the state evaluation index, n2 is a data quantity of associated state data in a medium state level in the state evaluation index, n3 is a data quantity of associated state data in a high state level in the state evaluation index, a first data difference value of the i th associated state data of the state evaluation index, a weight coefficient of the i th associated state data, a second data difference value of the c th associated state data of the state evaluation index, a weight coefficient of the c th associated state data, a third data difference value of the s th associated state data of the state evaluation index, a weight coefficient of the s th associated state data.

6. The data processing method of claim 5, wherein, a large-scale data set to be processed is obtained, and an expected computing demand of the large-scale data set is generated; an expected processing category is set based on the expected computing demand, and the similarity between the expected processing category and each data processing category is calculated; ​ Sort all data processing categories according to the similarity, and set the data processing category ranked first as the target processing category of the large-scale data set to be processed; Input the large-scale data set and the state coefficient of all work nodes for the target processing category into the data processing sub-model of the target processing category to obtain the data splitting strategy, task allocation strategy and summary calculation strategy of the large-scale data set.

7. The data processing method of claim 6, wherein, Before obtaining the work data packet of each work node according to the feedback time node, including: Determine the to-be-calculated task at each work node according to the task allocation strategy; Generate the attention coefficient according to the state coefficient of each work node for the target processing category and the complexity of the to-be-calculated task; Set the compensation coefficient according to the urgency of the target processing category; Modify the attention coefficient of each work node according to the compensation coefficient to obtain the modified attention coefficient; Select the corresponding time interval according to the relationship between the modified attention coefficient and the preset attention coefficient interval, and generate a plurality of feedback time nodes at the corresponding work node according to the selected time interval.

8. The data processing method of claim 7, wherein, Selecting the corresponding time interval according to the relationship between the modified attention coefficient and the preset attention coefficient interval, including: Pre-set the first, second, third and fourth preset attention coefficient intervals; When the modified attention coefficient is in the first preset attention coefficient interval, set the fourth preset time interval as the time interval between the feedback time nodes at the corresponding work node; When the modified attention coefficient is in the second preset attention coefficient interval, set the third preset time interval as the time interval between the feedback time nodes at the corresponding work node; When the modified attention coefficient is in the third preset attention coefficient interval, set the second preset time interval as the time interval between the feedback time nodes at the corresponding work node; When the modified attention coefficient is in the fourth preset attention coefficient interval, set the first preset time interval as the time interval between the feedback time nodes at the corresponding work node.

9. The data processing method of claim 8, wherein, According to the feedback time node, obtain the work data packet of each work node, and determine whether to generate an adjustment instruction, including: The work data packet includes real-time state data and real-time work data at each feedback time node; Pre-set a plurality of work evaluation indexes; According to a plurality of work evaluation indexes, evaluate the real-time work data in the work data packet of each work node to obtain a work sub-coefficient of each work evaluation index, and generate a real-time work coefficient of the corresponding feedback time node in combination with the weight coefficient of the corresponding work evaluation index; Generate a first correction coefficient of the real-time work coefficient at the current feedback time node according to the change trend and rate of the real-time work coefficients of the current feedback time node and the previous continuous feedback time node; Generate the real-time state coefficient of each work node for the target processing category at the feedback time node; Generate a second correction coefficient of the real-time state coefficient at the current feedback time node according to the change trend and rate of the real-time state coefficients of the current feedback time node and the previous continuous feedback time node; According to the real-time work coefficient, the first correction coefficient, the real-time state coefficient and the second correction coefficient of the same work node at the current feedback time node, a comprehensive processing coefficient of the corresponding work node at the current feedback time node is generated, and whether to generate an adjustment instruction is determined according to the comprehensive processing coefficient; The calculation formula of the comprehensive processing coefficient is: ； Wherein, Z is the comprehensive processing coefficient, q1 is the weight coefficient of the real-time work coefficient, q2 is the weight coefficient of the real-time state coefficient, w1 is the first correction coefficient, w2 is the second correction coefficient, g is the real-time work coefficient, and T2 is the real-time state coefficient.

10. The data processing method of claim 9, wherein, Whether to generate an adjustment instruction is determined according to the comprehensive processing coefficient, including: The comprehensive processing coefficient threshold is set in advance; When the comprehensive processing coefficient is less than the comprehensive processing coefficient threshold, an adjustment instruction is generated; When the comprehensive processing coefficient is not less than the comprehensive processing coefficient threshold, no adjustment instruction is generated.

Images