A method and system for distributed parallel processing of mass network data

Abstract

The application provides a kind of mass network data distributed parallel processing method and system, it is related to data processing field, solve the problem of low efficiency of distributed parallel processing caused by the mismatch of data characteristics and computing power characteristics and the large overhead of global rearrangement in prior art.The method comprises: obtaining network data to be processed;Based on the first feature information of network data, the time consumption of prediction calculation is determined based on the time consumption of calculation, and a plurality of data fragments are obtained based on the slice boundary;Based on the mapping relationship between the first feature information of data fragment and the second feature information of computing node, the data fragment is distributed to the corresponding computing node;In the parallel processing process of data fragment, in response to the deviation between the actual progress and the estimated progress of computing node exceeds the preset threshold, the task of computing node that has not been completed is divided into subtask slices, and is migrated to other computing nodes for local redistribution.The application is used for mass network data processing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a distributed parallel processing method and system for massive network data. Background Technology

[0002] With the rapid development of network technology, network data exhibits characteristics of massive volume, heterogeneity, and high dynamism. When processing such data in a distributed parallel manner, existing technologies typically employ a static, uniform sharding strategy based on data size, allocating shards to computing nodes using round-robin or random methods. When uneven node load occurs, global task reordering is often relied upon to restore balance. However, the computation time of network data is not solely linearly related to data volume; the internal data correlation and structural complexity lead to significant differences in actual computational overhead, making static sharding highly susceptible to severe computational skew. Furthermore, fixed allocation strategies fail to match data characteristics with node computing power characteristics, resulting in wasted computing resources. Global reordering mechanisms not only interrupt ongoing tasks but also incur substantial communication and scheduling overhead, further exacerbating system latency. Therefore, existing technologies suffer from low efficiency in distributed parallel processing due to the mismatch between data characteristics and computing power characteristics, as well as excessive global reordering overhead. Summary of the Invention

[0003] This application provides a distributed parallel processing method and system for massive network data, which solves the technical problem of low efficiency in distributed parallel processing caused by the mismatch between data characteristics and computing power characteristics and the excessive global rearrangement overhead in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] Firstly, a distributed parallel processing method for massive network data is provided, comprising: acquiring network data to be processed; predicting computation time based on first feature information of the network data, and determining sharding boundaries based on computation time to obtain multiple data shards, wherein the first feature information is the data feature of the network data; allocating the data shards to corresponding computing nodes based on the mapping relationship between the first feature information of the data shards and the second feature information of the computing nodes, wherein the second feature information is the computing power feature of the computing nodes; during the parallel processing of data shards, in response to the deviation between the actual progress and the estimated progress of the computing nodes exceeding a preset threshold, dividing the unprocessed tasks on the computing nodes into subtask slices and migrating them to other computing nodes for local reallocation.

[0006] The above scheme avoids computational skew caused by static sharding by predicting time consumption based on data characteristics and dynamically determining sharding boundaries. It improves resource utilization by adaptively matching and allocating data characteristics and computing power characteristics, and avoids the huge overhead of global reordering by migrating only the local reallocation of unfinished tasks. Thus, it achieves a significant improvement in the efficiency of distributed parallel processing.

[0007] In conjunction with the first aspect mentioned above, in one possible implementation, the first feature information includes multidimensional processing complexity features. The multidimensional processing complexity features include at least data correlation degree and structural complexity. Data correlation degree characterizes the dependency relationship between data units in the network data, and structural complexity characterizes the degree of hierarchical nesting of data units in the network data.

[0008] In conjunction with the first aspect mentioned above, in one possible implementation, the computation time is predicted based on the first feature information of the network data, and the partition boundaries are determined based on the computation time to obtain multiple data partitions. This includes: calculating the predicted computation time of each data unit in the network data by performing a weighted summation calculation based on the data correlation and structural complexity according to a preset weight coefficient; and determining the partition boundaries according to the principle that the sum of the predicted computation time of the data units in each data partition tends to be balanced to obtain multiple data partitions.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the mapping relationship includes: allocating data shards with high correlation characteristics to computing nodes with high memory bandwidth, and allocating data shards with high computational complexity characteristics to computing nodes with high computing power.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, data shards are allocated to corresponding computing nodes based on the mapping relationship between the first feature information of the data shards and the second feature information of the computing nodes. This includes: calculating a matching score between the data shards and each computing node based on the mapping relationship, wherein the matching score is determined based on the correlation coefficient between the quantized value of the first feature information of the data shards and the quantized value of the second feature information of the computing nodes; and allocating the data shards to the corresponding computing nodes according to the priority of the matching score from high to low.

[0011] In conjunction with the first aspect mentioned above, in one possible implementation, unfinished tasks on computing nodes are divided into subtask slices and migrated to other computing nodes for local redistribution. This includes: triggering rebalancing when the task processing progress deviation of a computing node exceeds a preset progress deviation threshold; computing nodes whose task processing progress deviation exceeds the preset progress deviation threshold are lagging computing nodes; and rebalancing does not involve completed tasks or global task rearrangement. Unfinished tasks on lagging computing nodes are treated as subtask slices and migrated to idle computing nodes, wherein idle computing nodes are computing nodes whose current load is below a preset load threshold.

[0012] In conjunction with the first aspect mentioned above, in one possible implementation, tasks that have not been processed on lagging computing nodes are treated as subtask slices and migrated to idle computing nodes. This includes: splitting the unprocessed tasks into at least two finer-grained subtask slices based on the first feature information of the unprocessed portions in the subtask slices; obtaining the current load index of each computing node; and migrating the finer-grained subtask slices to the idle computing node with the lowest current load index, wherein the current load index includes the resource utilization rate of the computing node.

[0013] In conjunction with the first aspect above, in one possible implementation, after migrating the unprocessed subtask slices on the computing nodes to other computing nodes for local reallocation, the method further includes: feeding back the actual computation time of the locally reallocated subtask slices on other computing nodes to the stage corresponding to the predicted computation time; and updating the weight coefficient of the first feature information or adjusting the prediction model parameters based on the actual computation time.

[0014] In conjunction with the first aspect mentioned above, in one possible implementation, rebalancing does not involve the reordering of completed tasks or global tasks.

[0015] Secondly, a distributed parallel processing system for massive network data is provided, comprising: a communication unit and a processing unit; the communication unit is used to acquire network data to be processed; the processing unit is used to predict the computation time based on the first feature information of the network data, and determine the partition boundaries based on the computation time to obtain multiple data partitions, wherein the first feature information is the data feature of the network data; based on the mapping relationship between the first feature information of the data partitions and the second feature information of the computing nodes, the data partitions are allocated to the corresponding computing nodes, wherein the second feature information is the computing power feature of the computing nodes; during the parallel processing of data partitions, in response to the deviation between the actual progress and the estimated progress of the computing nodes exceeding a preset threshold, the unfinished tasks on the computing nodes are divided into subtask slices and migrated to other computing nodes for local reallocation.

[0016] This application overcomes the computational skew problem caused by traditional static sharding neglecting the inherent complexity of data by predicting computation time based on the first feature information of network data and determining the sharding boundary accordingly, thus making the expected computational overhead of each shard more balanced. Adaptive allocation is achieved through the mapping relationship between the first feature information of data shards and the second feature information of computing nodes, ensuring a precise match between data features and node computing power advantages, avoiding misallocation and waste of computing resources. Furthermore, by responding to progress deviations during parallel processing by fine-grained splitting and local migration of unfinished tasks, the application abandons the crude approach of global task reordering, quickly eliminating the computational bottleneck of lagging nodes with minimal scheduling cost, thereby significantly improving the overall efficiency of distributed parallel processing of massive network data.

[0017] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a distributed parallel processing method for massive network data provided in this application embodiment;

[0019] Figure 2 A flowchart illustrating another distributed parallel processing method for massive network data provided in this application embodiment;

[0020] Figure 3 A flowchart illustrating another distributed parallel processing method for massive network data provided in this application embodiment;

[0021] Figure 4 A flowchart illustrating another distributed parallel processing method for massive network data provided in this application embodiment;

[0022] Figure 5 A flowchart illustrating another distributed parallel processing method for massive network data provided in this application embodiment;

[0023] Figure 6A flowchart illustrating another distributed parallel processing method for massive network data provided in this application embodiment;

[0024] Figure 7 A schematic diagram of the structure of a distributed parallel processing system for massive network data provided in this application embodiment;

[0025] Figure 8 This is a flowchart illustrating another distributed parallel processing method for massive network data provided in an embodiment of this application. Detailed Implementation

[0026] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0027] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0028] Example 1:

[0029] like Figure 1 As shown in the figure, this embodiment provides a distributed parallel processing method for massive network data. This method achieves efficient distributed processing of massive heterogeneous network data by establishing a core logical closed loop of data feature perception, adaptive matching and incremental reallocation. The following is a detailed description of each step.

[0030] S101. Obtain the network data to be processed.

[0031] Network data refers to the collection of data to be processed generated during network communication, business interaction, or system operation. Its form is not limited to structured data tables, but can also be semi-structured log files or unstructured message streams.

[0032] In this embodiment, the network data to be processed can be obtained by receiving traffic packets captured by a network probe in real time through a streaming interface, by batch reading historically stored log archives from a distributed file system, or by subscribing to real-time event data pushed by a business system through a message queue. It should be understood that the triggering mechanism for the acquisition action can be either periodic polling or event-driven passive reception, as long as the object to be processed can be imported into the system.

[0033] It should be noted that the network data targeted by this invention has obvious internal characteristic differences, and its computational cost is often not a simple linear relationship with the data volume. Therefore, the acquisition of data is usually accompanied by preliminary parsing of metadata in order to provide a basis for subsequent feature extraction.

[0034] As an example, in a traffic analysis scenario, a mirror probe deployed around the core switch continuously pushes the captured raw packet data to the access buffer of the processing cluster in a streaming manner.

[0035] Based on the above steps, the multi-source, multi-modal data acquisition mechanism ensures broad adaptability to massive heterogeneous network data, providing a complete input guarantee for subsequent differentiated processing based on the inherent characteristics of the data.

[0036] S102. Based on the first feature information of the network data, predict the computation time, and determine the fragmentation boundary based on the computation time to obtain multiple data fragments.

[0037] Among them, the first characteristic information refers to the inherent attribute dimension of network data that can affect processing overhead. It is not limited to data correlation or structural complexity, but can also be data size, data entropy value or keyword density, etc., which are the data characteristics of network data; computation time refers to the estimated time cost required to process a specific data unit; and partition boundary refers to the logical cutting point for dividing different data partitions.

[0038] In this embodiment, predicting computation time based on the first feature information essentially involves establishing a mapping model from the data attribute space to the time cost space. This mapping model is not limited to simple linear weighted regression; it can also be support vector regression or decision tree fitting based on historical samples. After obtaining the predicted computation time for each data unit, the process of determining the partition boundaries based on this time to obtain multiple data partitions is carried out according to the principle of making the total estimated cost of each partition tend to be consistent, thereby avoiding the phenomenon of computational skew where some partitions are extremely heavy while others are extremely light.

[0039] It should be noted that determining the sharding boundary based on the time consumption rather than simply the data size in bytes is the key to overcoming the long-tail effect caused by traditional static sharding. This is because data of the same size in bytes may have different processing times if their internal correlation is different.

[0040] As an example, in a log parsing scenario, the nesting level of log entries and the cross-entry IP correlation are extracted as the first feature information and input into a pre-trained time-consuming prediction model. Consecutive log entries with similar estimated total time consumption are divided into the same shard, forming multiple expected load-balanced data shards.

[0041] Based on the above steps, by introducing the inherent characteristics of the data to guide time prediction and boundary partitioning, the computational bias caused by the difference in data complexity is eliminated, so that the expected processing overhead of each partition tends to be balanced.

[0042] S103. Based on the mapping relationship between the first feature information of the data shard and the second feature information of the computing node, the data shard is allocated to the corresponding computing node.

[0043] The second feature information refers to the processing advantage attributes of the computing node's hardware resources. It is not limited to CPU computing power indicators, but can also be memory bandwidth capacity, disk read and write speed, or GPU parallel throughput, etc., which are the computing power characteristics of the computing node; the mapping relationship refers to the directional adaptation logic between data feature requirements and node resource advantages.

[0044] In this embodiment, allocation based on mapping relationships directs data shards with different characteristics to the hardware nodes best suited to handle those characteristics. For example, data shards with strong internal dependencies and frequent random memory accesses are allocated to nodes with ample memory bandwidth, while data shards with intensive logic operations and high computational complexity are allocated to nodes with higher CPU clock speeds or core counts.

[0045] It should be noted that this adaptive matching allocation based on feature mapping breaks the blindness of traditional polling or random allocation that ignores data and hardware characteristics, enabling the precise utilization of heterogeneous computing resources and avoiding resource mismatch and waste caused by high computing power nodes processing low-complexity data or low-memory nodes processing highly correlated data.

[0046] As an example, in a heterogeneous cluster containing large memory nodes and high computing power nodes, the scheduler prioritizes sending data shards marked with high correlation characteristics to large memory nodes and data shards marked with high computational complexity characteristics to high computing power nodes, thereby achieving precise matching between data characteristics and hardware advantages.

[0047] Based on the above steps, adaptive allocation is achieved by establishing a mapping relationship between data characteristics and computing power characteristics, which allows the advantages of heterogeneous computing power resources to be accurately utilized, significantly improving the overall cluster's resource utilization and processing throughput.

[0048] S104. During the parallel processing of data sharding, in response to the deviation between the actual progress and the estimated progress of the computing node exceeding a preset threshold, the unfinished tasks on the computing node are divided into subtask slices and migrated to other computing nodes for local redistribution.

[0049] Among them, actual progress refers to the proportion of tasks that have been completed or the time that has been consumed at the current computing node; estimated progress refers to the proportion of tasks that should be achieved at the current moment based on the predicted time consumption; preset threshold refers to the allowable progress lag tolerance limit; subtask slice refers to a smaller granular task unit cut out from the unfinished part on the lagging node; local redistribution refers to supplementary scheduling performed only on the unfinished part among a limited number of nodes.

[0050] In this embodiment, when the deviation exceeds a preset threshold, it indicates that a node has experienced a computational lag exceeding the tolerance range, triggering a rebalancing mechanism. Unfinished tasks are divided into subtask slices. This division is not limited to equal distribution based on remaining data entries; it can also be a fine-grained segmentation based on the first feature information of the unfinished portion. Migrating to other computing nodes for local redistribution involves scheduling these slices to currently less loaded, idle nodes to assist in completing the remaining computations.

[0051] It should be noted that local reallocation does not involve rolling back completed tasks or rearranging global tasks. It only performs incremental compensation scheduling for the unfinished parts, which fundamentally avoids the huge communication overhead and interruption of tasks being processed caused by global rescheduling.

[0052] As an example, if a node lags behind by more than 20% due to processing sudden complex data during parallel processing, the remaining 30% of the task on that node is immediately split into two subtask slices and migrated to two idle nodes with a current load rate of less than 15% to continue processing, while the task that has been 70% completed remains on the original node without interference.

[0053] Based on the above steps, by performing incremental local migration only on the incomplete parts, the computational bottleneck of lagging nodes is quickly eliminated with minimal scheduling cost, avoiding system oscillations and additional overhead caused by global reordering.

[0054] Based on the above technical solution, this embodiment avoids the computational skew caused by static sharding by predicting the time consumption based on data features and dynamically determining the sharding boundary. It improves resource utilization by adaptively matching and allocating data features and computing power features, and avoids the huge overhead of global reordering by migrating only the local reallocation of unfinished tasks. Thus, it achieves a significant improvement in the efficiency of distributed parallel processing.

[0055] Example 2:

[0056] This embodiment further refines the specific dimensions of the first feature information based on embodiment 1.

[0057] Preferably, the first feature information includes multidimensional processing complexity features, which at least include data correlation degree and structural complexity. Data correlation degree characterizes the dependency relationship between data units in the network data, and structural complexity characterizes the degree of hierarchical nesting of data units in the network data.

[0058] Among them, data correlation refers to the degree of connection between different data units in network data in terms of logic, business or time sequence, such as the frequency of the same IP address in different log entries or the context dependency of messages in the same session stream; structural complexity refers to the depth of the internal organizational structure of data units, such as the number of nested levels of key-value pairs in JSON format data or the branching level of the tag tree in XML messages.

[0059] In the embodiments of this application, the methods for quantifying data correlation are not limited to statistical co-occurrence frequency; they can also be based on graph theory models to calculate the weights of connection edges between nodes or on transaction sequences to calculate the transition probabilities of dependencies before and after. The methods for quantifying structural complexity are not limited to calculating nesting depth; they can also be based on evaluating the total number of nodes in a syntax tree or the branching factors of a parsing path. It should be understood that multidimensional processing complexity features, in addition to data correlation and structural complexity, can also encompass attribute dimensions that reflect processing overhead, such as data entropy and keyword density, as long as these dimensions can characterize the inherent nonlinear computational properties of the data.

[0060] It should be noted that introducing multi-dimensional processing complexity characteristics rather than simply the size of the data is the core premise for overcoming computational bias, because the actual processing time of network data is often exponentially positively correlated with the topological connections and hierarchical depth within the data, rather than linearly related to the size of bytes.

[0061] As an example, in the scenario of traffic log parsing, the frequency of interaction between the source IP and the destination IP in the log entry is extracted as a quantitative value of data correlation, and the nesting level of protocol fields such as HTTP headers in the log is extracted as a quantitative value of structural complexity. Together, they constitute the multidimensional processing complexity feature of the entry.

[0062] Based on the above steps, by introducing data correlation and structural complexity to construct multidimensional processing complexity features, the inherent nonlinear computational overhead of network data is accurately characterized, providing a reliable feature basis for subsequent accurate prediction of computation time.

[0063] In one possible implementation of the embodiments of this application, combined with Figure 1 ,like Figure 2 As shown, the above S102 can be implemented through the following S201 and S202, which are explained in detail below:

[0064] S201. Based on data correlation and structural complexity, perform weighted summation calculation according to preset weight coefficients to obtain the prediction calculation time of each data unit in the network data.

[0065] Among them, the preset weight coefficient refers to the parameter factor used to adjust the degree of influence of different feature dimensions on the final time consumption; the weighted summation calculation refers to the process of combining the quantized value of each feature dimension with its corresponding weight coefficient.

[0066] In the embodiments of this application, the weighted summation calculation is not a simple linear accumulation, but introduces a nonlinear mapping and exponential adjustment mechanism to truly reflect the surge in time consumption caused by high complexity and high correlation.

[0067] Optionally, the prediction calculation time satisfies the following formula:

[0068]

[0069] in, This indicates the time taken to predict a data unit. A quantitative value representing the degree of data correlation is obtained by extracting the dependencies between data units in network data. For example, it can be used to count the co-occurrence frequency of the same identifier in different data units or to calculate the weight of the connection edges between nodes based on graph theory models. The quantified value representing structural complexity is obtained by quantifying and extracting the degree of nesting of data units in network data. For example, it can be used to calculate the nesting level of key-value pairs in JSON data or to evaluate the total number of nodes in a syntax tree. W1 and W2 represent the preset weighting coefficients for data relevance and structural complexity, respectively. α represents the non-linear adjustment index for relevance, and β represents the non-linear adjustment index for structural complexity. This represents the time constant for basic calculations. This is achieved by introducing an exponential function. Sum of power functions It can capture the real physical mechanism that computation time increases non-linearly when the correlation or complexity exceeds a certain threshold.

[0070] It should be noted that the initial values ​​of the preset weight coefficients W1 and W2, as well as the nonlinear adjustment indices α and β, can be set based on regression analysis of historical sample data or expert experience, and can be dynamically updated through a closed-loop feedback mechanism during subsequent operation, so that the prediction model can adapt to the characteristics of different business data distributions.

[0071] As an example, when processing logs containing deeply nested protocols and high-frequency IP interactions, setting larger α and β values ​​allows for faster prediction of latency, even with a small data volume, due to the high correlation and deep nesting characteristics. It will also be significantly higher than ordinary flat and isolated data entries, thus preventing such complex data from being underestimated in terms of time consumption and causing long-tail lag.

[0072] Based on the above steps, by introducing a weighted summation of nonlinear mapping and exponential adjustment, the nonlinear steep increase relationship between data features and computation time was fitted, eliminating the potential computational skew caused by feature underestimation.

[0073] S202. Based on the principle that the sum of the prediction calculation time of the data units in each data shard tends to be balanced, the shard boundaries are determined to obtain the multiple data shards.

[0074] Among them, the partition boundary refers to the logical cutting point for dividing different data partitions; the principle of balancing the total time consumption means making the cumulative value of the prediction calculation time of all data units in each partition as consistent as possible.

[0075] In this embodiment, the principle for determining the sharding boundary is not limited to requiring the total time consumption of each shard to be absolutely equal. It can also be to minimize the variance of the total time consumption of each shard, or to control the difference between the maximum and minimum sharding time within a preset tolerance range. Specifically, during the sharding process, a greedy algorithm can be used to sequentially fill data units into the shard with the minimum total time consumption, or a dynamic programming algorithm can be used to find the sharding combination with the minimum variance globally. After determining the sharding boundary, the logically divided data blocks are encapsulated into independently schedulable task units and output. Each output data shard not only contains specific network data content but also carries a statistical summary of the first feature information aggregated within that shard, such as average correlation degree and maximum nesting depth. This allows for rapid extraction of shard-level features for computational power adaptation calculations in the subsequent adaptive matching stage without rescanning the entire dataset.

[0076] It's important to note that determining shard boundaries based on the predicted total processing time rather than the data size in bytes is crucial for ensuring load balancing across subsequent computing nodes. This is because two shards of the same size can have vastly different processing times if their internal data relationships and structural complexity distributions differ. Simply dividing data byte-wise can easily cause some shards to become computational bottlenecks. Furthermore, while sharding logically cuts data, it maintains the continuity of the original file in physical storage, identifying boundaries only by recording offsets and index ranges, thus avoiding the additional I / O overhead of data copying.

[0077] As an example, during log sharding, the scheduler combines a small number of entries containing highly correlated and deeply nested features with a large number of entries containing low correlated and flat structures within the same shard. This makes the total prediction time of the shard roughly consistent with the total prediction time of a shard containing only moderately complex entries, thus achieving true load balancing from a computational cost perspective. After logical sharding is completed, the scheduler generates a metadata tag for each shard, containing the shard's total prediction time, average correlation, and peak complexity. These tagged shard objects are then pushed to the allocation queue, awaiting the next step of computational power matching and scheduling.

[0078] Based on the above steps, by quantifying the time consumption and dividing the boundaries according to the principle of balancing the total time consumption, load balancing of data sharding is achieved from the dimension of computational overhead. The feature summary is bound to the sharded data for output, providing a direct and usable decision input for the adaptive matching of subsequent computing power features. This effectively eliminates the computational skew caused by the difference in the inherent complexity of the data, while avoiding the overhead of repeated feature calculations and improving the overall scheduling efficiency.

[0079] Based on the above technical solution, this embodiment constructs multi-dimensional features by introducing data correlation and structural complexity, and accurately predicts the computation time by using a nonlinear exponential adjustment formula. Finally, it determines the partitioning boundary based on the time balance principle, thereby achieving accurate characterization and load balancing of the inherent nonlinear computational overhead of network data. This effectively eliminates the computational skew caused by simple linear weighting or static byte equalization, and significantly improves the stability and efficiency of distributed parallel processing.

[0080] Example 3:

[0081] This embodiment, based on Embodiment 1, further refines the mapping relationship between data shards and computing nodes, as well as the quantitative calculation mechanism for matching degree scoring, to construct a more defensive computing power adaptation and scheduling decision model, such as... Figure 3 As shown, this specifically includes S301, S302, and S303, which are explained in detail below:

[0082] S301. Determining the mapping relationship includes: allocating data shards with high correlation characteristics to computing nodes with high memory bandwidth, and allocating data shards with high computational complexity characteristics to computing nodes with high computing power.

[0083] Among them, high correlation refers to the fact that the data fragment contains a large number of dependencies that require cross-unit joint processing, which will generate intensive random memory access operations during processing; high memory bandwidth refers to the high throughput of computing nodes that can read or write data from memory per unit time; high computational complexity refers to the fact that the data fragment contains deep nesting or complex logical operations, which mainly consume the logical operation resources of the central processing unit during processing; and high computing power refers to the fact that computing nodes have higher clock speeds, more cores, or stronger floating-point operation capabilities.

[0084] In this embodiment, the core logic of the aforementioned targeted adaptation mechanism lies in matching the bottleneck resources of the task. Data fragments with high correlation often require frequent searches for context-dependent data in memory during processing. At this time, computational operations may stall due to waiting for memory responses, forming a memory-intensive bottleneck. Allocating these fragments to computing nodes with high memory bandwidth can significantly shorten data loading latency, preventing the processor from being idle for extended periods. Conversely, data fragments with high computational complexity rely primarily on the processor's logical deduction and instruction execution for parsing and computation, forming a computationally intensive bottleneck. Allocating these fragments to computing nodes with high computing power can accelerate instruction flow and shorten pure computation time.

[0085] It should be noted that in reality, data sharding often has a certain degree of correlation and computational complexity. In this case, the mapping relationship is not an absolute either / or relationship, but rather a dominant adaptation based on the bottleneck attribute with the highest proportion in the sharding features.

[0086] Based on the above steps, by establishing a directional mapping relationship between data characteristics and node hardware advantages, memory-intensive data can make full use of memory bandwidth advantages to reduce memory access latency, and computation-intensive data can make full use of computing power advantages to accelerate logical operations, thus achieving precise matching and efficient utilization of heterogeneous computing power resources.

[0087] S302. Based on the mapping relationship, calculate the matching degree score between the data shards and each computing node.

[0088] Among them, the matching score is a quantitative indicator used to measure the degree of fit between the data sharding demand and the computing node supply. It is determined based on the correlation coefficient between the quantitative value of the first feature information of the data sharding and the quantitative value of the second feature information of the computing node. The correlation coefficient refers to the numerical value of the degree of correlation between two multidimensional vectors in terms of direction or distance.

[0089] In this embodiment, the process of calculating the matching score essentially involves constructing a demand feature vector from the first characteristic information of the data shards (such as relevance and complexity), constructing a supply feature vector from the second characteristic information of the computing nodes (such as memory bandwidth and computing power), and calculating the correlation coefficient between the two using a vector space model. The determination of the correlation coefficient is not limited to simple linear ratio calculation; it can also be based on cosine similarity calculation, i.e., calculating the cosine of the angle between the demand vector and the supply vector. The closer the cosine value is to 1, the more consistent the demand and supply directions are, and the higher the matching degree. Alternatively, it can use Euclidean distance inverse mapping, i.e., calculating the Euclidean distance between the two vectors in multidimensional space. The smaller the distance, the closer the demand and supply are, and the distance is mapped to a score between 0 and 1 using an exponential decay function, with a higher score for a smaller distance.

[0090] Based on the above steps, by quantifying the correlation coefficient to calculate the matching score, the abstract mapping relationship is transformed into a quantifiable scheduling decision model, ensuring that each data allocation can achieve the current optimal computing power adaptation effect.

[0091] S303. According to the matching score from high to low priority, the data shards are allocated to the corresponding computing nodes.

[0092] Priority refers to the allocation order based on the matching score; the higher the score, the higher the priority in obtaining an allocation opportunity.

[0093] In this embodiment, priority allocation means that the system prioritizes assigning data shards to the computing nodes with the highest matching degree of demand. However, in multi-shard concurrent scheduling scenarios, there is often multi-node competition logic where multiple data shards compete for the same optimal computing node. In this case, the allocation strategy is not limited to simple scoring and ranking for exclusive use; a competition coordination mechanism can also be introduced. For example, a priority queue can be set up for matching degree scores. When multiple shards compete for the same node, different competition weights are assigned according to the predicted processing time or urgency of the shards, and the shard with the longest processing time or the highest urgency has priority to preempt the node; or a random backoff and retry mechanism can be adopted, where shards that fail to preempt the node automatically switch to the node with the second-highest score after a short wait, avoiding deadlock and scheduling stagnation. It should be understood that the core of priority allocation is to ensure the optimal resource allocation from a global perspective, rather than the absolute optimal exclusive use of a single local shard. Complete allocation means that the data shard not only logically determines the processing node, but also physically pushes the shard data or its access pointer to the local storage or memory buffer of the target node through the network link. The target node then starts the parallel processing task for that shard. After allocation, the scheduler records the binding relationship between the shard and the node, as well as the estimated progress information, so as to monitor the progress and determine the trigger for local reallocation in subsequent processing.

[0094] It's important to note that priority allocation is a dynamic decision-making process. As shards are continuously assigned, the remaining available resources of computing nodes—the quantified value of the second feature information—decay in real time. Therefore, the matching score and priority ranking of subsequent shards need to be recalculated based on the real-time remaining supply capacity of the nodes, rather than using the initial static supply value. Furthermore, the defensive nature of adaptive matching allocation lies in its breaking away from the blindness of traditional polling or random allocation, which ignores data and hardware characteristics. It ensures that every allocation decision is supported by a quantitative model, and even in extreme cases of dynamically changing node resources or extremely uneven data feature distribution, it can find a relatively optimal landing node through real-time scoring and priority competition.

[0095] As an example, the scheduler calculates matching scores for node A for three shards to be assigned: 0.9, 0.85, and 0.8. When the shard with a score of 0.9 preempts node A, node A's remaining computing power and memory bandwidth decrease. The scheduler immediately recalculates the matching scores of the remaining two shards based on node A's updated supply vector. At this point, their scores for node A may drop to 0.5 and 0.4, thus naturally shifting to other idle nodes with higher compatibility, achieving dynamic optimal matching of global resources. In the log parsing cluster, the scheduler, based on cosine similarity scoring, adaptively pushes log shards with high IP correlation to nodes with large memory and log shards with deep protocol nesting to nodes with high computing power. Neither requires manual specification of the target queue; the scheduler automatically completes the precise matching of features and computing power and the data distribution.

[0096] Based on the above steps, dynamic allocation and competition coordination are carried out by prioritizing matching scores, and adaptive matching is completed throughout the entire process from logical binding to physical distribution. This ensures that heterogeneous computing resources can still be used accurately and orderly in complex concurrent scheduling environments, avoiding overall resource mismatch and waste caused by local optima, and significantly improving the overall cluster's resource utilization and processing throughput.

[0097] Based on the above technical solutions, this embodiment establishes a directional adaptation mechanism with high correlation and high memory bandwidth, high complexity and high computing power, and uses the correlation coefficient of the vector space model to quantify the matching degree score. Combined with priority allocation and multi-node competition coordination mechanism, the abstract mapping relationship is transformed into a precise and computable scheduling decision model, which ensures the accurate matching and efficient utilization of heterogeneous computing power resources and effectively prevents the waste of computing power and processing bottlenecks caused by feature mismatch.

[0098] Example 4:

[0099] This embodiment, based on Embodiment 1, further refines the triggering mechanism for local reallocation and the specific algorithm for fine-grained splitting and migration, such as... Figure 4As shown, this specifically includes S401 and S402, which will be described in detail below:

[0100] S401. When the task processing progress deviation value of the computing node exceeds the preset progress deviation threshold, rebalancing is triggered.

[0101] Among them, the computing nodes whose task processing progress deviation exceeds the preset progress deviation threshold are considered lagging computing nodes. Rebalancing does not involve the reordering of completed tasks or global tasks. The task processing progress deviation refers to the absolute value or proportion of the difference between the actual progress of the computing node and the estimated progress calculated based on the predicted time. The preset progress deviation threshold refers to the maximum allowable progress lag tolerance limit of the system. Lagging computing nodes are computing nodes whose progress deviation exceeds this tolerance limit. Rebalancing refers to supplementary scheduling adjustments for the lagging part.

[0102] In this embodiment, the preset progress deviation threshold is typically set between 10% and 30%, and the specific value can be dynamically adjusted according to the cluster's sensitivity to real-time performance. When the deviation is less than 10%, minor fluctuations in the system can be absorbed by the node's own acceleration catch-up mechanism without triggering scheduling overhead; however, when the deviation exceeds 30%, it often means that the node has encountered a serious hardware failure or a sudden surge in data. If rebalancing is not triggered at this time, it will lead to severe long-tail lag in the overall task.

[0103] It is particularly important to emphasize that the rebalancing mechanism defined in this invention has strict locality and incrementality. It does not involve the rollback and recalculation of completed tasks, nor does it involve the global reordering of tasks being processed or already processed on all nodes in the entire cluster. It only performs compensatory scheduling for the unfinished task portions on lagging nodes.

[0104] It should be noted that if a global reordering strategy is adopted, not only will it require interrupting the tasks currently being executed by all nodes, resulting in a huge waste of computing resources and loss of intermediate states, but it will also cause data migration and scheduling oscillations across the entire cluster. Its communication overhead and recovery delay far outweigh the benefits of local migration, which is the core reason why this invention insists on incremental local processing.

[0105] As an example, in a log parallel parsing cluster, a preset progress deviation threshold of 20% is set. When the actual parsing progress of a node is only 40% while the estimated progress should reach 65%, the progress deviation value reaches 25%, exceeding the threshold. The system immediately marks the node as a lagging computing node and triggers rebalancing. The 40% parsing task that the node has already completed is retained on the original node and continues to output without any interference.

[0106] Based on the above steps, by setting a reasonable progress deviation threshold, local rebalancing is precisely triggered, and global reordering is strictly excluded. This quickly eliminates local computational bottlenecks with minimal scheduling costs, avoiding global oscillations and additional overhead.

[0107] S402. Take the unfinished tasks on the lagging computing nodes as subtask slices and migrate them to idle computing nodes. The idle computing nodes are computing nodes whose current load is lower than the preset load threshold. The rebalancing does not involve the reprocessing of completed tasks or the global task rearrangement.

[0108] Among them, a subtask slice refers to a smaller-granularity task unit cut out from the unfinished part of a lagging node; an idle computing node refers to a computing node that currently bears a light processing load and has the ability to accept additional tasks; and a preset load threshold refers to the upper limit of the load that determines whether a node has the ability to receive idle tasks.

[0109] In this embodiment, the preset load threshold is typically set between 15% and 40% of the total node capacity. When a node's load is below 15%, it indicates that the node is almost idle and has ample receiving and acceleration processing capabilities. When the load exceeds 40%, the node is already operating at a medium to high load, and forcibly migrating in tasks may cause the node to quickly become a new lagging node, creating a vicious cycle of load migration. When screening idle computing nodes, the system will exclude nodes with loads exceeding the preset threshold in real time, only directing subtask slices to nodes that truly have surplus computing power. Similarly, the migration operation here still follows the incremental local principle and never involves completed processing tasks or global task rearrangement.

[0110] It should be noted that the preset progress deviation threshold and the preset load threshold are not set in isolation or statically. Logically, they form a linked defense system: the deviation threshold determines when external assistance must be sought, and the load threshold determines who is capable of providing assistance. The cooperation between the two ensures that the rebalancing action is taken by the right node at the right time, avoiding blind scheduling.

[0111] As an example, after discovering a lagging node, the scheduling center scans the cluster status and selects nodes with current CPU and memory utilization rates below 20% as idle computing nodes. Then, it packages the unfinished tasks on the lagging node into subtask slices and pushes them to the idle node via the network link. Other nodes in the cluster with a load exceeding 40% do not participate in this migration scheduling.

[0112] Based on the above steps, by setting a load threshold to accurately select idle nodes to undertake migration tasks, it is ensured that the receiving end of incremental local redistribution has sufficient processing capacity, thus avoiding secondary skew caused by load transfer.

[0113] In one possible implementation of the embodiments of this application, combined with Figure 4 ,like Figure 5 As shown, the above S402 can be implemented through the following S501 and S502, which are explained in detail below:

[0114] S501. Based on the first feature information of the unprocessed part in the subtask slice, the unprocessed task is split into at least two finer-grained subtask slices.

[0115] Among them, finer-grained subtask slices refer to smaller task blocks obtained by further subdividing the original incomplete task. These blocks contain fewer data units but have clearer feature boundaries.

[0116] In this embodiment, the splitting algorithm is not simply divided equally according to the number of data entries, but rather a recursive or binary split based on the first feature information of the unprocessed portion. Specifically, the predicted computation time of each data unit in the unprocessed portion is first extracted (calculated based on the correlation and complexity quantification values ​​in the first feature information). Then, a binary search algorithm is used to find a cutting point along the logical order of the data sequence, such that the sum of the predicted computation time of the left part of the cutting point is as close as possible to the sum of the predicted computation time of the right part. If the sum of the computation time of a certain part is still too large after the split, the binary search is recursively performed on that part until the sum of the predicted computation time of all finer-grained subtask slices falls within the preset slice computation time tolerance range. This feature information-based splitting method ensures that the computational cost of each subtask slice after splitting tends to be balanced, avoiding the local lag that occurs again after migration due to the concentration of all high-complexity data units in a certain slice.

[0117] It should be noted that fine-grained splitting based on the first feature information rather than the data byte size is the key to ensuring a smooth load transition after migration. This is because incomplete data of the same byte size may contain a core dependency graph with extremely high correlation. Blindly splitting it evenly may sever the dependency context or cause the receiving node to be overloaded.

[0118] As an example, when processing incomplete log entries, the relevance and nesting depth of the remaining entries are extracted to recalculate the predicted time. It is found that the top 30% of entries contain a large amount of highly relevant data with high IP interactions, accounting for 60% of the time. At this time, a binary splitting algorithm is used to cut these 30% of highly relevant entries into a finer-grained subtask slice, which is then migrated to idle nodes with large memory bandwidth. The remaining 70% of low-relevance entries are cut into another slice and migrated to ordinary computing power nodes, achieving a secondary accurate adaptation of features and computing power.

[0119] Based on the above steps, by recursively or binary fine-grainedly splitting the incomplete tasks based on the first feature information, it is ensured that the transferred subtask slices tend to be balanced in terms of computational cost and feature attributes, further improving the precision and timeliness of local rebalancing.

[0120] S502. Obtain the current load index of each computing node and migrate the finer-grained subtask slices to the idle computing node with the lowest current load index.

[0121] Among them, the current load indicators include the resource utilization rate of the computing node, which is a dynamic parameter that reflects the real-time resource consumption status of the computing node; the resource utilization rate refers to the proportion of the node's core hardware resources occupied at the current moment.

[0122] In this embodiment, the real-time acquisition mechanism for current load metrics is implemented through a lightweight monitoring agent deployed on the node side. This agent periodically collects the CPU core utilization, memory buffer usage, and network I / O throughput of the nodes using high-frequency heartbeat packets, and weights and merges them into a comprehensive resource utilization value before reporting it to the scheduling center. After receiving more granular subtask slices, the scheduling center sorts them in ascending order according to the latest comprehensive resource utilization reported by each idle node, and prioritizes migrating the slices to the idle computing node with the lowest resource utilization, ensuring that it can digest the migrated tasks as quickly as possible.

[0123] As an example, based on the data reported by the monitoring agent, the scheduling center finds that the current overall resource utilization rate of node B is 12% and that of node C is 18%. At this time, the system will prioritize migrating the finer-grained subtask slices based on the high correlation characteristics to node B, using its most abundant memory bandwidth and CPU resources to accelerate processing, thereby achieving a smooth transition and rapid resolution of the load.

[0124] Based on the above steps, by acquiring load indicators such as resource utilization in real time and prioritizing migration to the lowest load node, it is ensured that fine-grained slices can obtain the most abundant computing power support, thus guaranteeing a smooth transition and rapid convergence of the overall system load.

[0125] Based on the above technical solutions, this embodiment sets a linkage triggering mechanism between the progress deviation threshold and the load threshold, and strictly excludes global reordering, to accurately trigger local rebalancing with minimal scheduling cost; by performing recursive or binary fine-grained splitting based on the first feature information, it ensures the balance of migration slices in terms of computational overhead and feature attributes; by accurately guiding the lowest load node with real-time acquisition of resource utilization, it further improves the precision and timeliness of local rebalancing, effectively preventing secondary tilting and system oscillation caused by load transfer.

[0126] Example 5:

[0127] This embodiment, based on Embodiment 1, further refines the closed-loop feedback update mechanism after local redistribution to construct a dynamically adaptive optimization closed-loop logic, such as... Figure 6 As shown, this specifically includes S601 and S602, which are described in detail below:

[0128] S601. Feed back the actual computation time of the subtask slices after local redistribution on other computing nodes to the corresponding stage of the predicted computation time.

[0129] Among them, actual computation time refers to the actual time consumed from the start of processing to the final completion of the subtask slice after it has been migrated to a new computing node; feedback refers to the process of transmitting the data stream of the backend execution results back to the frontend decision model.

[0130] In this embodiment, the establishment of the feedback path essentially breaks down the information barrier between the execution layer and the decision-making layer. After a subtask slice is processed on an idle computing node, the monitoring agent on that node accurately records the clock cycles or timestamp difference consumed in processing the slice, forming an actual computation time data packet. This data packet not only includes the total time but may also include the detailed time consumption of each data unit within the slice. Subsequently, through the high-speed signaling network within the cluster, this data packet is transmitted back to the prediction model storage area of ​​the scheduling center, where it is compared and aligned with the predicted computation time generated for that slice during the initial sharding process.

[0131] It should be noted that this feedback mechanism breaks the traditional one-way flow pattern of decision-making and execution being separated in distributed scheduling. This makes the prediction model no longer a blind inference based on historical experience or static assumptions, but can continuously perceive the execution feedback of the real physical world, thus possessing the physical basis for self-correction and evolution.

[0132] As an example, in a log parsing cluster, a subtask slice that was migrated to an idle node due to a sudden complexity delay is completed after 2.5 seconds of actual processing on the idle node. The idle node immediately feeds back the actual computation time of 2.5 seconds, along with the slice's feature label, to the scheduling center's time prediction engine via the control bus, and compares it with the record that predicted its time to be 1.8 seconds.

[0133] Based on the above steps, by feeding back the actual computation time to the prediction stage, the information loop between execution and decision-making is opened up, providing a real and reliable error comparison basis for the subsequent adaptive correction of model parameters.

[0134] S602. Based on the actual computation time, update the weight coefficient of the first feature information or adjust the prediction model parameters.

[0135] Among them, updating the weight coefficients refers to dynamically adjusting the proportion of the influence of multidimensional features in the time-consuming calculation based on the prediction error; adjusting the prediction model parameters refers to correcting the nonlinear adjustment index or basic constant in the prediction formula.

[0136] In this embodiment, based on the error between the actual computation time and the predicted computation time, a specific optimization algorithm is used to iteratively correct the weight coefficients or model parameters. For updating the weight coefficients, a gradient descent algorithm can be optionally used to calculate the partial derivative of the predicted computation time with respect to each weight coefficient. The mean square error between the actual and predicted computation times is then used as the loss function, and the values ​​of W1 and W2 are gradually updated along the negative gradient direction with a preset learning rate, so that the predicted computation time of subsequent data units with similar characteristics can converge to the actual computation time. For adjusting the prediction model parameters, especially the nonlinear adjustment exponents α and β, an exponential smoothing algorithm can be optionally used for dynamic approximation. When it is found that the actual computation time of highly correlated data is systematically higher than the predicted value, the value of α is increased by a smoothing factor to strengthen the steep increase effect of the exponential function, making the model more sensitive to the surge in computation time caused by high correlation.

[0137] Based on the above steps, by dynamically updating the weight coefficients and model parameters using algorithms such as gradient descent or exponential smoothing based on feedback errors, the adaptive evolution of the prediction model is achieved, enabling the system to continuously approach the actual computational cost and completely eliminating the fixed bias of the static model.

[0138] Based on the above technical solution, this embodiment establishes a complete dynamic adaptive optimization closed-loop logic by feeding back the actual computation time to the prediction stage and dynamically updating the weight coefficients and model parameters using specific optimization algorithms. This enables the time prediction model to continuously self-correct and evolve as running data accumulates, continuously approaching the actual computational cost. This fundamentally avoids the fixed bias of static prediction models when facing dynamic heterogeneous data, and significantly enhances the scheduling accuracy and processing efficiency of the system in the long term.

[0139] Example 6:

[0140] like Figure 7 As shown in the figure, this embodiment provides a distributed parallel processing system 70 for massive network data. The system realizes a complete closed-loop logic of data feature perception, adaptive matching and incremental redistribution through hardware structure binding and underlying collaborative operation. The connection relationship and operation mechanism of each unit are described in detail below.

[0141] The distributed parallel processing system 70 for massive network data includes a communication unit 701 and a processing unit 702. The communication unit 701 and the processing unit 702 are logically connected through an internal high-speed bus or data channel to ensure low-latency flow of network data and control signaling.

[0142] Communication unit 701 is used to acquire network data to be processed. Specifically, as the interface between the system and the external network environment, communication unit 701's interface mechanism for acquiring network data is not limited to capturing real-time traffic packets by listening to the mirror port of the Ethernet network card. It can also be used to read historical log archives in batches from the distributed storage cluster through the Fibre Channel adapter, or to receive sensor data streams pushed by IoT edge nodes through the wireless radio frequency module. After receiving the raw data stream, communication unit 701 is responsible for performing preliminary physical layer decoding and frame synchronization, and caching the parsed data blocks in the shared memory area. Then, it sends a data ready interrupt signal to processing unit 702 to trigger the subsequent feature extraction and scheduling process. It should be understood that although... Figure 1 The communication unit 701 is shown to exist as a separate module, but in other embodiments, the communication unit 701 can also be integrated into the input / output controller of the processing unit 702, as long as it satisfies the function of stably importing external heterogeneous network data into the system's internal processing pipeline.

[0143] Processing unit 702 is used to predict computation time based on the first feature information of network data, and determine the sharding boundary based on the computation time to obtain multiple data shards. The first feature information refers to the data characteristics of the network data. Based on the mapping relationship between the first feature information of the data shards and the second feature information of the computing nodes, the data shards are allocated to the corresponding computing nodes. The second feature information refers to the computing power characteristics of the computing nodes. During the parallel processing of data shards, if the deviation between the actual progress and the estimated progress of a computing node exceeds a preset threshold, the unfinished tasks on the computing node are divided into subtask slices and migrated to other computing nodes for local reallocation. Specifically, processing unit 702, as the core scheduling and decision-making center of the system, integrates a feature perception engine, a time prediction module, an adaptive matching scheduler, and an incremental rebalancing controller. When the processing unit 702 receives the data ready signal from the communication unit 701, the feature perception engine first extracts the multi-dimensional processing complexity features of the data block from the shared memory as the first feature information. Then, the time consumption prediction module calculates the predicted computation time of each data unit based on the first feature information by calling the built-in nonlinear exponential adjustment formula, and determines the partition boundary according to the principle of total time consumption balance, and cuts the data into multiple logical partitions. Next, the adaptive matching scheduler extracts the quantized value of the first feature information of each partition and the quantized value of the second feature information reported by each computing node in the cluster, determines the matching degree score by calculating the correlation coefficient of the vector space, and adaptively allocates the partitions to the corresponding computing nodes according to the score priority. In the parallel processing stage, the incremental rebalancing controller continuously monitors the progress feedback of each node. Once it finds that the deviation between the actual progress and the estimated progress exceeds the preset threshold, it immediately triggers the local rebalancing logic, splits the unfinished part on the lagging node into sub-task slices based on the first feature information, and migrates them to the idle node with the lowest current load, and strictly ensures that this process does not involve the reordering of completed tasks or global tasks. The processing unit 702 is also used to feed back the actual computation time of the subtask slices after local redistribution on other computing nodes to the link corresponding to the predicted computation time; the processing unit 702 is also used to update the weight coefficient of the first feature information or adjust the prediction model parameters based on the actual computation time.

[0144] In one possible implementation, the processing unit 702 is further used to perform weighted summation calculation based on data correlation and structural complexity according to preset weight coefficients to obtain the prediction calculation time of each data unit in the network data; and to determine the partition boundary according to the principle that the sum of the prediction calculation time of the data units in each data partition tends to be balanced, thereby obtaining multiple data partitions.

[0145] In one possible implementation, the processing unit 702 is further configured to calculate the matching score between the data shards and each computing node based on the mapping relationship. The matching score is determined based on the correlation coefficient between the quantized value of the first feature information of the data shard and the quantized value of the second feature information of the computing node. The data shards are then allocated to the corresponding computing nodes according to the priority of the matching score from high to low.

[0146] In one possible implementation, the processing unit 702 is further configured to trigger rebalancing when the task processing progress deviation value of a computing node exceeds a preset progress deviation threshold. The computing node whose task processing progress deviation value exceeds the preset progress deviation threshold is a lagging computing node. The rebalancing does not involve completed processing tasks or global task reordering. The unfinished processing tasks on the lagging computing nodes are treated as subtask slices and migrated to idle computing nodes. The idle computing nodes are computing nodes whose current load is lower than a preset load threshold. The rebalancing does not involve completed processing tasks or global task reordering.

[0147] In one possible implementation, the processing unit 702 is further configured to split the unprocessed task into at least two finer-grained subtask slices based on the first feature information of the unprocessed portion in the subtask slice; obtain the current load index of each computing node; and migrate the finer-grained subtask slice to the idle computing node with the lowest current load index, wherein the current load index includes the resource utilization rate of the computing node.

[0148] Through the close collaboration and data flow of its internal sub-modules, the processing unit 702 has fully realized closed-loop control from data access to dynamic scheduling and local compensation in its hardware architecture.

[0149] Through the structural design and underlying collaborative operation of the communication unit 701 and the processing unit 702, this embodiment firmly binds the functional constraints of data feature perception, adaptive matching allocation, and incremental local rebalancing to specific interface mechanisms and logical connection relationships at the hardware system level. This ensures the efficient execution and system stability of distributed parallel processing of massive network data, fundamentally avoiding the technical problem of low efficiency in distributed parallel processing caused by mismatch between data features and computing power features and excessive global reordering overhead.

[0150] Example 7:

[0151] This embodiment provides a computer-readable storage medium storing instructions that, when executed on a distributed parallel processing system for massive network data, cause the distributed parallel processing system for massive network data to perform the distributed parallel processing method for massive network data as described in the foregoing embodiment.

[0152] Specifically, the computer-readable storage medium is not limited to non-volatile physical storage media such as solid-state drives and disk arrays, but can also be a medium capable of carrying instruction data, such as optical discs, read-only memory, or random access memory. When the processing device deployed on the master node of the distributed cluster reads and loads the aforementioned instructions from this medium through its communication unit, the execution logic of the instructions immediately triggers the operation of the entire dynamic scheduling mechanism. The instructions first drive the communication unit of the processing device to execute the step of acquiring the network data to be processed; subsequently, the instructions call the feature perception engine and time prediction module inside the processing unit to predict the computation time based on the first feature information of the network data and determine the sharding boundaries to obtain multiple data shards; next, the instructions activate the adaptive matching scheduler, which allocates the data shards to the corresponding computing nodes based on the mapping relationship between the first feature information of the data shards and the second feature information of the computing nodes; during the parallel processing phase, the instructions continuously drive the incremental rebalancing controller, responding to the deviation between the actual progress and the estimated progress of the computing nodes exceeding a preset threshold, dividing the unfinished tasks on the computing nodes into subtask slices and migrating them to other computing nodes for local reallocation. In addition, the instructions stored in the medium also include sub-instruction segments that trigger multi-dimensional feature definition, nonlinear time-consuming prediction formula calculation, matching degree score quantification, fine-grained splitting algorithm, and closed-loop feedback parameter update, ensuring that the system can completely reproduce the entire closed loop from feature perception to adaptive matching to incremental local redistribution and model evolution during execution.

[0153] It should be understood that solidifying the above-mentioned method steps into instructions in a computer-readable storage medium provides an extremely reliable carrier for the software distribution, rapid deployment, and off-site reuse of this distributed parallel processing method. Whether in a large-scale cluster scheduling system in a cloud data center or in a lightweight data processing unit in an edge computing node, simply connecting this medium and running the instructions enables hardware devices that originally lacked feature perception and dynamic balancing capabilities to instantly acquire efficient processing capabilities based on the inherent characteristics of the data, such as accurate data sharding, adaptive computing power matching, and incremental local rebalancing. This greatly reduces the threshold for implementing the technical solution and the cost of promotion.

[0154] Example 8:

[0155] To more clearly illustrate the technical solution of the present invention and its operational effect in a real commercial environment, such as Figure 8 As shown, taking the massive network traffic log analysis scenario of a telecom operator as an example, the abstract algorithm in the aforementioned embodiments is mapped to specific business objects, demonstrating the complete closed-loop process of data feature perception, adaptive matching, and incremental redistribution.

[0156] In the core networks of telecommunications operators, tens of terabytes of traffic logs are generated daily. These logs are not only massive in volume but also exhibit highly diverse internal characteristics. Traditional static sharding and round-robin allocation methods often severely underestimate the computational overhead of highly correlated and deeply nested logs when processing such data, leading to long-tail lag in the cluster and parsing throughput far below expectations. This embodiment effectively solves the above-mentioned pain points by applying the distributed parallel processing method of the present invention to this scenario. Specifically, it includes S801 to S805, which are described in detail below:

[0157] S801: Obtain massive traffic logs from telecom operators to be processed.

[0158] In this embodiment, a mirror probe deployed around the core switch continuously captures raw packet data and generates a semi-structured log entry stream after preliminary parsing. This log entry stream is then pushed in real-time to the access buffer of the processing cluster via a high-speed message queue subscription interface, completing the data acquisition process.

[0159] S802: Based on the multidimensional processing complexity characteristics of massive traffic logs, predict the computation time and determine the sharding boundary based on the computation time to obtain multiple data shards.

[0160] In this embodiment, for traffic log scenarios, the multidimensional processing complexity features include at least the data correlation degree representing the dependency relationship between log entries and the structural complexity representing the degree of hierarchical nesting of log entries. In this embodiment, the data correlation degree is specifically quantified as the frequency of occurrence and interaction topology density of the same IP address in different log entries, and the structural complexity is specifically quantified as the nesting level and branching factor of protocol fields such as HTTP Headers or TLS handshake information in the logs. Based on the IP dependency and protocol nesting depth of log entries, the predicted computation time of each entry is calculated according to a preset weight coefficient formula that includes nonlinear exponential adjustment. For example, if an entry contains a group of IPs with high-frequency interactions and protocol nesting reaches 5 levels, its predicted computation time will increase exponentially, far exceeding that of flat and isolated large-byte entries. Subsequently, according to the principle that the sum of the predicted computation time of log entries in each data shard tends to be balanced, the sharding boundary is determined, and high-time-consuming entries are combined with a large number of low-time-consuming entries in the same shard to obtain multiple data shards with expected load balancing. This mechanism fundamentally avoids the computational skew caused by the clustering of high-complexity logs due to traditional byte-based equal distribution.

[0161] S803: Based on the mapping relationship between the multidimensional processing complexity characteristics of data sharding and the hardware advantage attributes of each server, data shards are allocated to the corresponding computing nodes.

[0162] In this embodiment, the mapping relationship is explicitly defined as follows: data shards with high IP correlation are allocated to computing nodes with high memory bandwidth, and data shards with high computational complexity are allocated to computing nodes with high computing power. The targeted adaptation mechanism is that log parsing with high IP correlation requires frequent lookups of context dependency graphs in memory, forming a memory-intensive bottleneck; nodes with high memory bandwidth can significantly shorten its data loading latency. Conversely, log parsing with deep protocol nesting consumes a large amount of CPU logic operation resources, forming a computationally intensive bottleneck; nodes with high computing power can accelerate its instruction flow. Furthermore, based on this mapping relationship, a matching score is calculated between log shards and each server. This score is determined based on the cosine similarity correlation coefficient between the quantified values ​​of the shard's IP correlation and nesting depth and the quantified values ​​of the server's memory bandwidth and computing power. Subsequently, according to the priority of the matching score from high to low, the log shards are adaptively allocated to the corresponding optimal server nodes, achieving precise matching of data characteristics and hardware advantages, and avoiding misallocation and waste of computing resources.

[0163] S804. During the parallel processing of data shards, in response to the deviation between the actual progress and the estimated progress of the computing node exceeding a preset threshold, the unfinished tasks on the computing node are divided into subtask slices and migrated to other computing nodes for local redistribution.

[0164] In this embodiment, during log parsing, sudden traffic surges often cause a large number of unexpectedly high-relevance entries to flood some shards, severely delaying the actual parsing progress of nodes that were originally expected to be normal. When the task processing progress deviation of a server exceeds a preset progress deviation threshold (e.g., set to 20%), the system triggers rebalancing and marks the server as a lagging computing node. It is important to emphasize that this rebalancing does not involve rolling back and recalculating already parsed logs or reordering global tasks; it only performs incremental compensation scheduling for log entries that have not yet been completed on the lagging node. The system treats the log tasks that have not been parsed on the lagging node as subtask slices. Based on the first feature information of the incomplete parts, the system recursively splits the incomplete log entries into two subtask slices according to the remaining predicted time, forming at least two finer-grained subtask slices. For example, the part of the remaining entries containing high IP dependency is cut into a separate fine-grained slice. Subsequently, the system obtains the current load metrics of each server (including real-time CPU and memory resource utilization) and prioritizes migrating finer-grained log slices to idle computing nodes with the lowest current resource utilization (such as large memory nodes with a load below 15%) for further parsing. This local migration mechanism quickly eliminates local computing bottlenecks caused by sudden traffic surges with minimal scheduling overhead, avoiding system turbulence and huge communication overhead caused by global reordering.

[0165] S805. Feed back the actual computation time of the subtask slices after local redistribution on other computing nodes to the corresponding stage of predicted computation time; based on the actual computation time, update the weight coefficients of the multidimensional processing complexity features or adjust the prediction model parameters.

[0166] In this embodiment, after the migrated fine-grained log slice is parsed on an idle node, the idle node accurately records its actual clock cycles consumed as the actual computation time. This time, along with the slice's feature label, is then fed back to the scheduling center's time prediction engine via the cluster's internal signaling network. The scheduling center detects that the actual time consumed by the slice (e.g., 2.5 seconds) is significantly higher than the initial prediction (1.8 seconds). Based on this error, it increases the weight coefficient W1 of IP correlation using a gradient descent algorithm and fine-tunes the nonlinear adjustment exponent α using an exponential smoothing algorithm. This ensures that when encountering similar sudden bursts of highly correlated logs, the predicted time automatically increases to approximately 2.3 seconds, which is closer to the actual value. This closed-loop feedback update mechanism allows the time prediction model to continuously approximate the actual parsing overhead, eliminating the fixed bias of static prediction models when facing dynamic heterogeneous traffic, and ensuring the scheduling accuracy and processing efficiency of the system over the long term.

[0167] Based on the above technical solution, this embodiment fully demonstrates the closed-loop process in the scenario of massive traffic log analysis for telecom operators, from log input, prediction of time consumption based on protocol nesting and IP dependency and dynamic sharding, targeted allocation of high IP-related shards to nodes with large memory bandwidth, triggering fine-grained migration of incomplete log shards when sudden traffic causes lag, to optimizing the prediction model based on actual parsing time feedback. This method effectively overcomes the computational skew and resource mismatch problems caused by traditional static sharding and blind allocation when processing network data with greatly different internal characteristics. It quickly eliminates long-tail lag with minimal local scheduling cost and significantly improves the parsing throughput of heterogeneous clusters.

[0168] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0169] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0170] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A distributed parallel processing method for massive network data, characterized in that, include: Acquire network data to be processed; Based on the first feature information of the network data, the computation time is predicted, and the fragmentation boundary is determined based on the computation time to obtain multiple data fragments, wherein the first feature information is the data feature of the network data; Based on the mapping relationship between the first feature information of the data shard and the second feature information of the computing node, the data shard is allocated to the corresponding computing node, wherein the second feature information is the computing power feature of the computing node; During the parallel processing of the data shards, in response to the deviation between the actual progress and the estimated progress of the computing node exceeding a preset threshold, the unfinished tasks on the computing node are divided into subtask slices and migrated to other computing nodes for local redistribution.

2. The method according to claim 1, characterized in that, The first feature information includes multidimensional processing complexity features, which include at least data correlation degree and structural complexity. The data correlation degree characterizes the dependency relationship between data units in the network data, and the structural complexity characterizes the degree of hierarchical nesting of data units in the network data.

3. The method according to claim 2, characterized in that, The process involves predicting computation time based on the first feature information of the network data, determining fragmentation boundaries based on the computation time, and obtaining multiple data fragments, including: Based on the data correlation degree and the structural complexity, a weighted summation calculation is performed according to a preset weight coefficient to obtain the prediction calculation time of each data unit in the network data; Based on the principle that the sum of the prediction calculation time of data units within each data shard tends to be balanced, the shard boundaries are determined to obtain the multiple data shards.

4. The method according to claim 1, characterized in that, The mapping relationship includes: allocating data shards with high correlation characteristics to computing nodes with high memory bandwidth, and allocating data shards with high computational complexity characteristics to computing nodes with high computing power.

5. The method according to claim 4, characterized in that, The process of allocating data fragments to corresponding computing nodes based on the mapping relationship between the first feature information of the data fragments and the second feature information of the computing nodes includes: Based on the mapping relationship, a matching score is calculated between the data shard and each computing node. The matching score is determined based on the correlation coefficient between the quantized value of the first feature information of the data shard and the quantized value of the second feature information of the computing node. The data shards are allocated to the corresponding computing nodes according to the matching score from high to low priority.

6. The method according to claim 1, characterized in that, The step of dividing unprocessed tasks on the computing node into subtask slices and migrating them to other computing nodes for local reallocation includes: When the task processing progress deviation value of a computing node exceeds the preset progress deviation threshold, a rebalancing is triggered. The computing node whose task processing progress deviation value exceeds the preset progress deviation threshold is a lagging computing node. The rebalancing does not involve the reordering of completed tasks or global tasks. The unfinished tasks on the lagging computing nodes are treated as subtask slices and migrated to idle computing nodes, wherein the idle computing nodes are computing nodes whose current load is lower than a preset load threshold.

7. The method according to claim 6, characterized in that, The step of migrating unfinished tasks on the lagging computing nodes as subtask slices to idle computing nodes includes: Based on the first feature information of the unprocessed portion in the subtask slice, the unprocessed task is split into at least two finer-grained subtask slices. Obtain the current load metrics of each computing node, and migrate the finer-grained subtask slices to the idle computing node with the lowest current load metrics, wherein the current load metrics include the resource utilization rate of the computing node.

8. The method according to claim 1, characterized in that, After migrating the unprocessed subtask slices on the computing node to other computing nodes for local reallocation, the method further includes: The actual computation time of the subtask slices after local redistribution on other computing nodes is fed back to the link corresponding to the predicted computation time. Based on the actual computation time, update the weight coefficients of the first feature information or adjust the prediction model parameters.

9. The method according to claim 6, characterized in that, The rebalancing does not involve the reordering of completed tasks or global tasks.

10. A distributed parallel processing system for massive network data, characterized in that, The system includes: a communication unit and a processing unit; The communication unit is used to acquire network data to be processed; The processing unit is configured to predict computation time based on the first feature information of the network data, and determine the sharding boundary based on the computation time to obtain multiple data shards, wherein the first feature information is the data feature of the network data; based on the mapping relationship between the first feature information of the data shards and the second feature information of the computing nodes, allocate the data shards to the corresponding computing nodes, wherein the second feature information is the computing power feature of the computing nodes; during the parallel processing of the data shards, in response to the deviation between the actual progress and the estimated progress of the computing nodes exceeding a preset threshold, divide the unprocessed tasks on the computing nodes into subtask slices and migrate them to other computing nodes for local reallocation.

Images