A data processing method of a distributed computer cluster

Abstract

The application discloses a kind of data processing methods of distributed computer cluster, belong to distributed computing technical field.The method includes: obtaining the operation log generated in each computing node in distributed computer cluster in preset period, operation log is executed non-cooperative game characteristic analysis, obtain the resource competition degree matrix of each computing node;Resource competition degree matrix is processed with time-varying evolution graph model construction, generate the dynamic topology entropy sequence of interaction between nodes;Based on dynamic topology entropy sequence, task-data dependency graph is cooperatively constructed in distributed task scheduling queue processing, obtain heterogeneous computing task mapping relationship;Call the metadata index of distributed file system to carry out data fragmentation relocation processing to heterogeneous computing task mapping relationship, generate data routing rules between nodes.The method cooperatively optimizes task scheduling and data routing by game analysis and topology entropy perception, improves cluster data processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of distributed computing technology, specifically to a data processing method for a distributed computer cluster. Background Technology

[0002] Distributed computer clusters complete large-scale data processing tasks through multi-node collaboration. The core challenge lies in how to rationally allocate computing tasks and schedule data access paths among nodes. Existing distributed data processing methods generally employ static load balancing strategies or dynamic scheduling mechanisms based on simple threshold triggers. These methods typically rely on a central scheduler to collect node resource utilization metrics, distribute tasks to relatively idle nodes according to preset weights, and coordinate with the fixed data placement strategy of a distributed file system to complete data access.

[0003] Existing technologies of this kind have significant drawbacks. On the one hand, resource competition between nodes is simplified to an isolated measurement of resource availability, ignoring the strategic competition behavior of nodes running different applications and handling different user requests. Nodes do not passively accept tasks; their actual available computing power, I / O bandwidth, and other resources are affected by factors such as cache preemption and system call conflicts, exhibiting non-cooperative game-theoretic characteristics. Existing methods cannot quantify this competitive relationship, causing scheduling decisions to deviate from the actual node state, leading to task congestion and data locality of destruction. On the other hand, existing methods have weak perception of the dynamics of cluster topology. The interaction relationships between nodes evolve over time. When some nodes experience load fluctuations or network volatility, the static mapping of tasks and data cannot adapt to the dynamically changing topology, resulting in increased data transmission overhead across nodes and prolonged task completion time.

[0004] How to accurately depict the resource competition among nodes during task scheduling and data placement, and how to dynamically perceive changes in cluster topology and adjust task mapping and data routing accordingly, are urgent problems to be solved in current distributed cluster data processing. Summary of the Invention

[0005] This invention provides a data processing method for distributed computer clusters, which aims to extract game-theoretic features reflecting resource competition from node operation logs, and to collaboratively optimize task scheduling and data routing based on the dynamic evolution of the topology, so as to reduce data transmission latency and task execution overhead caused by resource competition and topology changes in the distributed cluster.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a data processing method for a distributed computer cluster, comprising: acquiring operation logs generated by each computing node in the distributed computer cluster within a preset period; performing non-cooperative game characteristic analysis on the operation logs to obtain a resource contention matrix for each computing node; performing time-varying evolution graph model construction processing on the resource contention matrix to generate a dynamic topological entropy sequence for inter-node interactions; performing task-data dependency graph collaborative construction processing on the distributed task scheduling queue based on the dynamic topological entropy sequence to obtain a heterogeneous computing task mapping relationship; calling the metadata index of the distributed file system to perform data fragmentation and relocation processing on the heterogeneous computing task mapping relationship to generate inter-node data routing rules; and performing data forwarding path reconfiguration processing on the operating system kernel of the distributed computer cluster according to the inter-node data routing rules to complete the data processing of the distributed computer cluster.

[0007] This invention combines non-cooperative game modeling with topological entropy time-series analysis to perceive the dynamic evolution of node resource competition and interaction structures, enabling task scheduling and data layout to adapt to real-time changes in cluster operating status, thereby improving the locality of task execution and overall throughput in distributed systems.

[0008] As a technical solution of this invention, when performing non-cooperative game-theoretic characteristic analysis on the operation log, the system call sequence is first extracted from the operation log to obtain the system call frequency vector of each computing node; the resource contention strategy space is constructed from the system call frequency vector to obtain the inter-node strategy game matrix; then, the Nash equilibrium point is solved from the inter-node strategy game matrix to generate the resource contention degree matrix of each computing node. This solution can accurately quantify the competitive relationship between nodes in resource usage from the underlying system behavior, providing a stable quantitative basis that reflects the real load conflict for subsequent topology analysis.

[0009] Preferably, when constructing a time-varying evolution graph model for the resource competition degree matrix, the resource competition degree matrix is ​​subjected to time window sliding slicing to obtain a sequence of competition degree sub-matrices for consecutive time slices; each competition degree sub-matrix is ​​then mapped using a graph structure adjacency matrix to generate a time-varying graph snapshot of inter-node interactions; and the time-varying graph snapshots are then processed to calculate topological entropy values ​​to generate a dynamic topological entropy sequence of inter-node interactions. Through this process, fluctuations in competition intensity and node interaction patterns are condensed into a measurable entropy sequence, which can sensitively capture sensitive periods when the cluster's internal structure becomes unstable or performance bottlenecks occur.

[0010] Furthermore, when constructing the task-data dependency graph collaboratively for the distributed task scheduling queue based on the dynamic topological entropy sequence, the dynamic topological entropy sequence is subjected to entropy mutation point detection processing to identify abnormal fluctuation ranges in the topological entropy of computing nodes. Within these abnormal fluctuation ranges, the task priority of the distributed task scheduling queue is reordered to obtain an adjusted task scheduling order. The adjusted task scheduling order is then used to construct a task-data dependency graph, generating a heterogeneous computing task mapping relationship. This scheme enables the task scheduling order to dynamically respond to changes in the topology, proactively adjusting the task execution order during windows of abnormal fluctuations in cluster interaction relationships, thus avoiding the exacerbation of resource conflicts due to blind scheduling.

[0011] As a preferred implementation of priority reordering, the data locality score and computational dependency depth score of each task are extracted from the distributed task scheduling queue. A weighted sum of these scores is then performed to generate a comprehensive scheduling priority for the task. This comprehensive scheduling priority is used as the sorting criterion to reorder the distributed task scheduling queue, resulting in an adjusted task scheduling order. This reordering mechanism balances the need for proximity-based data access with the complexity of computational paths, ensuring that high-priority tasks receive computing resources matching the data distribution within topological anomaly regions, thus reducing cross-node data migration overhead.

[0012] When performing data sharding relocation processing on heterogeneous computing task mapping relationships, the process involves parsing the task computation node identifiers and data storage node identifiers within the mapping relationships; utilizing the metadata index of the distributed file system to perform statistical processing on data block access frequencies, generating a distribution map of hot data blocks; and based on this distribution map, remapping the data shards to target nodes to generate data routing rules between nodes. This process combines task mapping relationships with real-time data access frequency, proactively guiding hot data towards nodes with intensive computing needs, thus reducing remote access latency for data reading.

[0013] Preferably, when remapping data shards to target nodes based on a hot data block distribution map, the hot data block distribution map is sorted by access popularity to obtain a priority sequence of hot data shards; an iterative data migration strategy is generated from the priority sequence of hot data shards to obtain a set of migration target nodes for the data shards; and a data forwarding path table between nodes is constructed based on the set of migration target nodes to generate data routing rules between nodes. Through orderly iterative data migration decisions, the data layout can be gradually optimized while ensuring cluster stability, forming efficient data forwarding paths.

[0014] When performing kernel-level reconfiguration based on inter-node data routing rules, these rules are parsed into kernel-level routing table entry update instructions. These instructions are then used to refresh the routing cache of the operating system's network protocol stack, resulting in an updated forwarding path table. Finally, using this updated table as a reference, forwarding paths are selected for data packets from the distributed computer cluster, completing the data processing for the cluster. This technique directly applies application-layer-generated routing rules to the operating system kernel, enabling data packets to be forwarded along the optimal path at the protocol stack level, eliminating the overhead of frequent switching between user mode and kernel mode.

[0015] Preferably, when generating kernel-level routing table entry update instructions, the target node IP address and routing priority fields of the inter-node data routing rules are extracted to obtain routing update parameter pairs; the routing update parameter pairs are then converted to kernel routing table entry formats to generate kernel-level routing table entry update instructions. This allows routing updates to be directly converted into standard kernel-executable instructions, ensuring the atomicity and consistency of the configuration process.

[0016] The technical effects and advantages provided by the present invention in the above technical solution are as follows: By obtaining the system call frequency vector of each computing node from the operation log, and constructing a strategy game matrix between nodes based on this vector, the Nash equilibrium point is solved to generate a resource contention matrix. This approach differs from simply statistically analyzing CPU or memory usage; instead, it captures resource contention strategies from the actual system call sequences executed by nodes, quantifying the implicit competitive relationships between nodes into a computable game equilibrium result. This allows the resource contention matrix to accurately reflect the interactive state of nodes in a multi-tasking environment, providing a more accurate description of node states for task scheduling.

[0017] The resource contention matrix is ​​sliced ​​using a sliding time window to generate a sequence of contention sub-matrices for consecutive time slices. These sub-matrices are then mapped to time-varying graph snapshots, and the topological entropy of each snapshot is calculated, forming a dynamic topological entropy sequence. By detecting entropy abrupt change points in this sequence, time intervals where node interaction topology changes drastically can be identified. Within these intervals, a weighted comprehensive scheduling priority reordering is performed by combining task data locality scores and computational dependency depth scores. The task-data dependency graph is then constructed according to the adjusted order, yielding the heterogeneous computing task mapping relationship. This topological entropy-aware scheduling adjustment mechanism tightly couples task mapping with the dynamic evolution of cluster topology, avoiding cross-node jumps in data access caused by using the original scheduling decisions during periods of topological instability. This reduces data migration and accelerates task execution.

[0018] Based on the identifiers of compute nodes and data storage nodes in the task mapping relationship, the distribution of hot data blocks is statistically analyzed using distributed file system metadata indexes. Then, iterative migration target node selection is performed according to the priority sequence of hot data fragments. Finally, data routing rules between nodes are constructed and converted into kernel-level routing table entry update instructions, refreshing the routing cache of the network protocol stack. This process coordinates data fragment relocation with task mapping, taking effect directly on the kernel forwarding plane. This allows data packets to select forwarding paths according to the updated forwarding path table, avoiding control delays caused by frequent switching between user space and kernel space, and quickly completing data forwarding path adaptation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a flowchart of a distributed computer cluster data processing method; Figure 2 It is a flowchart for constructing the mapping relationship of heterogeneous computing tasks based on dynamic topological entropy sequences; Figure 3 This is a schematic diagram of the dynamic topological entropy sequence and entropy value mutation points of interaction between nodes in a distributed computer cluster. Figure 4 This is a probability density distribution map of task data locality score and computational dependency depth score; Figure 5 This is a diagram showing the sorting and frequency thresholds of access frequencies for hot data blocks; Figure 6 This is a graph showing the relationship between the target node's kernel routing metric and forwarding priority. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See Figure 1This invention provides a data processing method for a distributed computer cluster, comprising: acquiring operation logs generated by each computing node in the distributed computer cluster within a preset period; performing non-cooperative game characteristic analysis on the operation logs to obtain a resource contention matrix for each computing node; performing time-varying evolution graph model construction processing on the resource contention matrix to generate a dynamic topological entropy sequence for inter-node interactions; performing task-data dependency graph collaborative construction processing on a distributed task scheduling queue based on the dynamic topological entropy sequence to obtain a heterogeneous computing task mapping relationship; calling the metadata index of the distributed file system to perform data fragmentation and relocation processing on the heterogeneous computing task mapping relationship to generate data routing rules between nodes; and performing data forwarding path reconfiguration processing on the operating system kernel of the distributed computer cluster according to the data routing rules between nodes to complete the data processing of the distributed computer cluster.

[0023] Example 1: In practice, the operation logs are processed to extract system call sequences, resulting in a system call frequency vector for each computing node. The operation logs originate from the running records of each computing node in the distributed computer cluster within a preset period. Each operation log entry contains a timestamp field, a node identifier field, a system call type identifier field, and a call parameter field. The operation logs are grouped according to the node identifier field, grouping operation logs from the same computing node into a subset. For each computing node's log subset, each operation log entry is traversed, the system call type identifier field is extracted, and the total number of occurrences of each system call type identifier within the preset period is counted. The statistical results are organized into a vector according to the preset lexicographical order of the system call type identifiers. The dimension of the vector equals the total number of system call types supported by the distributed computer cluster's operating system, and the value of one dimension represents the frequency of the corresponding system call type identifier being called on that computing node. This vector is denoted as the system call frequency vector for that computing node. The system call frequency vectors of all computing nodes form a system call frequency vector set.

[0024] The system call frequency vector is processed to construct a resource contention strategy space, resulting in a strategy game matrix between nodes. Each computing node's system call frequency vector represents its usage pattern of system resources such as computing, storage, and network. Each system call frequency vector is mapped to an independent strategy point in the resource contention strategy space. For any two different computing nodes, a resource conflict metric is calculated between their system call frequency vectors. This metric reflects the degree of overlap and exclusion in the system resource usage patterns of the two computing nodes. The system call frequency vector of computing node i is designed as follows: The system call frequency vector of computing node j is The resource conflict metric is calculated using the complement of the cosine similarity, as shown in the formula: ,in Representing vectors with vector The inner product, Representing vectors of Norm, Representing vectors of Norm. Calculated The values ​​are in the range [0,1], and when the directions of the two system call frequency vectors are completely orthogonal, A value of 1 indicates perfect competition; when the frequency vectors of two system calls are collinear and in the same direction... A value of 0 indicates no competition. In practice, the resource conflict metric can optionally be the absolute value of the Pearson correlation coefficient or the normalized Euclidean distance; however, this embodiment uses the complement of the cosine similarity as an example. The resource conflict metrics between all pairs of computing nodes are arranged in order of node identifier, forming an N×N matrix, where N is the total number of computing nodes in the distributed computer cluster. The off-diagonal elements of this matrix are the aforementioned resource conflict metrics, and the diagonal elements are set to 0. This matrix is ​​the inter-node strategy game matrix, where the element in the i-th row and j-th column represents the intensity of the strategy conflict between computing node i and computing node j in resource contention.

[0025] The inter-node strategy game matrix is ​​processed to find the Nash equilibrium point, generating the resource competition degree matrix for each computing node. The inter-node strategy game matrix is ​​viewed as the payoff matrix of a symmetric non-cooperative game involving all computing nodes. Each computing node is a player, and its strategy set corresponds to a mixed strategy in the resource competition strategy space. Calculating the mixed strategy Nash equilibrium point of the symmetric game involves finding a probability distribution vector such that unilateral deviation by any computing node under this distribution will not result in a higher payoff. In some embodiments, the solution approach of evolutionary stable strategies using the replication dynamics equations is adopted. The right principal eigenvector of the inter-node strategy game matrix is ​​calculated, and this right principal eigenvector is normalized to represent the equilibrium mixed strategy. Let the inter-node strategy game matrix be denoted as... , The dimension is N×N, elements in Let this be the resource conflict metric between compute node i and compute node j obtained above. Solve the equation. ,in For matrix The spectral radius, i.e., the eigenvalue with the largest modulus, To and The corresponding right eigenvector, Let be an N-dimensional column vector, where the i-th component corresponds to the resource contention weight of node i in the equilibrium state. For the vector... conduct Norm normalization makes the sum of all components of a vector equal to 1. The normalized vector is still denoted as . Generate the resource contention matrix for each computing node. The calculation formula is as follows:

[0026] in, This is a resource competitiveness matrix with N×N dimensions; The column vector obtained by the above normalization, The i-th component represents the probability of the balanced mixing strategy for computing node i; for The transpose row vector; express and The outer product operation. Resource contention matrix. elements in ,in For vectors The i-th component, For vectors The j-th component, This quantitatively describes the degree of resource competition between computing node i and computing node j under Nash equilibrium. Thus, the resource competition matrix for each computing node is obtained.

[0027] Example 2: In practical implementation, the process of constructing a time-varying evolution graph model for the resource contention matrix to generate a dynamic topological entropy sequence of inter-node interactions is as follows: The resource contention matrix is ​​processed by time window sliding slicing to obtain a sequence of contention sub-matrixes for consecutive time slices. In practical implementation, a preset period is divided into multiple time windows of equal length. The length of the time window is determined based on the generation rate of the operation logs and the time granularity of the distributed computer cluster topology changes. For example, the length of the time window is set to one-tenth of the preset period. If the preset period is T seconds, then the duration of a time window is T / 10 seconds. The sliding step size is set to half the length of the time window. Starting from the beginning of the preset period, resource contention data within each time window is sequentially extracted according to the sliding step size, forming a sequence of contention sub-matrixes. For each time window, obtain the resource contention matrix generated by each computing node within that time window. If the resource contention matrix is ​​generated by merging throughout the entire cycle, recalculate the window-level resource conflict metric using the system call frequency vector of each computing node within the time window, construct the window-level inter-node strategy game matrix, and solve for the window-level Nash equilibrium point to obtain the contention sub-matrix corresponding to that time window. Arrange the contention sub-matrices corresponding to all time windows in chronological order according to the start time of the window to generate a sequence of contention sub-matrices.

[0028] For each competition degree submatrix, a graph structure adjacency matrix mapping process is performed to generate a time-varying graph snapshot of the interactions between nodes. For each competition degree submatrix in the sequence, each computing node in the distributed computer cluster is used as a vertex of the graph structure, and the value of the corresponding element in the competition degree submatrix is ​​used as the weight of the edge between vertices to construct a weighted undirected graph. When the value of the element in the u-th row and v-th column of the competition degree submatrix is ​​greater than a preset connection threshold, an edge is established between vertex u and vertex v, and the value of this element is used as the weight of the edge; when the value of the element is less than or equal to the preset connection threshold, there is no edge between vertex u and vertex v. The preset connection threshold is set according to the distribution characteristics of resource competition degree. In specific implementation, the mean of the non-zero elements in the competition degree submatrix is ​​taken as the preset connection threshold. Each weighted undirected graph is converted into an adjacency matrix representation. The row and column indices in the adjacency matrix correspond to the computing node identifiers, and the matrix elements are the weight values ​​of the edges between the corresponding vertices. Vertices without edges have a value of zero for the corresponding matrix element. Each competition submatrix generates an adjacency matrix, which is a snapshot of the time-varying graph. All time-varying graph snapshots constitute a sequence of time-varying graph snapshots.

[0029] The topological entropy of time-varying graph snapshots is calculated to generate a dynamic topological entropy sequence of interactions between nodes. For each time-varying graph snapshot in the sequence, the spectral entropy of that snapshot is calculated as the topological entropy value. The adjacency matrix of a time-varying graph snapshot is denoted as... , The dimension is M×M, where M is the total number of computing nodes in the distributed computer cluster. Calculate the adjacency matrix. Laplace matrix Laplace matrix Defined as ,in The degree matrix, the degree matrix It is a diagonal matrix, degree matrix diagonal elements equal to adjacency matrix No. The sum of all elements in a row, i.e., the sum of the elements in the computed node. The sum of the weights of all connected edges. All off-diagonal elements are 0. Calculate the Laplacian matrix. From the eigenvalues, we obtain M eigenvalues, denoted as the eigenvalue of the eigenvalue. The feature values ​​are , The value of is in the range of 1 to M. The formula for calculating the topological entropy is:

[0030] in, Represents the topological entropy value of a snapshot of a time-varying graph; Represents the Laplace matrix The There are 10 eigenvalues, which are arranged in descending order of their numerical values. Represents the sum of all eigenvalues. For the index variables that iterate from 1 to M in the summation; Indicates the first Normalized weights for each feature value. The value of is greater than or equal to 0. A larger value indicates a more irregular topology in the time-varying graph snapshot and a higher degree of uncertainty in inter-node interactions. The above topology entropy calculation process is performed on each time-varying graph snapshot in the sequence. The calculated topology entropy values ​​are then arranged according to the time order of the time-varying graph snapshots to generate a dynamic topology entropy sequence of inter-node interactions. Each element in the dynamic topology entropy sequence corresponds to the entropy value of the topology of node interactions in a distributed computer cluster within a specific time window.

[0031] See Figure 3In the figure, the horizontal axis represents the time window number, and the vertical axis represents the topological entropy value of the snapshot of the time-varying graph. The blue solid curve represents the dynamic topological entropy sequence, which shows the trend of entropy value change in the interaction topology of the distributed computer cluster nodes within a continuous time window. The green dashed curve is the moving average difference value, reflecting the first-order difference stability of the topological entropy sequence within a local time range. The points marked with red crosses in the figure represent candidate entropy value abrupt change points. These abrupt change points correspond to time windows in the dynamic topological entropy sequence where the entropy value changes abnormally drastically, reflecting significant changes in the interaction topology of the computing nodes at these moments.

[0032] As shown in the figure, the dynamic topological entropy sequence fluctuates between 1.0 and 4.5, exhibiting several stages of fluctuation. Within time window numbers approximately 100 to 120, the topological entropy value shows a significant increase, peaking at approximately 3.5, accompanied by red candidate mutation points, indicating a significant reconfiguration of the node interaction topology or a change in resource contention within the distributed computer cluster during this stage. Subsequently, the entropy value decreases and stabilizes, remaining within the range of approximately 1.5 to 2.5, reflecting a relatively stable topological structure. Around window numbers 380 to 400, the topological entropy value shows an even higher peak, reaching approximately 4.6, also accompanied by multiple candidate mutation points, indicating a significant increase in the complexity and uncertainty of node interactions during this period, potentially corresponding to sudden resource contention within the system or drastic adjustments in inter-node interaction patterns.

[0033] Example 3: In specific implementation, please refer to Figure 2 This method performs entropy mutation detection on a dynamic topological entropy sequence to identify abnormal fluctuation ranges in topological entropy of computational nodes. The dynamic topological entropy sequence is a sequence of topological entropy values ​​arranged in time window order, with each entropy value corresponding to a snapshot of the time-varying graph within a time window. The entropy mutation detection uses a sliding window differencing method. A detection window length K is set, where K is the integer obtained by dividing the total length of the dynamic topological entropy sequence by the estimated frequency of topological change events within a preset period. If the estimated frequency is 5 to 10 times, K is rounded up to one-tenth of the total length of the dynamic topological entropy sequence. The first-order difference mean and standard deviation of the topological entropy values ​​within the detection window are calculated sequentially. For the m-th topological entropy value within the detection window, if the absolute value of its first-order difference exceeds three times the standard deviation of the first-order difference mean within the detection window, the time corresponding to that value is marked as a candidate mutation point. The time intervals containing consecutive candidate mutation points are merged into a continuous interval to obtain the abnormal fluctuation range of topological entropy. The abnormal fluctuation range of topological entropy represents the period during which the interaction topology between computing nodes in a distributed computer cluster changes significantly, and the resource competition relationship between nodes deviates from the stable state during this period.

[0034] Within the abnormal fluctuation range of topological entropy, the data locality score and computational dependency depth score of a task are extracted from the distributed task scheduling queue. The distributed task scheduling queue maintains a set of tasks to be scheduled. Each task includes a task identifier, a list of required input data block identifiers, a description of the task's computational logic, and a current priority field. The data locality score characterizes the spatial proximity between the data required by the task and the existing data caches on each computing node. For a task to be scheduled, the data locality score is calculated by counting the cache hits on the computing nodes covered by each data block identifier in the list of required input data block identifiers within the abnormal fluctuation range of topological entropy. If the data block corresponding to a data block identifier is already cached in the local storage of a computing node at the start of the current abnormal fluctuation range of topological entropy, it is considered a cache hit. The original data locality score of the task is obtained by dividing the number of cache-hit data blocks by the total number of required input data blocks, with a value ranging from [0,1]. The computational dependency depth score characterizes the hierarchical position of the task in the task dependency graph. The task dependency graph is constructed based on the data dependencies between tasks. If task B requires the output data of task A as input, then task B depends on task A. The steps for calculating the dependency depth score are as follows: Randomly select a root task in the task dependency graph that has no upstream dependencies and set its computational dependency depth score to 1.0; for non-root tasks, the computational dependency depth score is equal to the maximum value of the computational dependency depth scores of all its direct upstream tasks plus 1.0. Within a topological entropy anomaly fluctuation range, all tasks waiting to be scheduled and whose expected execution time overlaps with the topological entropy anomaly fluctuation range are retrieved from the distributed task scheduling queue. For each retrieved task, the aforementioned data locality score and computational dependency depth score are extracted.

[0035] A weighted summation of the data locality score and computational dependency depth score is performed to generate a comprehensive scheduling priority for the task. The data locality score of a task is denoted as... The dependency depth score is calculated as follows: Comprehensive scheduling priority of tasks The calculation formula is:

[0036] in, This indicates the overall scheduling priority of the task. A larger value indicates a higher scheduling priority for the task; The score represents the data locality of the task. ; This indicates that the task's computation depends on the depth score. ; For data locality weighting coefficients, To depend on depth weighting coefficients, and satisfy , and The values ​​are all greater than 0 and less than 1; As it depends on depth amplitude modulation parameters, The value of determines the degree of nonlinear compression of the contribution of dependency depth score to comprehensive scheduling priority. In specific implementation, Set to 0.6, Setting it to 0.4 makes the overall scheduling priority take into account the data locality of tasks while also taking into account the impact of computation dependency depth on the scheduling order. Set to 0.5 when calculating dependency depth score When it is 2.0, It can map the multi-layer dependency depth to a suitable value in the (0,1) interval, so that the impact of the computation dependency depth on the priority shows a marginal decreasing trend as the depth increases.

[0037] The distributed task scheduling queue is reordered based on the overall scheduling priority to obtain the adjusted task scheduling order. All tasks extracted within the abnormal topological entropy fluctuation range are then sorted according to their overall scheduling priority. The values ​​are sorted from largest to smallest. Tasks with the same overall scheduling priority are then further sorted according to the original task submission timestamps. After sorting, the sorted task order replaces the original position of the corresponding task in the distributed task scheduling queue, while maintaining the relative order of other tasks not affected by the abnormal fluctuation range of topological entropy, thus obtaining the adjusted task scheduling order.

[0038] The adjusted task scheduling order is processed to construct a task-data dependency graph, generating heterogeneous computing task mapping relationships. The task-data dependency graph construction process takes the adjusted task scheduling order as input and maps tasks to computing nodes in the distributed computer cluster one by one according to the scheduling order. During mapping, the list of input data block identifiers required by the task is read, and the metadata index of the distributed file system is queried to obtain the storage node identifier corresponding to each data block identifier. If the computing node corresponding to the storage node identifier of a data block is inconsistent with the computing node to which the task is mapped, a data dependency edge is added to the task-data dependency graph from the storage node identifier to the task computing node identifier. Simultaneously, task dependency edges are established between adjacent tasks in the task scheduling order that have a direct data transfer relationship. The final generated task-data dependency graph is a directed acyclic graph (DAG), where the vertices include task computing node identifiers and data storage node identifiers, and the directed edges represent data flow directions. The mapping relationships between the vertices of this DAG constitute the heterogeneous computing task mapping relationships. The heterogeneous computing task mapping relationship is stored in the form of a list of key-value pairs. Each key-value pair consists of a computing task identifier and a computing node identifier, indicating that the computing task is executed by the mapped computing node.

[0039] See Figure 4 In the figure, the horizontal axis represents the score value, and the vertical axis represents the probability density of the corresponding score value. The figure shows two probability density curves: the blue solid line represents the probability density distribution of the data locality score, and the red dashed line represents the probability density distribution of the computational dependency depth score. The data locality score ranges from [0,1]. Its probability density curve shows a clear peak in the 0-1 interval, with a maximum peak of approximately 1.55, corresponding to score values ​​between approximately 0.3 and 0.5, indicating that the data locality scores for most tasks are concentrated in the low to medium value range. The computational dependency depth score has a wider range, with the horizontal axis extending from approximately 1 to 15. Its probability density curve exhibits a right-skewed long-tail distribution. Initially, the probability density is highest in the score value range of 1-2, approximately 0.25, and then gradually decreases as the score value increases. The decrease is relatively gentle in the 1-5 interval, and the probability density value approaches zero after exceeding 5. The probability density distributions of the two scores reflect that within the abnormal fluctuation range of topological entropy, the data locality scores of the tasks are mainly concentrated at a low to medium level, while the computational dependency depth scores show a diversified and generally low trend.

[0040] Example 4: In practical implementation, the process of calling the distributed file system's metadata index to perform data fragmentation and relocation processing on heterogeneous computing task mapping relationships and generate data routing rules between nodes is as follows: Parsing the task computing node identifiers and data storage node identifiers in the heterogeneous computing task mapping relationships. The heterogeneous computing task mapping relationships are stored as a list of key-value pairs. Each key-value pair contains a computing task identifier and a task computing node identifier, indicating that the computing task is assigned to the computing node indicated by that task computing node identifier for execution. Traversing each key-value pair in the heterogeneous computing task mapping relationships, all task computing node identifiers are extracted to form a set of task computing node identifiers. For each computing task identifier in the heterogeneous computing task mapping relationships, the list of input data block identifiers required for the task corresponding to that computing task identifier is obtained from the distributed task scheduling queue. The storage location corresponding to each data block identifier is queried using the distributed file system's metadata index, and the data storage node identifier is parsed from the storage location. All data storage node identifiers corresponding to all tasks are aggregated and deduplicated to form a set of data storage node identifiers. After parsing, each element in the task computation node identifier set represents a computation node that carries a computation task, and each element in the data storage node identifier set represents a data storage node that stores the task input data block.

[0041] Data block access frequency statistics are performed using the metadata index of the distributed file system (DFS) to generate a hot data block distribution map. The DFS metadata index maintains access records corresponding to each data block identifier, including an access timestamp field and a source node identifier field. In the implementation, a time window for access frequency statistics is set, with the start and end times coinciding with the aforementioned abnormal fluctuation range of topological entropy. All access records within the time window are retrieved from the DFS metadata index, grouped by data block identifier, and the total number of times each data block identifier is accessed within the time window is counted. The total number of accesses is divided by the length of the time window to obtain the access frequency of each data block identifier. The data block identifier, its access frequency, and the identifier of the current storage node where the data block resides are organized into a triplet; the set of all triplets constitutes the hot data block distribution map. In the hot data block distribution map, data blocks with access frequencies exceeding a set frequency threshold are marked as candidate hot data blocks. The frequency threshold is set to twice the median access frequency of all data blocks within the time window; when the access frequency corresponding to the median is zero, the frequency threshold is set to 0.1 times per time unit.

[0042] The hot data block distribution map is processed by data sharding and access popularity sorting to obtain a priority sequence of hot data shards. Triples whose data block identifiers belong to candidate hot data blocks are selected from the hot data block distribution map. The data block identifier and access frequency of each selected triplet are extracted to generate an access frequency list. The access frequency list is sorted in descending order of access frequency value. For triples with the same access frequency, they are sorted in ascending order of data block identifier lexicographical order. The resulting ordered list of data block identifiers is the priority sequence of hot data shards. In the priority sequence of hot data shards, the higher the ranking of the data shard identifier, the higher its access popularity, and it should be processed first during the data shard relocation process.

[0043] An iterative data migration strategy generation process is performed on the priority sequence of hot data shards to obtain a set of migration target nodes for each data shard. The iterative data migration strategy generation process handles each hot data shard sequentially according to its priority sequence. For the currently processed hot data shard, the source storage node identifier of that hot data shard at the current moment is obtained. All task computation node identifiers that need to read this hot data shard as input data are searched from the task-data dependency graph, and these task computation node identifiers are compiled into a potential target node list. If the potential target node list contains only one task computation node identifier, that task computation node identifier is directly determined as the migration target node for that hot data shard. If the potential target node list contains at least two task computation node identifiers, the migration fit score corresponding to each potential target node identifier needs to be calculated. The currently processed hot data shard is denoted as a shard. A task computation node in the potential target node list is identified as a node. Migration fit score The calculation formula is:

[0044] in, This indicates that the data will be split into pieces. Migrate to node The transfer fit score The larger the value, the more likely it is to be a node More suitable for receiving fragments ; For data affinity weighting coefficients, The value range is greater than 0 and less than 1; The load balancing penalty weight coefficient, The value range is greater than 0 and less than 1, and ; In the task-data dependency graph, sharding represents... With nodes The sum of the weights of the task-data dependency edges between them. The value depends on the partition. And at the node The number of tasks executed; This indicates a dependency on fragmentation. The total number of all tasks; Represents a node The sum of the data size of the migration data fragments allocated in the current iteration step, in megabytes. If the current iteration step has not yet migrated the data to the node... Assign any migration data shards, then The value is 0; For nodes Migration reception capacity, Value is a node The product of total available storage space and a preset reservation ratio, where the preset reservation ratio is set to 0.2. In practice, Set to 0.75. The value is set to 0.25. This setting is based on the principle that migration fit scoring should be primarily driven by data affinity, while allowing for moderate adjustment of load balancing factors. The calculation is performed on the task computation node identifier corresponding to each task in the potential target node list. Value, select The task computation node identifier with the largest value is used as the partition. The migration target node. After determining the migration target node for a hot data shard, update the migration target node's... The value will be the current shard. The amount of data accumulated to Then, the next hot data shard is processed. After all hot data shards in the priority sequence have been processed, the identifier of each hot data shard is paired with its migration target node identifier to generate a set of migration target nodes for the data shards.

[0045] A data forwarding path table is constructed based on the set of target nodes to be migrated, generating data routing rules between nodes. Each element in the set of target nodes contains a data shard identifier and a target node identifier. For each element, the source storage node identifier for that data shard identifier is queried from the metadata index of the distributed file system, constructing a data forwarding path record. The fields of the data forwarding path record include the source storage node identifier, the target node identifier, the data shard identifier, the forwarding priority, and the effective timestamp. The value of the forwarding priority field is determined by the order of the data shard in the priority sequence of hot data shards; data shards that are ranked higher are assigned higher forwarding priority. The effective timestamp field is set to the current system time plus a preset delay bias value, which is set to twice the average inter-node message round-trip latency in the distributed computer cluster. All data forwarding path records are aggregated into a data forwarding path table. Each record in the data forwarding path table is aggregated according to the target node identifier to generate a list of source node identifiers and forwarding conditions corresponding to each target node identifier, forming a series of inter-node data routing rules. Each inter-node data routing rule includes a target node identifier, a list of source storage node identifiers, a list of data shard identifiers, and a forwarding priority. Inter-node data routing rules are used to guide the distributed file system and cluster network layer to complete the directed forwarding of data shards from the source storage node to the migration target node.

[0046] See Figure 5 In the graph, the horizontal axis represents the data block ranking, and the vertical axis represents the access frequency of the corresponding data block. The vertical axis uses a logarithmic scale to highlight the order-of-magnitude differences in frequency. The solid blue line indicates that the access frequency of each data block decreases as the ranking increases, gradually decreasing from several thousand times per unit time to approximately 10 times per unit time, reflecting a clear long-tail distribution of data access. The dashed red line represents the frequency threshold, set at twice the median access frequency of all data blocks within the time window, specifically approximately 50 times per unit time. The dotted green line represents the median access frequency, located at approximately 25 times per unit time. The blue access frequency curve intersects the red frequency threshold line at approximately 170th place, meaning that the access frequency of the top 170 data blocks is higher than the frequency threshold and is marked as candidate hot data blocks. Subsequent rankings show access frequencies below the frequency threshold, classifying them as non-hot data blocks.

[0047] Example 5: In practical implementation, the data forwarding path reconfiguration process of the operating system kernel of the distributed computer cluster is performed according to the inter-node data routing rules. The data processing process of the distributed computer cluster is as follows: The target node IP address and routing priority fields are extracted from the inter-node data routing rules to obtain routing update parameter pairs. Each inter-node data routing rule contains a target node identifier, a list of source storage node identifiers, a list of data fragment identifiers, and a forwarding priority field. The target node identifier is read from the inter-node data routing rule, and the node management service of the distributed computer cluster is used to resolve the target node identifier to the target node IP address by querying a pre-maintained node identifier-IP address mapping table. Simultaneously, the value of the forwarding priority field is read from the inter-node data routing rule. This forwarding priority field is an integer, ranging from 1 to the total number of all inter-node data routing rules; the smaller the value, the higher the forwarding priority. The target node IP address and the forwarding priority field value are combined into a tuple; each tuple represents a routing update parameter pair. All inter-node data routing rules are traversed to extract the set of routing update parameter pairs.

[0048] The routing update parameter pairs undergo kernel routing table entry format conversion to generate kernel-level routing table entry update instructions. For each routing update parameter pair in the set, the following conversion operations are performed: Determine the target node IP address and forwarding priority field values ​​contained in a routing update parameter pair. Fill the target node IP address into the target network field of the kernel routing table entry, and set the subnet mask field to 255.255.255.255, indicating a host route. Set the next-hop gateway address field to the IP address corresponding to the first source storage node in the source storage node identifier list of the inter-node data routing rules. When the source storage node identifier list contains at least two source storage nodes, generate a kernel routing table entry record for each source storage node to form an equal-cost multipath route. Set the network interface field to the name of the local network interface that passes through the corresponding next-hop gateway address. The routing metric field is calculated based on the forwarding priority field value, using the following formula:

[0049] in, Indicates from the first Each route update parameter corresponds to a generated kernel route metric value. It is an integer, and its value range is 1. ; This is a constant representing the upper limit of the kernel routing metric. The value of is determined based on the kernel routing protocol type of the distributed computer cluster operating system. In specific implementations, the operating system kernel uses the Linux kernel, and the upper limit of the routing metric value in the Linux kernel is 65535. The value is 65535; Indicates from the first The value of the forwarding priority field extracted from each route update parameter pair. The range of values ​​is , This represents the maximum value of the forwarding priority field in the data routing rules among all nodes. Iterate through the set of route update parameters and take the maximum value. Value obtained; Map the forwarding priority field value to Within the interval, the route update parameter pair with the higher forwarding priority is the one with the higher weight value (i.e., The smaller the value, the larger the weight, thus yielding a smaller result. The kernel prioritizes route entries with smaller metric values ​​during routing. This involves combining the target network field, subnet mask field, next-hop gateway address field, network interface field, and the calculated metric value. The value is filled into the kernel routing table entry structure, generating a kernel-level routing table entry update instruction. A kernel-level routing table entry update instruction contains the opcode for adding the routing entry and the complete routing table entry data.

[0050] The kernel-level routing table entry update instructions are used to refresh the routing cache of the operating system's network protocol stack, resulting in an updated forwarding path table. All generated kernel-level routing table entry update instructions are sent one by one to the operating system kernel's network protocol stack via the network routing management socket interface provided by the operating system kernel. Upon receiving each kernel-level routing table entry update instruction, the network protocol stack updates the corresponding routing entry in the kernel routing information base and marks the affected forwarding information base cache entries as invalid. After all kernel-level routing table entry update instructions have been sent, a cache refresh instruction is sent to the operating system kernel's forwarding information base, triggering the kernel to traverse all forwarding information base cache entries marked as invalid. The kernel recalculates the forwarding path based on the routing entries in the kernel routing information base, performing longest prefix matching and metric comparison during the calculation process, and repopulating the forwarding information base cache entries. After the cache refresh is complete, all routing entries related to data fragment relocation are read from the operating system kernel's forwarding information base, and these routing entries are organized into the updated forwarding path table. Each entry in the updated forwarding path table contains the destination node IP address, the next-hop IP address list, the outgoing interface, and the metric.

[0051] Based on the updated forwarding path table, data packets from the distributed computer cluster are processed for forwarding path selection, thus completing the data processing for the distributed computer cluster. When any computing node in the distributed computer cluster needs to send a data packet to the target node where the migrated data fragment is located, the operating system kernel protocol stack of that computing node extracts the destination IP address from the data packet and uses the destination IP address as the lookup key to perform a longest prefix match search in the updated forwarding path table. After matching the updated forwarding path table entry, a next-hop IP address is selected from the next-hop IP address list of the entry according to the hash algorithm of equivalent multi-path selection, and the data packet is sent out from the corresponding outgoing interface. The input of the hash algorithm is the source IP address field, destination IP address field, and transport layer port number field of the data packet, to ensure that data packets belonging to the same data flow select the same next-hop path. After the data packet has undergone forwarding path selection processing, it arrives at the target node, completing the redirection transmission of the data fragment within the distributed computer cluster, realizing the reconfiguration of the data forwarding path based on the data routing rules between nodes, and completing the data processing for the distributed computer cluster.

[0052] See Figure 6 In the figure, the horizontal axis represents the forwarding priority field value in the data routing rules between nodes, ranging from 0 to 255, with smaller values ​​indicating higher forwarding priority; the vertical axis represents the routing metric value of the corresponding target node in the operating system kernel, ranging from 0 to approximately 65535. The figure shows the distribution of kernel routing metrics for two target nodes A and B, with target node A represented by blue dots and target node B by red squares.

[0053] As shown in the figure, the kernel routing metric values ​​of target node A and target node B exhibit a near-linear decreasing trend with the increase of the forwarding priority field value, and their metric curves highly overlap, indicating that the routing entries of both target nodes are calculated according to the same mapping rule. The metric value is calculated based on the following formula:

[0054] in, Forwarding priority field value, The maximum priority value is 255. The forwarding priority field value in this diagram is... The maximum value of the corresponding metric is close to 65535, and the minimum value is close to 0.

[0055] In the diagram, when the forwarding priority value is close to 0, the kernel routing metric reaches its maximum value of approximately 65535, indicating that the routing entry has the lowest priority at this time. As the forwarding priority field value gradually increases, the kernel routing metric gradually decreases, and the priority gradually increases. Finally, when the forwarding priority value is close to 255, the kernel routing metric approaches 0, indicating that the routing entry has the highest priority.

[0056] The trend of the curve is consistent with the design idea of ​​using kernel routing metric priority mapping described in Example 5. That is, the forwarding priority field value is inversely proportional to the kernel routing metric value, which ensures that the operating system kernel prioritizes the routing entries with high forwarding priority (small value) when selecting routes, and realizes effective data forwarding path reconfiguration based on inter-node data routing rules.

[0057] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data processing method for a distributed computer cluster, characterized in that, include: Obtain the operation logs generated by each computing node in the distributed computer cluster within a preset period, perform non-cooperative game characteristic analysis on the operation logs, and obtain the resource competition degree matrix of each computing node. The resource competition degree matrix is ​​processed by a time-varying evolution graph model to generate a dynamic topological entropy sequence of interactions between nodes; Based on the dynamic topological entropy sequence, the distributed task scheduling queue is processed to collaboratively construct a task-data dependency graph, thereby obtaining the heterogeneous computing task mapping relationship. The distributed file system's metadata index is invoked to perform data fragmentation and relocation processing on the heterogeneous computing task mapping relationship, generating data routing rules between nodes; Based on the inter-node data routing rules, the operating system kernel of the distributed computer cluster is reconfigured to handle data forwarding paths, thereby completing the data processing of the distributed computer cluster.

2. The data processing method for a distributed computer cluster according to claim 1, characterized in that, The non-cooperative game characteristic analysis performed on the operation logs yields the resource competition matrix for each computing node, including: The operation log is processed by extracting system call sequences to obtain the system call frequency vector of each computing node; The system call frequency vector is processed to construct a resource contention strategy space, resulting in a strategy game matrix between nodes; The Nash equilibrium point is solved by processing the strategy game matrix between the nodes to generate the resource competition degree matrix of each computing node.

3. The data processing method for a distributed computer cluster according to claim 1, characterized in that, The step of constructing a time-varying evolution graph model for the resource competition matrix to generate a dynamic topological entropy sequence of interactions between nodes includes: The resource contention matrix is ​​subjected to time window sliding slicing to obtain a sequence of contention submatrices for consecutive time slices; Each of the competition submatrices is processed by graph structure adjacency matrix mapping to generate a time-varying graph snapshot of the interactions between nodes; The time-varying graph snapshot is processed by topological entropy calculation to generate a dynamic topological entropy sequence of interactions between nodes.

4. The data processing method for a distributed computer cluster according to claim 1, characterized in that, The process of collaboratively constructing a task-data dependency graph on the distributed task scheduling queue based on the dynamic topological entropy sequence to obtain the heterogeneous computing task mapping relationship includes: The dynamic topological entropy sequence is subjected to entropy mutation point detection processing to identify the abnormal fluctuation range of topological entropy of computing nodes; Within the abnormal fluctuation range of the topological entropy, the task priority of the distributed task scheduling queue is reordered to obtain the adjusted task scheduling order. The adjusted task scheduling order is processed to construct a task-data dependency graph, generating a heterogeneous computing task mapping relationship.

5. The data processing method for a distributed computer cluster according to claim 4, characterized in that, The step of reordering the task priorities of the distributed task scheduling queue within the abnormal fluctuation range of the topological entropy to obtain the adjusted task scheduling order includes: Extract data locality scores and computational dependency depth scores for tasks from the distributed task scheduling queue; The data locality score and computational dependency depth score are weighted and summed to generate a comprehensive scheduling priority for the task. The distributed task scheduling queue is reordered based on the comprehensive scheduling priority to obtain the adjusted task scheduling order.

6. The data processing method for a distributed computer cluster according to claim 1, characterized in that, The process of calling the metadata index of the distributed file system to perform data fragmentation and relocation processing on the heterogeneous computing task mapping relationship, and generating data routing rules between nodes, includes: Parse the task computing node identifier and data storage node identifier in the heterogeneous computing task mapping relationship; The distributed file system's metadata index is used to perform statistical processing of data block access frequency, generating a distribution map of hot data blocks; Based on the hotspot data block distribution map, the data shards are remapped to target nodes to generate data routing rules between nodes.

7. The data processing method for a distributed computer cluster according to claim 6, characterized in that, The process of remapping data shards based on the hotspot data block distribution map to generate data routing rules between nodes includes: The hot data block distribution map is sorted by access popularity to obtain a priority sequence of hot data blocks; An iterative data migration strategy is generated by performing an iterative data migration strategy on the priority sequence of the hot data shards to obtain a set of migration target nodes for the data shards. Based on the set of target migration nodes, construct a data forwarding path table between nodes and generate data routing rules between nodes.

8. The data processing method for a distributed computer cluster according to claim 1, characterized in that, The step of reconfiguring the data forwarding path of the operating system kernel of the distributed computer cluster according to the data routing rules between nodes to complete the data processing of the distributed computer cluster includes: The inter-node data routing rules are parsed into kernel-level routing table entry update instructions; The kernel-level routing table entry update instruction is used to refresh the routing cache of the operating system's network protocol stack to obtain the updated forwarding path table. Based on the updated forwarding path table, the data packets of the distributed computer cluster are processed to select forwarding paths, thus completing the data processing of the distributed computer cluster.

9. A data processing method for a distributed computer cluster according to claim 8, characterized in that, The step of parsing the inter-node data routing rules into kernel-level routing table entry update instructions includes: The target node IP address and routing priority fields are extracted from the data routing rules between the nodes to obtain routing update parameter pairs; The routing update parameter pair is processed by kernel routing table entry format conversion to generate kernel-level routing table entry update instructions.

10. A data processing method for a distributed computer cluster according to claim 1, characterized in that, The process of calling the metadata index of the distributed file system to perform data fragmentation and relocation processing on the heterogeneous computing task mapping relationship, and generating data routing rules between nodes, includes: Parse the task computing node identifier and data storage node identifier in the heterogeneous computing task mapping relationship; The distributed file system's metadata index is used to perform statistical processing of data block access frequency, generating a distribution map of hot data blocks; Based on the hotspot data block distribution map, the data shards are remapped to target nodes to generate data routing rules between nodes.

Images