A distributed heterogeneous node optimization method and system

Abstract

The present invention discloses a distributed heterogeneous node optimization method and system, which relate to the field of data processing technology. The method collects network traffic feature data and data access pattern data of a distributed heterogeneous node system, performs correlation analysis through a traffic access correlation model, extracts key correlation features, inputs the key correlation features into a performance impact assessment model, and generates a system performance impact level. If the system is judged to be in a performance degradation state according to the system performance impact level, a performance bottleneck identification model based on causal reasoning is used to identify the performance bottleneck source based on the network traffic feature data, the data access pattern data, and the system performance impact level. Based on the performance bottleneck source, the key correlation features are input into an optimization strategy generation model based on deep reinforcement learning to dynamically adjust the optimization threshold and generate a corresponding data sharding strategy, thereby improving the comprehensiveness and accuracy of system monitoring.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a distributed heterogeneous node optimization method and system. Background Art

[0002] A distributed heterogeneous node network refers to a distributed system composed of computing nodes of different types, architectures, or performance characteristics. In a distributed heterogeneous network, network traffic characteristics between nodes, such as packet size distribution and transmission delay fluctuations, as well as data access patterns, such as read-write ratio and access frequency, exhibit nonlinear correlations. Certain seemingly normal data access patterns, when combined with specific network traffic characteristics, can lead to system performance degradation. For example, when a many-to-one communication model is used, such as in a distributed storage system where a client simultaneously sends requests to multiple servers and expects synchronous responses, performance degradation can occur. This performance degradation stems from microbursts generated when multiple servers simultaneously send data to a single client, leading to switch buffer overflows, packet loss, and TCP retransmissions. This correlation requires cross-layer data analysis at the network and application layers, making it difficult to capture with traditional single-layer monitoring.

[0003] To this end, the present invention proposes a distributed heterogeneous node optimization method and system. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a distributed heterogeneous node optimization method and system, which improves the comprehensiveness and accuracy of system monitoring.

[0005] To achieve the above objectives, a distributed heterogeneous node optimization method is proposed, which includes the following steps:

[0006] Step 1: Collect network traffic characteristic data and data access pattern data of distributed heterogeneous node systems;

[0007] Step 2: performing correlation analysis on the network traffic feature data and the data access pattern data using a pre-trained traffic access correlation model to extract key correlation features;

[0008] Step 3: Input the key correlation features into a pre-trained performance impact assessment model to generate a system performance impact level; and determine whether the current system is in a performance degradation state based on the system performance impact level;

[0009] Step 4: If the system is in a performance degradation state, identifying the source of the performance bottleneck based on the network traffic feature data, the data access pattern data, and the system performance impact level through a performance bottleneck identification model based on causal reasoning;

[0010] Step 5: Based on the performance bottleneck source, the key correlation features are input into an optimization strategy generation model based on deep reinforcement learning. The optimization strategy generation model dynamically adjusts the optimization threshold based on historical optimization experience and current system parameters, and generates a corresponding data sharding strategy based on the optimization threshold and network traffic characteristics.

[0011] The method of collecting network traffic characteristic data and data access pattern data of the distributed heterogeneous node system is:

[0012] The network traffic monitoring component and data access monitoring component deployed in the distributed heterogeneous node system collect real-time network traffic characteristic data and data access pattern data of each node respectively;

[0013] Among them, the network traffic monitoring component is deployed on the network switching equipment, routers and computing nodes of the distributed heterogeneous node system to monitor the network traffic characteristic data composed of packet size distribution, transmission delay, throughput and packet loss rate in real time; the computing node refers to the server or virtual machine instance that performs computing tasks in the distributed heterogeneous node system.

[0014] The data access monitoring component is installed on the storage nodes and computing nodes of the distributed heterogeneous node system to monitor data access characteristic data consisting of read-write ratio, access frequency, data block size, and access mode; the storage node refers to the server or virtual machine instance that performs storage tasks in the distributed heterogeneous node system;

[0015] The method of performing correlation analysis on the network traffic feature data and the data access pattern data using a pre-trained traffic access correlation model to extract key correlation features includes the following steps:

[0016] Step 21: Collect historical monitoring data of the distributed heterogeneous node system and build a multi-source heterogeneous data training sample library;

[0017] Step 22: Construct a traffic access correlation model, which adopts a multi-flow architecture design and includes a feature extraction module, a time series modeling module, a correlation analysis module, and a feature output module;

[0018] Step 23: Based on the constructed multi-source heterogeneous data training sample library, use an end-to-end supervised learning approach to train the traffic access association model and optimize network parameters.

[0019] Step 24: Use the trained traffic access correlation model to process the real-time network traffic feature data and data access pattern data to extract the key correlation features of the distributed heterogeneous node system.

[0020] Inputting the key correlation features into a pre-trained performance impact assessment model to generate a system performance impact level comprises the following steps:

[0021] Step 31: Synchronously construct a training sample library for the performance impact assessment model with the multi-source heterogeneous data training sample library; the training sample library contains two types of data: data with no performance degradation and data with performance degradation;

[0022] Step 32: Design a performance impact evaluation model of a network structure including a feature extraction layer, a time series modeling layer, and a performance evaluation layer;

[0023] Step 33: Based on the constructed training sample library, a supervised learning method is used to train the performance impact assessment model with the goal of optimizing performance classification;

[0024] Step 34: The performance impact assessment model receives the key correlation features as input and generates a performance impact level classification result of the system performance impact level in real time.

[0025] The method of determining whether the current system is in a performance degradation state according to the system performance impact level is:

[0026] If the performance impact level of the distributed heterogeneous node system is indicated as moderate degradation or severe degradation, it is determined that the current distributed heterogeneous node system is in a performance degradation state;

[0027] The identifying of the performance bottleneck source by using the causal reasoning-based performance bottleneck identification model based on the network traffic feature data, the data access pattern data, and the system performance impact level comprises the following steps:

[0028] Step 41: Based on the historical monitoring data of the distributed heterogeneous node system and the pre-labeled performance degradation event records, a second training sample library including the performance degradation type, cause, and symptom causal relationship is constructed;

[0029] Step 42: Based on the second training sample library, a structured learning method is used to construct a three-layer hierarchical causal reasoning network structure including a root cause node layer, an intermediate state node layer, and an observation feature node layer;

[0030] Step 43: Based on the performance degradation event records in the second training sample library, use the maximum likelihood estimation and expectation maximization algorithm to learn the conditional probability table of the causal relationship between each node in the causal reasoning network;

[0031] Step 44: Based on the real-time key correlation features extracted from the collected network traffic feature data and data access pattern data, the performance bottleneck identification model uses the joint tree algorithm for probabilistic reasoning to calculate the posterior probability distribution of each root cause node and output the performance bottleneck source with the highest probability.

[0032] Step 45: Based on the identified performance bottleneck source, a performance bottleneck feature library is constructed to classify and extract features of the performance bottlenecks;

[0033] Step 46: Based on the constructed performance bottleneck feature library and in combination with the identified performance bottleneck source, the performance bottleneck is identified and located.

[0034] The dynamic adjustment of the optimization threshold comprises the following steps:

[0035] Step 51: Construct a third training sample library for the optimization strategy generation model, which includes historical performance bottleneck data, key correlation feature data, optimization threshold setting records, and optimization effect evaluation data;

[0036] Step 52: Design an optimization strategy generation model based on deep reinforcement learning, including a state representation module, an action generation module, and a value assessment module;

[0037] The optimization strategy generation model adopts a deep reinforcement learning framework, regards the optimization strategy generation model as an intelligent agent, optimizes the threshold adjustment as the agent's action, and optimizes the effect evaluation as the reward signal of the environment feedback.

[0038] Step 53: Design an optimization effect evaluation function for the optimization strategy generation model to comprehensively evaluate performance improvement, resource utilization, system stability, and optimization overhead factors;

[0039] Step 54: Based on the constructed third training sample library, a deep reinforcement learning algorithm is used to train the optimization strategy generation model;

[0040] Step 55: Based on the newly identified performance bottlenecks and the newly extracted key correlation features, a model is generated through the trained optimization strategy, and the optimization threshold is adjusted in real time.

[0041] Based on the optimization threshold and network traffic characteristics, generating a corresponding data sharding strategy includes:

[0042] Based on the optimized threshold and identified network traffic characteristics, a data access pattern analysis model is constructed to generate a hierarchical data sharding strategy;

[0043] The hierarchical data sharding strategy is converted into execution instructions, and an optimization guidance plan is generated based on the node role type and network location.

[0044] A distributed heterogeneous node optimization system is proposed, which includes a traffic access data collection module, a correlation analysis module, a performance judgment module, a bottleneck source identification module, and a sharding strategy generation module. The modules are electrically connected to each other.

[0045] A traffic access data collection module collects network traffic feature data and data access pattern data of a distributed heterogeneous node system, and sends the network traffic feature data and data access pattern data to the correlation analysis module and the bottleneck source identification module;

[0046] an association analysis module, which performs association analysis on the network traffic feature data and the data access pattern data using a pre-trained traffic access association model, extracts key association features, and sends the key association features to the performance judgment module;

[0047] a performance judgment module that inputs the key correlation features into a pre-trained performance impact assessment model to generate a system performance impact level; based on the system performance impact level, determines whether the current system is in a performance degradation state; and if so, sends the system performance impact level to the bottleneck source identification module;

[0048] A bottleneck source identification module, which identifies the performance bottleneck source based on the network traffic feature data, the data access pattern data and the system performance impact level through a performance bottleneck identification model based on causal reasoning, and sends the performance bottleneck source to the sharding strategy generation module;

[0049] The sharding strategy generation module inputs the key correlation features into the optimization strategy generation model based on deep reinforcement learning based on the performance bottleneck source. The optimization strategy generation model dynamically adjusts the optimization threshold based on historical optimization experience and current system parameters, and generates a corresponding data sharding strategy based on the optimization threshold and network traffic characteristics.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The present invention first collects network traffic feature data and data access pattern data of a distributed heterogeneous node system, and inputs the collected data into a pre-trained traffic access correlation model. The traffic access correlation model uses a deep learning algorithm to analyze the correlation between network traffic and data access behavior, extracting key correlation features that can reflect the system's operating status to comprehensively reflect the system's operating status; the extracted key correlation features are input into a performance impact assessment model. The performance impact assessment model generates a system performance impact level based on historical data and current features, and determines whether the system is in a performance degradation state. If performance degradation is detected, the subsequent bottleneck identification and optimization process is triggered. When the system is confirmed to be in a performance degradation state, the performance bottleneck identification model based on causal reasoning analyzes the causal relationship between network traffic features, data access patterns and performance impact levels, identifies the performance bottleneck source that causes performance degradation, and then inputs the identified performance bottleneck source and key correlation features into an optimization strategy generation model based on deep reinforcement learning. In order to combine historical optimization experience and current system parameters, the optimization threshold is dynamically adjusted, and a targeted data sharding strategy is generated based on this information. This solves the problem of the separation between the network layer and the application layer in traditional optimization methods. Through the traffic access association model, the present invention can reveal the intrinsic connection between network traffic characteristics and data access patterns, realize multi-dimensional system status perception, and improve the comprehensiveness and accuracy of system monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a distributed heterogeneous node optimization method in Example 1 of the present invention;

[0053] Figure 2 This is a model structure diagram of the traffic access association model in Example 1 of the present invention;

[0054] Figure 3 This is a module connection diagram of a distributed heterogeneous node optimization system in Example 2 of the present invention. DETAILED DESCRIPTION

[0055] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] Example 1

[0057] like Figure 1 As shown, a distributed heterogeneous node optimization method includes the following steps:

[0058] Step 1: Collect network traffic characteristic data and data access pattern data of distributed heterogeneous node systems;

[0059] Step 2: performing correlation analysis on the network traffic feature data and the data access pattern data using a pre-trained traffic access correlation model to extract key correlation features;

[0060] Step 3: Input the key correlation features into a pre-trained performance impact assessment model to generate a system performance impact level; and determine whether the current system is in a performance degradation state based on the system performance impact level;

[0061] Step 4: If the system is in a performance degradation state, identifying the source of the performance bottleneck based on the network traffic feature data, the data access pattern data, and the system performance impact level through a performance bottleneck identification model based on causal reasoning;

[0062] Step 5: Based on the performance bottleneck source, the key correlation features are input into an optimization strategy generation model based on deep reinforcement learning. The optimization strategy generation model dynamically adjusts the optimization threshold based on historical optimization experience and current system parameters, and generates a corresponding data sharding strategy based on the optimization threshold and network traffic characteristics.

[0063] In an embodiment of the present invention, the method of collecting network traffic feature data and data access pattern data of a distributed heterogeneous node system is:

[0064] The network traffic monitoring component and data access monitoring component deployed in the distributed heterogeneous node system collect real-time network traffic characteristic data and data access pattern data of each node respectively;

[0065] Among them, the network traffic monitoring component is deployed on the network switching equipment, routers and computing nodes of the distributed heterogeneous node system to monitor the network traffic characteristic data composed of packet size distribution, transmission delay, throughput, packet loss rate, etc. in real time; the computing node refers to the server or virtual machine instance that performs computing tasks in the distributed heterogeneous node system.

[0066] The data access monitoring component is installed on the storage nodes and computing nodes of the distributed heterogeneous node system to monitor data access characteristic data composed of read-write ratio, access frequency, data block size, access mode, etc. The storage node refers to the server or virtual machine instance that performs storage tasks in the distributed heterogeneous node system;

[0067] In the specific implementation of the present invention, during the data collection process, the network traffic monitoring component and the data access monitoring component continuously collect parameter data according to the preset sampling frequency. The sampling frequency of the network traffic monitoring component is set to once every 100 milliseconds to ensure timely capture of network traffic change trends; the sampling frequency of the data access monitoring component is set to once every 200 milliseconds to monitor data access status in real time;

[0068] It should be noted that during the monitoring process, the system performance monitoring component is deployed to collect real-time performance characteristic data to collect the label values ​​of the sample data required for the subsequent performance impact assessment model. This system performance monitoring component monitors system performance characteristics such as CPU utilization, memory usage, and disk I / O through agent programs deployed on each compute node and storage node. The system performance monitoring component samples every 500 milliseconds to ensure real-time monitoring of system performance.

[0069] Furthermore, the method of performing correlation analysis on the network traffic feature data and the data access pattern data using a pre-trained traffic access correlation model to extract key correlation features includes the following steps:

[0070] Step 21: Collect historical monitoring data of the distributed heterogeneous node system and build a multi-source heterogeneous data training sample library;

[0071] Specifically, in this embodiment, during the collection of the historical monitoring data, the same method as step one is used to extract network traffic characteristic data, data access pattern data and performance characteristic data for at least 6 consecutive months from the distributed heterogeneous node system management platform to ensure that the samples cover system operating conditions under different load types, different network conditions and different node configurations, that is, the collected data includes three types of heterogeneous data: network traffic characteristics, data access characteristics and system performance characteristics during the operation of the historical distributed heterogeneous node system.

[0072] In an embodiment of the present invention, the multi-source heterogeneous data training sample library is organized in time series. Each sample contains monitoring data from a time window. Typically, the time window length is set to 10 seconds, with a sliding step size of 2 seconds. For each time window, network traffic characteristics, data access characteristics, and system performance characteristics within that time window are extracted to form a multi-source heterogeneous data sample. Simultaneously, the system performance indicators corresponding to that time window, including response time, throughput, and resource utilization, are recorded as sample labels.

[0073] Step 22: Build Figure 2The traffic access correlation model of the model structure shown adopts a multi-flow architecture design, including a feature extraction module, a time series modeling module, a correlation analysis module and a feature output module;

[0074] Specifically, the traffic access correlation model adopts the overall architecture of "multi-stream-correlation-output". According to the characteristics of three types of heterogeneous data in distributed heterogeneous node systems, namely network traffic characteristics, data access characteristics and system performance characteristics, three parallel feature extraction streams are constructed to process different types of data respectively. The features extracted from each stream are then integrated through the correlation module, and finally the key correlation features are output.

[0075] In the feature extraction module, a one-dimensional convolutional neural network architecture consisting of three consecutive one-dimensional convolutional blocks is used to extract network traffic feature data. Each convolutional block consists of a one-dimensional convolutional layer with a kernel size of 5, a batch normalization layer, a ReLU activation function, and a max pooling layer. The first convolutional block has 64 output channels, and this number doubles with each subsequent block, effectively capturing spatial correlations and short-term variation patterns among network traffic features.

[0076] To address the data flow of data access features, a multi-scale residual network architecture consisting of four residual blocks is employed. Each residual block contains two one-dimensional convolutional layers and a skip connection to capture data access features at different time scales. Residual connections help alleviate the vanishing gradient problem in deep networks, improving model training stability. This allows for handling slowly changing and periodic features in data access features.

[0077] In the time series modeling module, a temporal attention mechanism is applied to the features extracted from the two data streams to learn the importance weights of features at different time points. Specifically, the temporal attention mechanism calculates the similarity between the query vector and the features at each time point to generate attention weights, and then performs a weighted summation of the features at each time point to obtain a feature representation with temporal importance. This mechanism enables the traffic access association model to adaptively focus on key time points during the operation of a distributed heterogeneous node system, such as sudden changes in network traffic and changes in data access patterns. A time point feature is a representation of the data state at a specific moment in a distributed heterogeneous node system. Specifically, for network traffic data streams, a time point feature represents the numerical state of network metrics such as packet size distribution, transmission delay, and throughput at that moment; for data access data streams, a time point feature represents the state of access parameters such as read-write ratio, access frequency, and data block size at that moment.

[0078] In the association analysis module, an adaptive feature association strategy is employed, comprising three steps: feature alignment, feature interaction, and feature fusion. The feature alignment step maps the features output by the time series modeling module from two data streams to the same feature space via a fully connected layer. The feature interaction step uses a cross-stream attention mechanism to calculate the mutual influence between the feature spaces corresponding to different data streams, such as the impact of network traffic on data access performance or the impact of data access patterns on network congestion. The feature fusion step uses a gated fusion unit to adaptively determine fusion weights based on the correlation between the feature spaces output by the feature interaction step for each data stream, generating a comprehensive feature representation.

[0079] In the feature output module, a two-layer fully connected network performs dimensionality reduction and feature selection on the comprehensive feature representation to extract the most representative key correlation features. The output dimension of the first fully connected layer is 256, using the ReLU activation function; the output dimension of the second fully connected layer is 128, corresponding to the final key correlation feature dimension.

[0080] Step 23: Based on the constructed multi-source heterogeneous data training sample library, use an end-to-end supervised learning approach to train the traffic access association model and optimize network parameters.

[0081] Specifically, the training of the traffic access correlation model is based on the performance of the distributed heterogeneous node system. The training process is divided into the following steps:

[0082] First, the multi-source heterogeneous data training sample library is divided into training, validation, and test sets in a ratio of 7:2:1. The training set is used to learn model parameters, the validation set is used for model selection and hyperparameter adjustment, and the test set is used to evaluate the performance of the final model.

[0083] For the loss function of the traffic access association model, a multi-task learning framework is used to optimize multiple objectives simultaneously. In this embodiment, the objectives of the loss function include: performance degradation prediction loss, which uses binary cross entropy to calculate the difference between the predicted performance degradation probability and the corresponding actual label; performance impact level prediction loss, which uses mean square error (MSE) to calculate the difference between the predicted value and the actual value; performance bottleneck identification loss, which uses multi-label classification loss to calculate the difference between the predicted performance bottleneck and the corresponding actual label. The three loss functions are combined by weighted summation, and the weights are set according to the importance of each task.

[0084] The traffic access association model optimization algorithm uses the Adam optimizer. The initial learning rate is set to 0.0001, and a learning rate decay strategy is used, multiplying the learning rate by 0.9 every 10 training epochs. The batch size is set to 64, and the number of training epochs is 100. During training, an early stopping strategy is used. When the loss on the validation set does not improve for five consecutive epochs, training is stopped to avoid overfitting. At the same time, the model parameters with the best performance on the validation set are saved as the final model.

[0085] Step 24: Use the trained traffic access correlation model to process the real-time network traffic feature data and data access pattern data to extract key correlation features of the distributed heterogeneous node system;

[0086] Specifically, in the actual application stage of this embodiment, the network traffic feature data and data access pattern data obtained in step one are input into the trained traffic access association model, and through the feature extraction module, timing modeling module, association analysis module and feature output module in step 22, a 128-dimensional key association feature vector is generated.

[0087] The key association feature vector contains key information about the current state of the distributed heterogeneous node system. In an embodiment of the present invention, it specifically includes: a network feature sub-vector, which characterizes the key characteristics and change trends of various types of network traffic; and a data access feature sub-vector, which characterizes the state and change characteristics of data access features.

[0088] It should be noted that the traffic access correlation model utilizes a multi-stream parallel feature extraction architecture to address the multi-source and heterogeneous nature of data in distributed heterogeneous node systems. Each data stream utilizes a neural network structure tailored to the data type, such as a one-dimensional convolutional network for network traffic features and a residual network for data access features. This enables efficient feature extraction for different data types. Secondly, the traffic access correlation model incorporates temporal and cross-stream attention mechanisms to adaptively focus on key moments in time series and the interactions between different data sources, thereby capturing the dynamic changes in distributed heterogeneous node systems. Thirdly, an adaptive feature fusion strategy is employed, dynamically adjusting the fusion weights of different data sources through a gated fusion unit. This enables the traffic access correlation model to flexibly integrate multi-source information based on the current operating conditions, improving the relevance and accuracy of feature extraction. The traffic access correlation model is trained using a multi-task learning framework, simultaneously optimizing multiple objectives, including performance degradation prediction, performance impact level prediction, and performance bottleneck identification. This results in extracted key correlation features with enhanced comprehensive representation capabilities, enabling subsequent tasks such as performance impact assessment, performance bottleneck identification, and optimization strategy generation.

[0089] Furthermore, inputting the key correlation features into a pre-trained performance impact assessment model to generate a system performance impact level includes the following steps:

[0090] Step 31: Synchronously construct a training sample library for the performance impact assessment model with the multi-source heterogeneous data training sample library; the training sample library contains two types of data: data with no performance degradation and data with performance degradation;

[0091] Specifically, in a preferred embodiment, the process of constructing the training sample library of the performance impact assessment model is performed simultaneously with the collection of the multi-source heterogeneous data training sample library of the traffic access association model in step 2.

[0092] During the data collection phase when performance did not degrade, 60 consecutive days of data were selected from monitoring data collected during the normal operation of the distributed heterogeneous node system as the basic sample set. Each basic sample record includes a timestamp, network traffic characteristics, data access characteristics, and system performance characteristics of each node, as well as corresponding performance evaluation indicators (such as response time and throughput). Normal operation is based on engineers' empirical judgment of the system's operating status.

[0093] During the performance degradation data collection phase, marked performance degradation event data is extracted from historical monitoring data as performance degradation samples, and the performance degradation samples are marked with the degree of performance impact according to the preset degradation threshold. The performance impact degree is such as "normal", "slight degradation", "moderate degradation" and "severe degradation" to record all parameter changes during the system response process.

[0094] In a preferred embodiment, each performance degradation sample can also be annotated by experts to determine the degree of performance degradation and the scope of impact; performance degradation conditions are divided into network congestion, data access conflict, resource competition, etc.; the scope of impact marks the affected nodes and possible chain reactions.

[0095] Step 32: Design a performance impact evaluation model of a network structure including a feature extraction layer, a time series modeling layer, and a performance evaluation layer;

[0096] Specifically, in the feature extraction layer, a multilayer perceptron network is used to perform nonlinear transformation and dimensionality reduction on the key input features. This layer consists of five fully connected layers, each with 128 neurons. The ReLU activation function enhances the performance impact assessment model's ability to express nonlinear relationships. Dropout layers are added between the fully connected layers to prevent overfitting.

[0097] In the time series modeling layer, a bidirectional long short-term memory (Bi-LSTM) network is used to capture the temporal dependencies of parameters in a distributed, heterogeneous node system. This layer comprises three Bi-LSTM layers, each containing 64 LSTM units. This layer simultaneously evaluates information from past and future time points to provide a more comprehensive understanding of parameter trends. The output of the Bi-LSTM layer is processed through an attention mechanism, adaptively assigning weights to features at different time points and highlighting state changes at key moments.

[0098] At the performance evaluation layer, the output features of the time series modeling layer are further abstracted through two fully connected layers. Finally, a Softmax function is used to output the performance impact classification results for the distributed heterogeneous node system. In this embodiment, the performance impact classification results include four categories: "normal," "slightly degraded," "moderately degraded," and "severely degraded."

[0099] Step 33: Based on the constructed training sample library, a supervised learning method is used to train the performance impact assessment model with the goal of optimizing performance classification;

[0100] Specifically, first, the training sample library is divided into a training set and a validation set according to a preset ratio.

[0101] During model training, we used a mini-batch gradient descent algorithm with a batch size of 32 and an initial learning rate of 0.0001. We also used a learning rate decay strategy, reducing the learning rate by a factor of 0.9 every several rounds. We also used an early stopping strategy during training, stopping training if performance on the validation set did not improve for several consecutive rounds to prevent overfitting.

[0102] The loss function of the model is the performance impact level classification loss. The performance impact level classification loss uses the cross entropy loss function to measure the difference between the predicted performance impact level and the actual performance impact level.

[0103] Step 34: The performance impact assessment model receives the key correlation features as input and generates a performance impact level classification result of the system performance impact level in real time.

[0104] Furthermore, the method of determining whether the current system is in a performance degradation state according to the system performance impact level is:

[0105] If the performance impact level of the distributed heterogeneous node system is indicated as "moderate degradation" or "severe degradation", it is determined that the current distributed heterogeneous node system is in a performance degradation state;

[0106] Furthermore, the identifying of the performance bottleneck source by using the performance bottleneck identification model based on causal reasoning, based on the network traffic feature data, the data access pattern data, and the system performance impact level, includes the following steps:

[0107] Step 41: Based on the historical monitoring data of the distributed heterogeneous node system and the performance degradation event records annotated by experts, a second training sample library containing the performance degradation type, cause, and symptom causal relationship is constructed;

[0108] Specifically, in an embodiment of the present invention, during the construction phase of the second training sample library, all confirmed performance degradation event data are screened from the historical monitoring data of the distributed heterogeneous node system, and the following information is synchronously supplemented for each performance degradation working condition sample: the time period of performance degradation, real-time monitoring data before and after the performance degradation, the type and root cause of performance degradation, the propagation path of performance degradation, the scope and extent of the impact of performance degradation, the measures taken to deal with the performance degradation and their effects. It can be understood that, unlike the training sample library of the performance impact assessment model, the second training sample library pays more attention to the causal chain record of performance degradation events, ensuring that the propagation law of performance degradation in the distributed heterogeneous node system can be accurately captured.

[0109] In a preferred embodiment of the present invention, performance degradation types can be divided into three categories based on their source: network traffic anomalies, data access anomalies, and system resource anomalies. Network traffic anomalies include, but are not limited to, network congestion, high transmission latency, high packet loss rate, and insufficient bandwidth; data access anomalies include, but are not limited to, read-write conflicts, hotspot data contention, unbalanced data sharding, and low cache hit rates; and system resource anomalies include, but are not limited to, excessive CPU utilization, insufficient memory, disk I / O bottlenecks, and thread contention.

[0110] A team of experts, comprised of distributed systems and performance optimization specialists, annotated each performance degradation event in the second training sample library, identifying the causal chain between the root cause, secondary causes, and symptoms of the performance degradation. For example, unbalanced data sharding (the root cause) can lead to the emergence of a hotspot node (a secondary cause), which in turn causes network congestion at that node (an intermediate state), ultimately increasing overall system response time (a symptom). These detailed causal chain annotations provide precise prior knowledge for learning the causal inference network structure, complementing the performance impact assessment model's training sample library, which focuses solely on the level of performance impact.

[0111] Step 42: Based on the second training sample library, a method combining structural learning and expert knowledge is used to construct a three-layer hierarchical causal reasoning network structure including a root cause node layer, an intermediate state node layer, and an observation feature node layer;

[0112] Specifically, the causal reasoning network structure adopts a hierarchical design and includes three layers of nodes:

[0113] The first layer is the root cause node layer, representing the fundamental causes of performance degradation with a probability greater than the preset probability threshold. This includes the network anomaly node group, the data access anomaly node group, and the system resource anomaly node group. The network anomaly node group includes nodes with network congestion, insufficient bandwidth, and routing anomalies; the data access anomaly node group includes nodes with unbalanced data sharding, improper caching policies, and read / write mode conflicts; and the system resource anomaly node group includes nodes with CPU overload, insufficient memory, and disk I / O bottlenecks.

[0114] The second layer is the intermediate status node layer, which represents system status changes caused by fundamental performance degradation. It includes network status node groups, data access status node groups, and system resource status node groups. The network status node group includes nodes with increased transmission latency, decreased throughput, and increased packet loss rate. The data access status node group includes nodes with increased read and write latency, decreased cache hit rate, and increased data consistency overhead. The system resource status node group includes nodes with increased CPU queues, increased memory swap frequency, and increased disk queue length.

[0115] The third layer is the observation feature node layer, representing abnormal features observable through the monitoring system. It includes network feature node groups, data access feature node groups, and system resource feature node groups. The network feature node group includes nodes with abnormal packet size distribution, transmission delay fluctuations, and throughput degradation; the data access feature node group includes nodes with abnormal read-write ratios, sudden access frequency changes, and abnormal data block sizes; and the system resource feature node group includes nodes with abnormal CPU utilization, abnormal memory usage, and abnormal disk I / O wait time.

[0116] In the process of constructing the causal inference network structure, a method combining structural learning and expert knowledge is employed. First, the initial causal relationship structure between nodes is determined based on prior knowledge provided by domain experts. Then, a scoring-based structural learning algorithm, such as the K² algorithm or the Bayesian Information Criterion (BIC) algorithm, is used to learn and optimize the network structure from a secondary training sample database. During the structural learning process, the strong causal relationships defined by the experts are maintained, and only weak causal relationships are optimized and adjusted.

[0117] Step 43: Based on the performance degradation event records in the second training sample library, use the maximum likelihood estimation and expectation maximization algorithm to learn the conditional probability table of the causal relationship between each node in the causal reasoning network;

[0118] Specifically, after determining the causal reasoning network structure, based on the performance degradation event records in the second training sample library, the conditional probability table of each node is learned, that is, the probability distribution of the current node in each possible state given the parent node state.

[0119] For discrete nodes, conditional probabilities are calculated from the second training sample library using maximum likelihood estimation. For example, the conditional probability table for nodes with "unbalanced data sharding" (a value of "yes" or "no") to nodes with "hotspot" (a value of "yes" or "no") is determined by counting the proportion of hotspot nodes that appear when data sharding is unbalanced in the second training sample library, and the proportion of hotspot nodes that appear when data sharding is balanced.

[0120] In a further preferred embodiment, for the conditional probabilities in the second training sample library where data is sparse, a Bayesian estimation method is used, introducing a prior distribution to smooth the probability estimate and avoid the zero probability problem. Specifically, a small pseudo-count (e.g., 0.5) is added to each conditional probability to ensure that even state combinations not observed in the training data have non-zero probabilities.

[0121] For discrete child nodes with continuous parent nodes, a soft discretization method is used, mapping the continuous value to the probability of a discrete state using the sigmoid function. For example, to map "CPU utilization" (a continuous value) to "CPU overload status" (a discrete state), the function P(CPU overload status = "yes" | CPU utilization = x) = 1 / (1 + exp(α(x - β))) is used, where α and β are parameters learned from the training data.

[0122] In a preferred embodiment of the present invention, conditional probability learning employs the expectation-maximization algorithm, which can handle the problem of missing data in training samples. The EM algorithm iteratively estimates the distribution of missing data in the E step and maximizes the likelihood function that includes the estimated missing data in the M step until convergence.

[0123] Step 44: Based on the real-time key correlation features extracted from the collected network traffic feature data and data access pattern data, the performance bottleneck identification model uses the joint tree algorithm for probabilistic reasoning to calculate the posterior probability distribution of each root cause node and output the performance bottleneck source with the highest probability.

[0124] Specifically, in the actual application stage, when step three determines that the current operating state is in a performance degradation state, the diagnostic model of the causal reasoning network trained based on the second training sample library receives the real-time key correlation features obtained in step one as input information.

[0125] First, map the real-time key correlation features into evidence at the observation feature node level in the causal inference network. For example, if the real-time key correlation features show that the CPU utilization of a node is 95%, far higher than normal, then the "CPU Utilization Abnormal" node is set as "Yes" evidence; if the access frequency of a data shard suddenly increases 10 times, then the "Access Frequency Sudden Change" node is set as "Yes" evidence.

[0126] Then, based on the provided evidence, a causal inference network inference algorithm is used to calculate the posterior probability distribution of each root cause node in the root cause node layer. In an embodiment of the present invention, a joint tree algorithm is used for inference, and probability propagation calculation is implemented by converting the causal inference network into a joint tree structure.

[0127] Finally, based on the posterior probabilities of each node in the root cause node layer, the performance bottleneck source with the highest probability is identified and output. Specifically, the root cause node with the highest posterior probability is selected as the primary performance bottleneck source. Other root cause nodes with posterior probabilities exceeding a preset threshold (e.g., 0.3) are evaluated as secondary performance bottleneck sources. These performance bottleneck sources and their corresponding posterior probabilities are directly output, with the performance bottleneck source with the highest posterior probability being designated as the primary performance bottleneck source.

[0128] It should be noted that the performance bottleneck identification model based on causal reasoning has the following advantages: first, the graph structure of the causal reasoning network intuitively expresses the causal relationship between various parameters and performance factors in the distributed heterogeneous node system, making the diagnosis results interpretable; second, the causal reasoning network can process incomplete observation data and can still perform effective reasoning even if some monitoring data is missing; third, through the probabilistic reasoning mechanism, the causal reasoning network can quantify uncertainty and provide a confidence assessment for each possible source of performance bottlenecks, avoiding the misjudgment that may be caused by deterministic diagnostic methods; finally, the causal reasoning network model can continuously update the conditional probability table as new data accumulates, thereby achieving continuous optimization of diagnostic capabilities.

[0129] Step 45: Based on the identified performance bottleneck source, a performance bottleneck feature library is constructed to classify and extract features of the performance bottlenecks;

[0130] Specifically, the performance bottleneck feature library is a knowledge base that systematically organizes and extracts features from the performance bottleneck sources identified in step 44, and is used to support accurate identification and classification of performance bottlenecks.

[0131] In a preferred embodiment of the present invention, the performance bottleneck feature library is organized according to the type, source, and impact characteristics of performance bottlenecks. Performance bottleneck types include three categories: network traffic anomalies, data access anomalies, and system resource anomalies. Each performance bottleneck category is further divided into several subcategories. For example, network traffic anomalies include network congestion, high transmission latency, and insufficient bandwidth; data access anomalies include read-write conflicts, hotspot data contention, and unbalanced data sharding; and system resource anomalies include CPU overload, insufficient memory, and disk I / O bottlenecks.

[0132] For each performance bottleneck, the feature library records its typical features, including but not limited to perception features, association features, temporal features, and spatial features;

[0133] The perception features describe the typical manifestations of this type of performance bottleneck in the monitoring data, such as the abnormal range and change pattern of specific network traffic, abnormal combinations of data access characteristics, and abnormal indicators of system resources. These features directly correspond to the network traffic feature data and data access pattern data obtained in step 1, facilitating rapid identification.

[0134] Correlation features describe the relationship between a performance bottleneck and other system parameters or states, such as the relationship between certain types of network congestion and data access patterns, or the relationship between data sharding imbalance and node load. These correlation features help us understand the impact mechanism and propagation patterns of performance bottlenecks.

[0135] Among them, time series features describe the typical patterns of performance bottlenecks over time, such as performance degradation rate, fluctuation period, duration, and other time dimension characteristics. These time series features help predict the development trend and severity of performance bottlenecks.

[0136] Among them, spatial features describe the typical patterns of node distribution of such performance bottlenecks, such as the impact range, performance gradient, and distribution of high-risk nodes. These spatial features help determine the impact range of performance bottlenecks.

[0137] During the performance bottleneck feature extraction process, based on the identified performance bottleneck source, feature vectors related to that performance bottleneck are extracted from real-time monitoring data. These feature vectors include multi-dimensional features such as current value, rate of change, fluctuation amplitude, and spatial distribution. The extracted feature vectors are then matched with pre-collected typical feature templates in the feature library, and a similarity score is calculated to determine the specific subcategory and severity of the performance bottleneck.

[0138] It's important to understand that the performance bottleneck feature library is not static; it's constantly updated and refined as the system operates and data accumulates. Each time a performance bottleneck is successfully identified and addressed, the system adds new feature patterns to the library, enriching the diversity and representativeness of the feature templates and improving the accuracy and robustness of subsequent identifications.

[0139] Step 46: Based on the constructed performance bottleneck feature library and the identified performance bottleneck source, the performance bottleneck is identified and located;

[0140] In a preferred embodiment of the present invention, the performance bottleneck identification process adopts a multi-feature fusion method to comprehensively evaluate perceptual features, correlation features, temporal features, and spatial features to improve the accuracy and reliability of identification. The specific steps are as follows: First, based on the identified performance bottleneck source, the relevant performance bottleneck types and their feature templates are retrieved from the performance bottleneck feature library. For example, if the identified performance bottleneck source is "unbalanced data sharding", the related performance bottleneck types such as "hotspot data competition", "unbalanced node load", etc. and their feature templates are retrieved.

[0141] The final results of accurate identification and location of the performance bottleneck include the following: the specific type and subcategory of the performance bottleneck (such as "data sharding imbalance - severe", "network congestion - moderate", etc.); the core location and impact range of the performance bottleneck; the severity rating of the performance bottleneck (such as "minor", "moderate", "severe", "extreme"); the development trend prediction of the performance bottleneck (such as "stable", "slowly deteriorating", "rapidly deteriorating"); and the number and distribution of affected nodes.

[0142] Furthermore, the dynamic adjustment of the optimization threshold comprises the following steps:

[0143] Step 51: Construct a third training sample library for the optimization strategy generation model, which includes historical performance bottleneck data, key correlation feature data, optimization threshold setting records, and optimization effect evaluation data;

[0144] Specifically, in an embodiment of the present invention, the construction process of the third training sample library is carried out simultaneously with the collection of the multi-source heterogeneous data third training sample library of the traffic access association model in step 2, but more emphasis is placed on recording the correlation relationship between the optimization threshold and the optimization effect.

[0145] During the training sample collection phase of the optimization strategy generation model, the following information is extracted from the historical monitoring data of the distributed heterogeneous node system: the type, location, and severity of performance bottlenecks; key correlation feature vectors, including network feature subvectors, data access feature subvectors, and key correlation feature vectors; historical optimization threshold settings, including data sharding thresholds, cache strategy thresholds, and adjustment records; and optimization effect evaluation data, including performance improvement, resource utilization changes, system stability changes, and other indicators.

[0146] For each historical performance bottleneck event, we record the sequence of changes in key correlated features preceding the bottleneck, the optimization threshold settings used at the time, the execution of the optimization strategy, and the actual performance improvement process. These records form a "state-action-reward" triplet, which serves as training data for the reinforcement learning model. The state represents the current key correlated features and performance bottleneck, the action represents the optimization threshold settings, and the reward represents the score of the optimization effect.

[0147] Step 52: Design an optimization strategy generation model based on deep reinforcement learning, including a state representation module, an action generation module, and a value assessment module;

[0148] Specifically, the optimization strategy generation model adopts a deep reinforcement learning framework, regards the optimization strategy generation model as an intelligent agent, optimizes the threshold adjustment as the agent's action, and optimizes the effect evaluation as the reward signal of the environment feedback.

[0149] In the state representation module, the current critical correlation feature vector, identified performance bottlenecks, and historical optimization records are combined to form a state vector. The critical correlation feature vector comes from the traffic access correlation model output in step 2 and includes network feature subvectors, data access feature subvectors, and correlation feature subvectors. The performance bottleneck comes from the output of step 4 and includes the type, location, severity, and development trend of the performance bottleneck. The historical optimization records include the recent optimization strategy execution status and optimization effect evaluation.

[0150] In the action generation module, an optimized threshold adjustment strategy is generated based on the current state. The action space includes the adjustment direction and magnitude of the data sharding threshold and the cache policy threshold. For example, for the data sharding size threshold, the action could be "increase by a certain number of KB," "remain unchanged," or "decrease by a certain number of KB." For the hot data determination threshold, the action could be "increase the number of visits," "remain unchanged," or "decrease the number of visits." Action generation is implemented using a policy network, which consists of a multi-layer, fully connected neural network. The input is a state vector, and the output is a probability distribution of each possible action.

[0151] In the value assessment module, the long-term value of each possible action under the current state is evaluated. This value assessment is implemented using a value network, which is also composed of a multi-layer fully connected neural network. The input is a combination of a state vector and an action vector, and the output is an estimated value for that state-action pair.

[0152] Step 53: Design an optimization effect evaluation function for the optimization strategy generation model, and comprehensively evaluate factors such as performance improvement, resource utilization, system stability, and optimization overhead;

[0153] Specifically, the optimization effect evaluation function serves as a reward function for reinforcement learning, guiding the learning direction of the optimization strategy generation model. The optimization effect is evaluated using the following indicators:

[0154] Performance improvement refers to the degree of improvement in system performance before and after optimization. The calculation formula is: (post-optimization performance - pre-optimization performance) / pre-optimization performance. A greater performance improvement indicates a better optimization effect.

[0155] Resource utilization balance refers to the degree of balance in the use of system resources. It's calculated as: 1 - standard deviation of resource utilization / average resource utilization. A higher resource utilization balance indicates a more balanced distribution of system resources, preventing overloading of some nodes while leaving others idle.

[0156] System stability refers to the degree of fluctuation in system performance after optimization. It's calculated as: 1 - the coefficient of variation of the performance indicator. Higher system stability indicates more stable system performance and a better user experience.

[0157] Optimization overhead refers to the resource consumption and time cost required to execute the optimization strategy. The lower the optimization overhead, the more efficient the optimization strategy.

[0158] In a preferred embodiment of the present invention, the optimization effect evaluation function R takes the form of weighted summation:

[0159] R = w1 × performance improvement + w2 × resource utilization balance + w3 × system stability - w4 × optimization overhead;

[0160] Among them, w1, w2, w3, and w4 are preset weight coefficients, which are set according to the performance requirements of different systems. The performance improvement, resource utilization balance, system stability, and optimization overhead are all normalized to the range of [0, 1] to ensure the comparability of various indicators.

[0161] Step 54: Based on the constructed third training sample library, a deep reinforcement learning algorithm is used to train the optimization strategy generation model;

[0162] Specifically, the optimization strategy generation model is trained using the Deep Q Network (DQN) algorithm or its improved versions, such as dual DQN, priority experience replay DQN, etc. The training process includes the following steps:

[0163] First, initialize the parameters of the policy network and value network, and set the experience replay buffer.

[0164] Next, historical data is loaded from the third training sample library in chronological order to simulate the optimization system's operation. At each time step, an ε-greedy strategy is used to select an action to optimize the threshold adjustment based on the current state (including key correlation features and performance bottlenecks). This strategy randomly selects an action for exploration with a probability of ε, and then selects the action deemed optimal by the current strategy with a probability of 1-ε.

[0165] Next, the selected action is executed, and the feedback from the environment is observed, including the new state and the reward generated by the optimization effect evaluation. The four-tuple (state, action, reward, new state) is stored in the experience replay buffer.

[0166] A batch of samples are randomly sampled from the experience replay buffer and the value network parameters are updated using temporal difference learning. The loss function L of the value network is: L = E[(r + γ × max_a'Q(s',a';θ') - Q(s,a;θ))²], where r is the immediate reward, γ is the discount factor, Q(s,a;θ) is the value network's estimate of the value of taking action a in state s, θ is the value network parameter, and θ' is the target network parameter.

[0167] At fixed intervals, the parameters of the value network are copied to the target network to improve the stability of training. As training progresses, the value of ε is gradually reduced to reduce random exploration and increase the utilization of learned knowledge.

[0168] During training, model performance is regularly evaluated on the validation dataset to monitor the learning curve and avoid overfitting. Training is stopped when validation performance stops improving or when the pre-set number of training rounds is reached.

[0169] Step 55: Based on the newly identified performance bottlenecks and the newly extracted key correlation features, a model is generated through the trained optimization strategy, and the optimization threshold is adjusted in real time to improve the accuracy and effectiveness of the optimization.

[0170] Specifically, in the actual application stage, the optimization strategy generation model receives the key correlation feature vector from step 2 and the performance bottleneck identified in step 4, and calculates the optimal optimization threshold adjustment strategy in real time.

[0171] First, the current key correlation feature vector, performance bottlenecks, and historical optimization records are combined to form a state vector, which is then fed into the trained policy network. The policy network outputs a probability distribution of possible optimization threshold adjustment actions, and the system selects the action with the highest probability as the current threshold adjustment policy.

[0172] Then, based on the selected adjustment strategy, the data sharding threshold and cache policy threshold are updated. For example, if the newly identified performance bottleneck is a "hotspot node caused by unbalanced data sharding" and key correlation characteristics indicate a highly skewed data access pattern, the system may choose to lower the hotspot data determination threshold to trigger data migration in advance. Furthermore, if a strong correlation between network traffic characteristics and data access patterns is assessed, the system may also adjust the cache prefetching policy threshold to enhance adaptability to changing network conditions.

[0173] The adjustment of optimization thresholds is evaluated based on the following factors: the type and severity of performance bottlenecks. Different types of performance bottlenecks correspond to different key parameters and optimization strategies. The development trend of performance bottlenecks. Rapidly worsening performance bottlenecks require more aggressive optimization thresholds. The rate of change of system parameters. Abnormal parameter change rates may indicate worsening performance issues. Historical optimization results. The sensitivity of the threshold is adjusted based on the effectiveness and stability of recent optimizations.

[0174] While adjusting the optimization threshold, the system also records the current state, actions taken, and subsequent optimization results, generating new empirical data for online model learning and continuous optimization. Through this closed-loop feedback mechanism, the optimization strategy generation model can continuously adapt to system and load changes, maintaining the efficiency and reliability of the optimization system.

[0175] It should be noted that the optimization strategy generation model has the following advantages: First, based on the deep reinforcement learning framework, it can learn the optimal threshold adjustment strategy from historical experience, avoiding the subjectivity and static nature of threshold setting in traditional methods; second, by comprehensively evaluating key correlation features and performance bottlenecks, it realizes contextual awareness of the optimization threshold, making the optimization more targeted and intelligent; third, through continuous learning and optimization, the model can adapt to factors such as system changes and load changes, and maintain the long-term effectiveness of the optimization system; finally, the dynamic adjustment of the optimization threshold balances the radicalness and stability of the optimization, while improving the optimization effect, reducing system fluctuations and resource waste, and improving the overall optimization effect.

[0176] Based on the optimization threshold and network traffic characteristics, generating a corresponding data sharding strategy includes:

[0177] Based on the optimized threshold and identified network traffic characteristics, a data access pattern analysis model is constructed to generate a hierarchical data sharding strategy;

[0178] Specifically, the data access pattern analysis model receives the optimization threshold dynamically adjusted from step 55 and the identified network traffic characteristics, and generates a targeted hierarchical data sharding strategy in combination with the current node load and data access status.

[0179] In a preferred embodiment of the present invention, the data fragmentation strategy is divided into four levels: static fragmentation level, local adjustment level, global reorganization level and emergency migration level, which correspond to different system load conditions and network environment conditions respectively.

[0180] Static sharding is used when network traffic is stable and node loads are balanced. It includes basic strategies such as data distribution based on historical access frequency, separation of hot and cold data, and proximity deployment. This level of strategy aims to maintain stable system operation and avoid unnecessary data migration overhead.

[0181] Local adjustment policies are triggered when local network traffic fluctuates or when certain nodes experience abnormal load. These policies include hotspot data replication, load balancing adjustments, cache optimization, and delayed synchronization of non-critical data. These policies aim to alleviate system pressure and prevent the spread of problems through local adjustments.

[0182] Global reorganization policies are triggered when multi-region network traffic patterns change or overall system performance degrades. These policies include replanning data sharding, migrating data across regions, optimizing access paths, and reallocating resources. These policies are designed to address system-level performance challenges and restore system performance through global optimization.

[0183] The emergency migration policy, triggered when the network environment deteriorates dramatically or a node faces a risk of failure, includes measures such as emergency backup of critical data, migration of core services, activation of degraded services, and emergency resource deployment. This policy aims to safeguard the security of core system functions and data, minimizing losses.

[0184] It's important to understand that the generation of data sharding strategies not only considers the relationship between the current value of network traffic characteristics and the optimization threshold, but also the type, scope, severity, and development trend of the network traffic characteristics. For example, for network latency that also exceeds the optimization threshold, if the network traffic characteristics indicate network congestion and a trend of continued deterioration, the system will generate a higher-level sharding strategy; if it indicates a temporary traffic spike and the trend is stable, a lower-level sharding strategy may be generated.

[0185] The data sharding strategy also considers the spatial correlation between data access patterns and network traffic characteristics. For data shards within the impact range of network traffic anomalies, the system generates more proactive migration and replication strategies. For data shards outside the impact range but potentially affected, the system generates preventative monitoring and preparation strategies.

[0186] The hierarchical data sharding strategy is converted into specific execution instructions, and a personalized optimization guidance plan is generated based on the node role type and network location.

[0187] In a preferred embodiment of the present invention, differentiated optimization guidance solutions are generated for each type of node based on the node's role type (such as edge node, aggregation node, central node, and dedicated storage node) and current network location.

[0188] As examples of optimization guidance, for edge nodes, the optimization guidance focuses on resource efficiency and responsiveness, such as "Increase local cache capacity to several MB, prioritize caching hot data for users A, B, and C, use an improved LRU-K replacement algorithm, and adjust the batch commit interval for write operations to several milliseconds." These instructions are communicated through the node management interface via configuration files, API calls, or direct commands, ensuring accurate execution across different deployment environments. Simultaneously, edge nodes receive real-time analysis of data access patterns and dynamically adjust prefetching strategies and cache priorities.

[0189] For aggregation nodes, optimization guidance includes more comprehensive data routing and cache coordination strategies, such as rebalancing the data sharding ratio between regions A1 and B1, replicating hotspot dataset X to nodes 3 and 5, establishing inter-region data access indexes, and optimizing cross-region request routing paths. Furthermore, it provides load status and data access hotspot analysis for each child node, assisting aggregation nodes with resource scheduling and load balancing.

[0190] For central nodes, the optimization guidance plan includes global resource scheduling and policy coordination instructions, such as "Initiate a global data resharding plan, prioritize the migration of datasets D, E, and F, adjust the replication factor to a certain value, suspend non-core synchronization tasks, and reserve bandwidth for critical services." It also provides overall system performance trend and bottleneck analysis to help central nodes evaluate optimization results and formulate long-term strategies.

[0191] The generation of personalized optimization guidance plans takes into account the following factors: the hardware capabilities and resource limitations of the node, and reasonable optimization goals are customized based on the node's computing, storage, and network capabilities; the position and role of the node in the network topology, and different optimization strategies are adopted for core nodes and edge nodes; the current workload characteristics and service quality requirements, with key business nodes prioritizing stability, and non-critical nodes can adopt more aggressive optimization; historical optimization effect data, and based on optimization experience under similar conditions, the parameters and focus of the guidance plan are adjusted.

[0192] Through personalized optimization guidance plans, abstract sharding strategies and caching strategies can be converted into specific executable configuration changes and operation instructions, ensuring that various nodes can implement optimization measures in a coordinated and consistent manner under different network conditions, thereby maximizing system performance and resource utilization efficiency.

[0193] like Figure 3As shown, a distributed heterogeneous node optimization system includes a traffic access data collection module, a correlation analysis module, a performance judgment module, a bottleneck source identification module, and a sharding strategy generation module; wherein each module is electrically connected;

[0194] A traffic access data collection module collects network traffic feature data and data access pattern data of a distributed heterogeneous node system, and sends the network traffic feature data and data access pattern data to the correlation analysis module and the bottleneck source identification module;

[0195] an association analysis module, which performs association analysis on the network traffic feature data and the data access pattern data using a pre-trained traffic access association model, extracts key association features, and sends the key association features to the performance judgment module;

[0196] a performance judgment module that inputs the key correlation features into a pre-trained performance impact assessment model to generate a system performance impact level; based on the system performance impact level, determines whether the current system is in a performance degradation state; and if so, sends the system performance impact level to the bottleneck source identification module;

[0197] A bottleneck source identification module, which identifies the performance bottleneck source based on the network traffic feature data, the data access pattern data and the system performance impact level through a performance bottleneck identification model based on causal reasoning, and sends the performance bottleneck source to the sharding strategy generation module;

[0198] The sharding strategy generation module inputs the key correlation features into the optimization strategy generation model based on deep reinforcement learning based on the performance bottleneck source. The optimization strategy generation model dynamically adjusts the optimization threshold based on historical optimization experience and current system parameters, and generates a corresponding data sharding strategy based on the optimization threshold and network traffic characteristics.

[0199] In addition, the parts of the above technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0200] The above-described specific embodiments further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above description is merely a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

[0201] The above preset parameters or preset thresholds are all set by those skilled in the art according to actual conditions or obtained through large amounts of data simulation.

[0202] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A distributed heterogeneous node optimization method, characterized in that: Step 1: Collect network traffic characteristic data and data access pattern data of distributed heterogeneous node systems; Step 2: performing correlation analysis on the network traffic feature data and the data access pattern data using a pre-trained traffic access correlation model to extract key correlation features; Step 3: Input the key correlation features into a pre-trained performance impact assessment model to generate a system performance impact level; and determine whether the current system is in a performance degradation state based on the system performance impact level; Step 4: If the system is in a performance degradation state, identifying the source of the performance bottleneck based on the network traffic feature data, the data access pattern data, and the system performance impact level through a performance bottleneck identification model based on causal reasoning; Step 5: Based on the performance bottleneck source, the key correlation features are input into an optimization strategy generation model based on deep reinforcement learning. The optimization strategy generation model dynamically adjusts the optimization threshold based on historical optimization experience and current system parameters, and generates a corresponding data sharding strategy based on the optimization threshold and network traffic characteristics. Step 41: Based on the historical monitoring data of the distributed heterogeneous node system and the pre-labeled performance degradation event records, a second training sample library including the performance degradation type, cause, and symptom causal relationship is constructed; Step 42: Based on the second training sample library, a structured learning method is used to construct a three-layer hierarchical causal reasoning network structure including a root cause node layer, an intermediate state node layer, and an observation feature node layer; Step 43: Based on the performance degradation event records in the second training sample library, use the maximum likelihood estimation and expectation maximization algorithm to learn the conditional probability table of the causal relationship between each node in the causal reasoning network; Step 44: Based on the real-time key correlation features extracted from the collected network traffic feature data and data access pattern data, the performance bottleneck identification model uses the joint tree algorithm for probabilistic reasoning to calculate the posterior probability distribution of each root cause node and output the performance bottleneck source with the highest probability. Step 45: Based on the identified performance bottleneck source, a performance bottleneck feature library is constructed to classify and extract features of the performance bottlenecks; Step 46: Based on the constructed performance bottleneck feature library and in combination with the identified performance bottleneck source, the performance bottleneck is identified and located.

2. A distributed heterogeneous node optimization method according to claim 1, characterized in that: The method of collecting network traffic characteristic data and data access pattern data of the distributed heterogeneous node system is: The network traffic monitoring components and data access monitoring components deployed in the distributed heterogeneous node system collect the real-time network traffic characteristic data and data access pattern data of each node.

3. A distributed heterogeneous node optimization method according to claim 2, characterized in that: The network traffic monitoring component is deployed on the network switching devices, routers and computing nodes of the distributed heterogeneous node system to monitor the network traffic characteristic data consisting of packet size distribution, transmission delay, throughput and packet loss rate in real time; The computing node refers to a server or virtual machine instance that performs computing tasks in a distributed heterogeneous node system; Among them, the data access monitoring component is installed on the storage nodes and computing nodes of the distributed heterogeneous node system to monitor data access characteristic data consisting of read-write ratio, access frequency, data block size, and access mode; the storage node refers to the server or virtual machine instance that performs storage tasks in the distributed heterogeneous node system.

4. A distributed heterogeneous node optimization method according to claim 3, characterized in that: The method of performing correlation analysis on the network traffic feature data and the data access pattern data using a pre-trained traffic access correlation model to extract key correlation features includes the following steps: Step 21: Collect historical monitoring data of the distributed heterogeneous node system and build a multi-source heterogeneous data training sample library; Step 22: Construct a traffic access correlation model, which adopts a multi-flow architecture design and includes a feature extraction module, a time series modeling module, a correlation analysis module, and a feature output module; Step 23: Based on the constructed multi-source heterogeneous data training sample library, use an end-to-end supervised learning approach to train the traffic access association model and optimize network parameters. Step 24: Use the trained traffic access correlation model to process the real-time network traffic feature data and data access pattern data to extract the key correlation features of the distributed heterogeneous node system.

5. A distributed heterogeneous node optimization method according to claim 4, characterized in that: Inputting the key correlation features into a pre-trained performance impact assessment model to generate a system performance impact level comprises the following steps: Step 31: Synchronously construct a training sample library for the performance impact assessment model with the multi-source heterogeneous data training sample library; the training sample library contains two types of data: data with no performance degradation and data with performance degradation; Step 32: Design a performance impact evaluation model of a network structure including a feature extraction layer, a time series modeling layer, and a performance evaluation layer; Step 33: Based on the constructed training sample library, a supervised learning method is used to train the performance impact assessment model with the goal of optimizing performance classification; Step 34: The performance impact assessment model receives the key correlation features as input and generates a performance impact level classification result of the system performance impact level in real time.

6. A distributed heterogeneous node optimization method according to claim 5, characterized in that: The dynamic adjustment of the optimization threshold comprises the following steps: Step 51: Construct a third training sample library for the optimization strategy generation model, which includes historical performance bottleneck data, key correlation feature data, optimization threshold setting records, and optimization effect evaluation data; Step 52: Design an optimization strategy generation model based on deep reinforcement learning, including a state representation module, an action generation module, and a value assessment module; Step 53: Design an optimization effect evaluation function for the optimization strategy generation model to comprehensively evaluate performance improvement, resource utilization, system stability, and optimization overhead factors; Step 54: Based on the constructed third training sample library, a deep reinforcement learning algorithm is used to train the optimization strategy generation model; Step 55: Based on the newly identified performance bottlenecks and the newly extracted key correlation features, a model is generated through the trained optimization strategy, and the optimization threshold is adjusted in real time.

7. A distributed heterogeneous node optimization method according to claim 6, characterized in that: The training of the optimization strategy generation model includes: initializing the parameters of the strategy network and the value network, and setting the experience replay buffer; Historical data is loaded from the third training sample library in chronological order to simulate the operation of the optimization system. For each time step, an optimization threshold adjustment action is selected using an ε-greedy strategy based on the current state. This strategy randomly selects an action for exploration with a probability of ε and selects the action deemed optimal by the current strategy for utilization with a probability of 1-ε. The current state includes key correlation features and performance bottlenecks. Execute the selected action and store the four-tuple including state, action, reward, and new state into the experience replay buffer; Randomly extract samples from the experience replay buffer and use temporal difference learning to update the value network parameters; every fixed number of steps, copy the parameters of the value network to the target network. As training progresses, gradually reduce the ε value to reduce random exploration; during training, regularly evaluate the model performance on the validation dataset and monitor the learning curve. When the validation performance no longer improves or reaches the preset training rounds, stop the training process.

8. A distributed heterogeneous node optimization method according to claim 6, characterized in that: The optimization strategy generation model adopts a deep reinforcement learning framework, regards the optimization strategy generation model as an intelligent agent, optimizes the threshold adjustment as the agent's action, and optimizes the effect evaluation as the reward signal of the environment feedback.

9. A distributed heterogeneous node optimization method according to claim 8, characterized in that: Based on the optimization threshold and network traffic characteristics, generating a corresponding data sharding strategy includes: Based on the optimized threshold and identified network traffic characteristics, a data access pattern analysis model is constructed to generate a hierarchical data sharding strategy; The hierarchical data sharding strategy is converted into execution instructions, and an optimization guidance plan is generated based on the node role type and network location.

10. A distributed heterogeneous node optimization system, used to implement a distributed heterogeneous node optimization method according to any one of claims 1 to 9, characterized in that: It includes a traffic access data collection module, a correlation analysis module, a performance judgment module, a bottleneck source identification module, and a sharding strategy generation module; wherein each module is electrically connected; A traffic access data collection module collects network traffic feature data and data access pattern data of a distributed heterogeneous node system, and sends the network traffic feature data and data access pattern data to the correlation analysis module and the bottleneck source identification module; an association analysis module, which performs association analysis on the network traffic feature data and the data access pattern data using a pre-trained traffic access association model, extracts key association features, and sends the key association features to the performance judgment module; a performance judgment module that inputs the key correlation features into a pre-trained performance impact assessment model to generate a system performance impact level; based on the system performance impact level, determines whether the current system is in a performance degradation state; and if so, sends the system performance impact level to the bottleneck source identification module; A bottleneck source identification module, which identifies the performance bottleneck source based on the network traffic feature data, the data access pattern data and the system performance impact level through a performance bottleneck identification model based on causal reasoning, and sends the performance bottleneck source to the sharding strategy generation module; The sharding strategy generation module inputs the key correlation features into the optimization strategy generation model based on deep reinforcement learning based on the performance bottleneck source. The optimization strategy generation model dynamically adjusts the optimization threshold based on historical optimization experience and current system parameters, and generates a corresponding data sharding strategy based on the optimization threshold and network traffic characteristics.

Images