A data processing method, apparatus and related device

Abstract

This application provides a data processing method, apparatus, and related equipment, belonging to the field of artificial intelligence technology. One data processing method provided in this application includes: acquiring Application Performance Management (APM) data from multiple application nodes, wherein the APM data includes performance index data and link tracing data; extracting periodic features from the performance index data using a preset temporal convolutional network model, generating a low-dimensional tensor based on the periodic features, processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix; concatenating the low-dimensional tensor and the sparse dependency matrix to obtain a joint encoding vector, generating a discrete symbol sequence based on the joint encoding vector using a preset dynamic quantizer; and processing the discrete symbol sequence to obtain processed data. This method can improve data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus and related equipment. Background Technology

[0002] During peak business periods, application performance management data generated by various application nodes experiences explosive growth. When hardware resources (such as high-performance servers, large-capacity storage devices, and high-speed network bandwidth) are limited or cannot be expanded in a short time, processing this massive amount of data (such as compression, storage, and analysis) will greatly increase data processing time, resulting in low data processing efficiency. Summary of the Invention

[0003] This application provides a data processing method, apparatus, and related equipment that can solve the technical problem of low data processing efficiency.

[0004] In a first aspect, embodiments of this application provide a data processing method, the method comprising:

[0005] Acquire application performance management (APM) data from multiple application nodes, wherein the APM data includes performance metric data and link tracing data;

[0006] The periodic features of the performance index data are extracted using a preset temporal convolutional network model. A low-dimensional tensor is generated based on the periodic features. The link tracing data is then processed using a preset graph attention network model to obtain a sparse dependency matrix.

[0007] The low-dimensional tensor and the sparsed dependency matrix are concatenated to obtain a joint encoding vector, and a discrete symbol sequence is generated based on the joint encoding vector using a preset dynamic quantizer.

[0008] The discrete symbol sequence is processed to obtain processed data.

[0009] Optionally, generating a low-dimensional tensor based on the periodicity feature includes:

[0010] The periodic features are filtered for noise using the temporal convolutional network model to obtain intermediate features;

[0011] The intermediate features are pooled using the temporal convolutional network model according to the pooling window to obtain a low-dimensional tensor.

[0012] The size of the pooling window is determined by the sampling frequency at which the APM data is acquired.

[0013] Optionally, the step of processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix includes:

[0014] A service call relationship graph is constructed based on the link tracing data, and the graph nodes of the service call relationship graph are constructed based on the multiple application nodes;

[0015] Based on the preset graph attention network model and the service call relationship graph, the adjacency matrix of multiple graph nodes in the service call relationship graph is determined respectively;

[0016] Based on the adjacency matrix, spectral clustering is performed to obtain the similarity scores of the multiple graph nodes.

[0017] The graph nodes whose similarity scores do not exceed the preset first threshold are kept as they are, and the graph nodes whose similarity scores exceed the preset first threshold are merged to obtain a compressed service call relationship graph.

[0018] A sparse dependency matrix is ​​generated based on the compressed service call relationship graph.

[0019] Optionally, processing the discrete symbol sequence to obtain processed data includes:

[0020] Obtain the processor utilization, memory utilization, and network bandwidth usage of multiple storage nodes respectively;

[0021] The load coefficients of the multiple storage nodes are obtained by weighted summation of the processor utilization, memory utilization, and network bandwidth utilization.

[0022] If the load factor of the target storage node is less than a preset second threshold, the discrete symbol sequence is stored in the target storage node in a round-robin sharding manner; if the load factor of the target storage node is greater than or equal to the preset second threshold and less than a preset third threshold, the discrete symbol sequence is stored in the target storage node in a consistent hash sharding manner; if the load factor of the target storage node is greater than or equal to the preset third threshold, the discrete symbol sequence is cut into multiple sub-data blocks according to a preset time window, the sub-data blocks located within the preset time range are stored in the storage nodes with load factors less than the second threshold, and the sub-data blocks located outside the preset time range are migrated to a preset storage device.

[0023] The target storage node is one or more of the plurality of storage nodes.

[0024] Optionally, the step of extracting periodic features from the performance index data using a preset temporal convolutional network model, generating a low-dimensional tensor based on the periodic features, and processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix includes:

[0025] The data compression ratio is determined based on the ratio of the amount of APM data to the amount of discrete symbol sequence data.

[0026] Obtain system load parameters, wherein the system load parameters are the system load parameters corresponding to the device executing the data processing method;

[0027] When the data compression ratio is less than a preset fourth threshold and the system load parameter is greater than a preset fifth threshold: the convolution stride of the temporal convolutional network model is adjusted to a preset sixth threshold to obtain an adjusted temporal convolutional network model. The periodic features of the performance index data are extracted using the adjusted temporal convolutional network model. A low-dimensional tensor is generated based on the periodic features. The first threshold is adjusted to a preset seventh threshold. Graph nodes whose similarity scores do not exceed the preset seventh threshold are kept as they are, and graph nodes whose similarity scores exceed the seventh threshold are merged to obtain a compressed service call relationship graph. A sparse dependency matrix is ​​generated based on the compressed service call relationship graph.

[0028] When the data compression ratio is greater than a preset seventh threshold and the system load parameter is less than a preset eighth threshold: the periodic features of the performance index data are extracted using a preset temporal convolutional network model, and a low-dimensional tensor is generated based on the periodic features. During the process of processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix, the spectral clustering operation based on the adjacency matrix is ​​stopped, and a sparse dependency matrix is ​​generated based on the service call relationship graph.

[0029] When the following conditions are met simultaneously: the data compression ratio is greater than or equal to the fourth threshold, the data compression ratio is less than or equal to the preset seventh threshold, the system load parameter is greater than or equal to the preset eighth threshold, and the system load parameter is less than or equal to the preset fifth threshold: the periodic features of the performance index data are extracted using a preset temporal convolutional network model, a low-dimensional tensor is generated based on the periodic features, and the link tracing data is processed using a preset graph attention network model to obtain a sparse dependency matrix.

[0030] Optionally, the method further includes:

[0031] Get the query request;

[0032] Determine the required query precision based on the query request;

[0033] Given the required query accuracy, the discrete symbol sequence is deconvolutionally decoded to recover the key trend points of the performance index data.

[0034] When the query accuracy requirement is the same as the depth requirement, topology recovery is performed on the discrete symbol sequence to obtain link tracing reconstruction data and / or performance indicator reconstruction data.

[0035] Optionally, the APM data is the APM data after data feature vector annotation;

[0036] The data feature vectors include: timestamp feature vector, service name feature vector, call level feature vector, and host tag feature vector.

[0037] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0038] The acquisition module is used to acquire application performance management (APM) data from multiple application nodes. The APM data includes performance metric data and link tracing data.

[0039] The first processing module is used to extract the periodic features of the performance index data using a preset temporal convolutional network model, generate a low-dimensional tensor based on the periodic features, and process the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix.

[0040] The second processing module is used to concatenate the low-dimensional tensor and the sparsed dependency matrix to obtain a joint encoding vector, and use a preset dynamic quantizer to generate a discrete symbol sequence based on the joint encoding vector.

[0041] The third processing module is used to process the discrete symbol sequence to obtain processed data.

[0042] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the data processing method as described in the first aspect.

[0043] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method described in the first aspect.

[0044] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the data processing method as described in the first aspect.

[0045] In this embodiment, a low-dimensional tensor is generated using a preset temporal convolutional network model, and a sparse dependency matrix is ​​obtained using a preset graph attention network model, thereby generating a discrete symbol sequence. Since the generated discrete symbol sequence has a significantly reduced data volume compared to the APM, it can reduce resource overhead during data processing and thus improve data processing efficiency. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is one of the flowcharts of a data processing method provided in the embodiments of this application;

[0048] Figure 2 This is a second flowchart of a data processing method provided in an embodiment of this application;

[0049] Figure 3 This is one of the flowcharts of a data processing method provided in an embodiment of this application;

[0050] Figure 4 This is a second partial flowchart of a data processing method provided in an embodiment of this application;

[0051] Figure 5 This is a partial flowchart of a data processing method provided in an embodiment of this application;

[0052] Figure 6 This is a fourth partial flowchart of a data processing method provided in an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0054] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, "and / or" in this application indicates at least one of the connected objects. For example, the scope of protection of "A and / or B" covers at least three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. Additionally, the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0057] See Figure 1 , Figure 1 This is one of the flowcharts of a data processing method provided in the embodiments of this application, such as... Figure 1 As shown, the method includes the following steps:

[0058] Step 101: Obtain application performance management (APM) data from multiple application nodes, wherein the APM data includes performance metric data and link tracing data;

[0059] The multiple application nodes can be nodes (such as an instance, a container, and / or a service) in a system or network that are responsible for running business applications and processing business logic.

[0060] The APM data can be Figure 2 The multi-dimensional APM data can specifically include performance metric data and link tracing data. The performance metric data can include metrics such as Central Processing Unit (CPU) utilization, memory utilization, network bandwidth usage, interface response time, and data throughput. In this application, the performance metric data can also be referred to as performance data. The link tracing data can include data such as the start time, end time, call path, request parameters, and response results of the service calls corresponding to the multiple application nodes. In this application, the link tracing data can also be referred to as link data.

[0061] The APM data obtained from multiple application nodes can be obtained through probes pre-deployed on the multiple application nodes.

[0062] In this step, APM data from multiple application nodes are acquired to obtain performance index data and link tracing data for these multiple application nodes, providing a data foundation for data processing.

[0063] Step 102: Extract the periodic features of the performance index data using a preset temporal convolutional network model, generate a low-dimensional tensor based on the periodic features, and process the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix.

[0064] The Temporal Convolutional Network (TCN) model can be a neural network model employing multiple layers of temporal convolutional networks. These multiple layers of temporal convolutional networks (e.g., could be...) Figure 3 The 5-layer temporal convolutional network in the model can adopt a causal convolutional structure;

[0065] The method utilizes a preset temporal convolutional network model to extract the periodic features of the performance index data, and generates a low-dimensional tensor based on the periodic features. This significantly reduces the amount of data while preserving the main periodic and trend characteristics of the performance index data.

[0066] The Graph Attention Network (GAT) model can be a graph neural network model based on the attention mechanism. It can learn the representation vector of a node or graph by assigning adaptive attention weights to the neighbors of each node in the graph structure data and weighting and aggregating the features of neighboring nodes. It can be used for node classification, link prediction, graph classification, etc.

[0067] In this step, a preset temporal convolutional network model is used to extract the periodic features of the performance index data, and a low-dimensional tensor is generated based on the periodic features, which can compress the data volume of the performance index data; a preset graph attention network model is used to process the link tracing data to obtain a sparse dependency matrix, which can compress the data volume of the link tracing data.

[0068] Step 103: Concatenate the low-dimensional tensor and the sparsed dependency matrix to obtain a joint encoding vector, and use a preset dynamic quantizer to generate a discrete symbol sequence based on the joint encoding vector;

[0069] Wherein, concatenating the low-dimensional tensor and the sparsed dependency matrix to obtain the joint encoding vector can be achieved by using the low-dimensional tensor... and sparse dependency matrix Flatten and concatenate to obtain the joint encoding vector. This achieves the fusion of features from the time domain and the structural domain.

[0070] Please see Figure 3The dynamic quantizer can be an algorithm model that can dynamically quantize parameters (e.g., compress floating-point data into low-bit integers to reduce data volume); the dynamic quantizer can include an entropy-aware clustering model and a Gumbel-Softmax model based on the normalized exponential function of the Gumbel distribution.

[0071] Specifically, in the entropy-aware quantization stage, the complexity of the jointly encoded features can be measured based on Shannon entropy, thereby dynamically quantizing the number of cluster centers K in the process. The specific calculation can be referred to the following first formula:

[0072]

[0073] In the first formula, It can represent the number of cluster centers; It can identify the entropy value of the joint coding vector; and It can represent a pre-defined entropy interval boundary (e.g., an entropy interval boundary set by those skilled in the art based on experience); for example , ,make sure The dynamic adjustment range is within a certain preset range value (e.g., [8,64]). The function can represent a rounding function (e.g., a round-up function);

[0074] In the clustering stage, the K-means clustering algorithm can be used, which introduces a soft attribution mechanism to cluster each vector. Assigned to different cluster centers probability For the specific calculation formula, please refer to the second formula below:

[0075]

[0076] In the second formula, Vectors can be represented Assigned to cluster center The probability of; It can represent preset parameters and can be used to control the sharpness of the probability distribution; It can represent the Euclidean distance metric, which can be used to represent the similarity between a sample and its center;

[0077] Furthermore, during training, the Gumbel-Softmax model can be used to approximate sampling to maintain gradient transferability, and during the inference stage, the center index with the highest probability can be directly selected to generate a discrete symbol sequence. The discrete symbols can be used as compressed outputs and input to a distributed storage system to achieve efficient encoding representation of joint encoded features.

[0078] In this step, a preset dynamic quantizer is used to generate a discrete symbol sequence based on the joint coding vector, thereby achieving efficient compression while maintaining the integrity of the semantic information of the joint coding vector, significantly reducing the computational overhead of data processing, and thus improving data processing efficiency.

[0079] Step 104: Process the discrete symbol sequence to obtain processed data.

[0080] In this step, the processing of the discrete symbol sequence may include one or more of the following: storage, transmission, analysis, conversion, modeling, reasoning, and decoding of the discrete symbol sequence.

[0081] In this step, the discrete symbol sequence is processed to obtain processed data. Since the amount of data in the discrete symbol sequence has been reduced relative to the APM data, the resource overhead in the data processing process can be reduced, thereby improving data processing efficiency.

[0082] In this embodiment, a low-dimensional tensor is generated using a preset temporal convolutional network model, and a sparse dependency matrix is ​​obtained using a preset graph attention network model, thereby generating a discrete symbol sequence. Since the generated discrete symbol sequence has a smaller data volume than the APM, it can reduce the resource overhead in the data processing process and thus improve the data processing efficiency.

[0083] In some implementations, please refer to Figure 2 And Figure 3, Figure 2 This is the second flowchart of a data processing method provided in the embodiments of this application. Figure 3 This is one of the flowcharts of a data processing method provided in an embodiment of this application, such as... Figure 2 and Figure 3 As shown:

[0084] The APM data is the APM data after data feature vector annotation;

[0085] The data feature vectors include: timestamp feature vector, service name feature vector, call level feature vector, and host tag feature vector.

[0086] The data feature vector may include Figure 2 and Figure 3 The four-dimensional feature vector in;

[0087] The timestamp feature vector can be a feature vector corresponding to the timestamp of the APM data;

[0088] The service name feature vector can be a feature vector corresponding to a string identifier used to uniquely identify an application node;

[0089] The call hierarchy feature vector can be used to characterize the call depth of the service corresponding to the application node in the entire call chain;

[0090] The host tag feature vector can be a feature vector in APM data used to identify the host identity information corresponding to the application node. The host tag can include one or more of the following: hostname, Internet Protocol (IP) address, and host name.

[0091] It is understood that the APM data is the APM data after data feature vector annotation; that is, the performance index data can be the performance index data after data feature vector annotation, and the link tracing data can be the link tracing data after data feature vector annotation.

[0092] Please refer to Figure 2 The process of obtaining the joint encoding vector and the discrete symbol sequence described above can also be a data compression process. Figure 2 The dual-channel neural network can be the temporal convolutional network model and the graph attention network model in step 102;

[0093] exist Figure 3 In this context, feature splitting can be achieved by processing the labeled APM data separately.

[0094] In this embodiment, the APM data is the APM data after data feature vector annotation, which can provide clear inference guidance for the above-mentioned temporal convolutional network model and / or the above-mentioned graph attention network model; thereby effectively reducing the number of inference steps of the model, speeding up the processing speed of the above-mentioned temporal convolutional network model and / or the above-mentioned graph attention network model for their respective APM data, and thus improving data processing efficiency.

[0095] In some implementations, generating a low-dimensional tensor based on the periodic feature includes:

[0096] The periodic features are filtered for noise using the temporal convolutional network model to obtain intermediate features;

[0097] The intermediate features are pooled using the temporal convolutional network model according to the pooling window to obtain a low-dimensional tensor.

[0098] The size of the pooling window is determined by the sampling frequency at which the APM data is acquired.

[0099] For details, please see Figure 3The method involves extracting periodic features from the performance index data using a preset temporal convolutional network model, and then using the temporal convolutional network model to filter noise from these periodic features to obtain intermediate features. Specifically, this can be achieved by introducing a gating mechanism into the convolution calculation of each layer in the aforementioned TCN model (which may include main channel convolutional kernels and gated channel convolutional kernels) to obtain intermediate features. For detailed calculations, please refer to the third formula below:

[0100]

[0101] In the third formula, It can represent Intermediate features of the layer output; It can represent The intermediate features output by the previous temporal convolutional network (specifically, these may include intermediate features corresponding to performance metrics such as CPU utilization trends and memory fluctuation patterns collected at the previous time step). It can represent the main channel convolution kernel of the current layer and can be used to capture the core trends of indicator data, such as the periodic changes in interface response time, the long-term growth trend of throughput, and other key performance indicators. It can represent gated channel convolution kernels and can dynamically filter noise (e.g., preset non-critical features, low-weight features and / or features corresponding to interference signals that change with the environment, such as features corresponding to redundant information such as index fluctuations caused by instantaneous network jitter and sudden but brief CPU peaks).

[0102] In the third formula, the hyperbolic tangent tanh function can enhance the capture of stationary and trend changes in indicator data through nonlinear transformation, such as the slow upward trend of memory usage and the periodic fluctuations of interface response time; normalization The (Sigmoid) function can generate gating weights between 0 and 1, which can adaptively adjust the intensity of information flow. For example, it can assign low weights to sudden abnormal indicators (such as instantaneous high CPU utilization) to reduce interference, and assign high weights to stable trends to enhance stability. It can represent element-wise multiplication operations, realize the fusion of main channel characteristics and gating weights, and accurately retain the indicator characteristics at key time points, such as the peak throughput during business peak periods and the sudden change point in response time during faults.

[0103] More specifically, the step of using the temporal convolutional network model to perform pooling operations on the intermediate features according to a pooling window to obtain a low-dimensional tensor can be achieved by performing an adaptive max pooling operation immediately after each convolutional layer; the pooling window size... The mapping frequency is determined based on the sampling frequency of the acquired APM data to ensure that the low-dimensional tensor retains its main trend structure at different sampling rates; the specific mapping rule can be referred to the following fourth formula:

[0104]

[0105] In the fourth formula, It can represent the size of the pooling window (e.g.) The size of the pooling window is indicated by The size of the pooling window is determined by the sampling frequency at which the APM data is acquired;

[0106] More specifically, in the fourth formula, This can directly correspond to the sampling frequency of probe metrics on the aforementioned multiple application nodes, such as CPU utilization being sampled once per second (1Hz), and interface response time being sampled once every 100 milliseconds (10Hz), etc. In this case, Set to 5 to preserve fine-grained features; in In this case, Set to 10; in In this case, Setting it to 20 ensures that the compressed representation retains the main trend under different sampling rates.

[0107] In the third and fourth formulas mentioned above, , , and It can be a required parameter that can directly relate to the processing effect of performance metrics data in APM data. These can be dynamic parameters and can be dependent on... The adaptive changes do not require a fixed input; the size of the pooling window is adjusted according to the sampling frequency of the APM data, so that even in high-frequency scenarios (i.e., scenarios where data changes very frequently), the low-dimensional tensor obtained after compression can still reflect the subtle changes in the data in detail, while in low-frequency scenarios, it focuses more on macroscopic change features.

[0108] More specifically, the output of the aforementioned TCN model can be passed through a single layer. Convolution performs dimensionality compression, reducing the original data size to 1 / 8; the result is the aforementioned low-dimensional tensor. This enables significant dimensionality reduction of performance index data, greatly reducing the amount of data.

[0109] In this embodiment, the temporal convolutional network model is used to perform noise filtering and pooling operations to obtain a low-dimensional tensor. The size of the pooling window is determined by the sampling frequency of the APM data. On the one hand, this can suppress noise while ensuring the periodicity and trend characteristics of the performance index data, greatly reducing the amount of performance index data and improving data processing efficiency. On the other hand, the size of the pooling window is determined by the sampling frequency of the APM data, which can improve the adaptability of the data processing method.

[0110] In some implementations, the step of processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix includes:

[0111] A service call relationship graph is constructed based on the link tracing data, and the graph nodes of the service call relationship graph are constructed based on the multiple application nodes;

[0112] Based on the preset graph attention network model and the service call relationship graph, the adjacency matrix of multiple graph nodes in the service call relationship graph is determined respectively;

[0113] Based on the adjacency matrix, spectral clustering is performed to obtain the similarity scores of the multiple graph nodes.

[0114] The graph nodes whose similarity scores do not exceed the preset first threshold are kept as they are, and the graph nodes whose similarity scores exceed the preset first threshold are merged to obtain a compressed service call relationship graph.

[0115] A sparse dependency matrix is ​​generated based on the compressed service call relationship graph.

[0116] Among them, see Figure 3 Determining the adjacency matrix of multiple graph nodes in the service call relationship graph may include: the graph attention network determines the adjacency matrix of multiple graph nodes in the service call relationship graph based on a preset dynamic topology awareness strategy; for example, the initial weight of an edge with a link latency exceeding 100 milliseconds can be set to 0.8; the initial weight of an edge with a call error rate higher than 1% can be set to 1.2 to highlight the influence of the abnormal link in the graph construction process; in the multi-head attention calculation stage, each attention head k assigns a different attention coefficient to the edge between node i and its neighbor node j, and the specific calculation can be referred to the following fifth formula:

[0117]

[0118] Prior weights for link attributes (such as latency and error rate) can also be added to the calculation of the fifth formula. For specific calculations, please refer to the sixth formula below:

[0119]

[0120] In the fifth and sixth formulas mentioned above, It can represent the attention coefficient assigned by attention head k to the edge between node i and its neighbor node j, and can be used to represent the adaptive edge weight between node i and its neighbor node j in the service call relationship graph. Prior weights can be used to represent link attributes (such as link latency or error rate), making the attention mechanism more suitable for APM link anomaly identification requirements; for example, when the link latency is greater than 100ms, When the error rate is greater than 1%, ; and Nodes can be represented separately. and eigenvectors; It can represent the first A trainable linear transformation matrix of size; It can represent feature concatenation operations;

[0121] In the fifth and sixth formulas mentioned above, It can be represented as a trainable parameter vector projected as attention coefficients; It can represent a node The set of adjacent nodes; Leaky Rectified LinearUnit (LeakyReLU function can be used as an activation function to introduce an asymmetric response, which is beneficial for handling sparse connection characteristics.

[0122] The step of performing spectral clustering based on the adjacency matrix to obtain similarity scores for the plurality of graph nodes may include: performing spectral clustering based on the adjacency matrix to obtain the representations of the plurality of graph nodes in the low-dimensional spectral embedding space, and thereby determining the similarity scores between the plurality of graph nodes.

[0123] Specifically, after the multi-head attention mechanism is completed, the output layer can use spectral clustering to identify highly similar service nodes in the spectral embedding space; nodes with similarity scores exceeding a preset first threshold (e.g., 0.9) are merged, thereby compressing the original service call topology and generating the aforementioned sparse dependency matrix. It can achieve a sparsity rate of 60% to 85%; this significantly reduces the amount of data in the sparsity dependency matrix and the amount of data in the discrete symbol sequence, improves the efficiency of data processing, and can also enhance the expressive concentration of the critical path.

[0124] In this embodiment, a compressed service call relationship graph is obtained by using a graph attention network model and spectral clustering operation; a sparse dependency matrix is ​​generated based on the compressed service call relationship graph, and then a discrete symbol sequence is obtained, which can greatly reduce the amount of link tracing data, reduce the resource overhead of data processing, and thus improve data processing efficiency.

[0125] In some embodiments, processing the discrete symbol sequence to obtain processed data includes:

[0126] Obtain the processor utilization, memory utilization, and network bandwidth usage of multiple storage nodes respectively;

[0127] The load coefficients of the multiple storage nodes are obtained by weighted summation of the processor utilization, memory utilization, and network bandwidth utilization.

[0128] If the load factor of the target storage node is less than a preset second threshold, the discrete symbol sequence is stored in the target storage node in a round-robin sharding manner; if the load factor of the target storage node is greater than or equal to the preset second threshold and less than a preset third threshold, the discrete symbol sequence is stored in the target storage node in a consistent hash sharding manner; if the load factor of the target storage node is greater than or equal to the preset third threshold, the discrete symbol sequence is cut into multiple sub-data blocks according to a preset time window, the sub-data blocks located within the preset time range are stored in the storage nodes with load factors less than the second threshold, and the sub-data blocks located outside the preset time range are migrated to a preset storage device.

[0129] The target storage node is one or more of the plurality of storage nodes.

[0130] Among these, the storage nodes can refer to server or device nodes used to store discrete symbol sequences, and can be physical servers, virtual machine nodes, or other computing nodes with data storage capabilities; the multiple storage nodes can be used to store and / or process data in a distributed environment;

[0131] The plurality of storage nodes can be distributed storage nodes corresponding to each of the plurality of application nodes, or they can be distributed storage nodes deployed independently of the plurality of application nodes.

[0132] The preset storage device can be a local storage medium on a terminal device or server, such as a disk, solid-state drive, or flash memory, which is different from the multiple storage nodes mentioned above; the preset storage device can be an object storage system with sufficient storage capacity.

[0133] In this embodiment, the storage method of the discrete symbol sequence in the target storage node is adjusted according to the load coefficient of the multiple storage nodes. This enables the fragmented storage of the discrete symbol sequence under the condition of limited storage resources. This allows the fragmented data to be distributed more evenly among multiple storage nodes, avoiding access latency and performance bottlenecks caused by overload of a single storage node. This effectively reduces the dependence on hardware storage resources, ensures that the discrete symbol sequence achieves load balancing and resource utilization optimization in the distributed storage process, and further improves the efficiency of data processing.

[0134] For example, please see Figure 4 , Figure 4 This is a second partial flowchart of a data processing method provided in an embodiment of this application, such as... Figure 4 As shown, processing the discrete symbol sequence to obtain processed data may include the following steps:

[0135] For a discrete symbol sequence (which may be an input discrete symbol sequence); monitor the load factor of the nodes (i.e., the multiple storage nodes); and store the data in different storage nodes according to a preset sharding strategy based on the load factor.

[0136] Specifically, the operating status of the monitoring nodes is used to obtain the processor utilization of the multiple storage nodes. Memory usage and network bandwidth usage ratio The processor utilization rate of the multiple storage nodes Memory usage and network bandwidth usage ratio For weighted summation, please refer to the seventh formula below:

[0137]

[0138] In the seventh formula, It can represent the load factor of a storage node; , and These can be represented as processor utilization, memory utilization, and network bandwidth usage, respectively. It can represent the remaining memory percentage; , and They can be represented separately , and The corresponding weights can be set as needed by those skilled in the art; , and The values ​​can be 0.6, 0.3, and 0.1 respectively, reflecting the priority given to the consumption of computing resources.

[0139] For more details, please see Figure 4 ,exist If the value is less than a preset second threshold (which can be selected as 0.4), it indicates that the target storage node is in an idle state. The round-robin sharding strategy is then enabled, and data blocks are allocated sequentially (the data blocks can be data blocks obtained by sharding the discrete symbol sequence according to preset rules). The data blocks are then stored in the target storage node to achieve balanced use of storage resources.

[0140] If the load factor is greater than or equal to a preset second threshold, and the load factor of the target storage node is less than a preset third threshold (the third threshold can be selected as 0.7, i.e.) In the case of a moderate load on the target storage node, a consistent hashing algorithm is used for sharding, and the data blocks are stored in the target storage node to achieve balanced use of storage resources.

[0141] When the load factor of the target storage node is greater than or equal to the preset third threshold (i.e.) In the event that the target storage node is under high load, an emergency sharding mode is immediately activated. The discrete symbol sequence is divided into multiple sub-data blocks according to a fixed time window. Sub-data blocks within a preset time range (e.g., within the last 5 minutes) are forwarded to low-load nodes (storage nodes with a load coefficient less than the second threshold) to ensure the performance of hot data access. At the same time, historical sub-data blocks (e.g., sub-data blocks outside the last 5 minutes) are migrated to an object storage system with sufficient capacity to complete the migration of historical data to object storage. This ensures that the service availability and write latency of the core storage node are minimized during high-pressure operation.

[0142] For more details, please see Figure 4 Storing the data block (which may be a data block obtained by fragmenting the discrete symbol sequence according to a preset rule) may involve attaching a consistency hash check code to the data block to obtain an appended data block, and then storing the appended data block.

[0143] In some implementations, the step of extracting periodic features from the performance metric data using a preset temporal convolutional network model, generating a low-dimensional tensor based on the periodic features, and processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix includes:

[0144] The data compression ratio is determined based on the ratio of the amount of APM data to the amount of discrete symbol sequence data.

[0145] Obtain system load parameters, wherein the system load parameters are the system load parameters corresponding to the device executing the data processing method;

[0146] When the data compression ratio is less than a preset fourth threshold and the system load parameter is greater than a preset fifth threshold: the convolution stride of the temporal convolutional network model is adjusted to a preset sixth threshold to obtain an adjusted temporal convolutional network model. The periodic features of the performance index data are extracted using the adjusted temporal convolutional network model. A low-dimensional tensor is generated based on the periodic features. The first threshold is adjusted to a preset seventh threshold. Graph nodes whose similarity scores do not exceed the preset seventh threshold are kept as they are, and graph nodes whose similarity scores exceed the seventh threshold are merged to obtain a compressed service call relationship graph. A sparse dependency matrix is ​​generated based on the compressed service call relationship graph.

[0147] When the data compression ratio is greater than a preset seventh threshold and the system load parameter is less than a preset eighth threshold: the periodic features of the performance index data are extracted using a preset temporal convolutional network model, and a low-dimensional tensor is generated based on the periodic features. During the process of processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix, the spectral clustering operation based on the adjacency matrix is ​​stopped, and a sparse dependency matrix is ​​generated based on the service call relationship graph.

[0148] When the following conditions are met simultaneously: the data compression ratio is greater than or equal to the fourth threshold, the data compression ratio is less than or equal to the preset seventh threshold, the system load parameter is greater than or equal to the preset eighth threshold, and the system load parameter is less than or equal to the preset fifth threshold: the periodic features of the performance index data are extracted using a preset temporal convolutional network model, a low-dimensional tensor is generated based on the periodic features, and the link tracing data is processed using a preset graph attention network model to obtain a sparse dependency matrix.

[0149] The data compression ratio is determined based on the ratio of the APM data volume to the discrete symbol sequence data volume. The specific calculation can be found in the following formula (Eighth Formula):

[0150]

[0151] In the eighth formula, This can represent the data compression ratio; the... It can be initial It can also be the one marked above. The amount of data can be selected by those skilled in the art as needed.

[0152] In this embodiment, the parameters corresponding to the temporal convolutional network model and the graph attention network model are adjusted according to the data compression ratio and the system load, thereby adjusting the data volume of the low-dimensional tensor and the sparse dependency matrix, and further adjusting the data volume of the discrete symbol sequence. This can optimize the system load and data processing efficiency, and improve data processing efficiency.

[0153] For example, please see Figure 5 , Figure 5 This is a partial flowchart of a data processing method provided in an embodiment of this application, such as... Figure 5 As shown:

[0154] When the compression ratio is less than the fourth threshold (in Figure 5 The fourth threshold can be set to 15, i.e. ), and the system load parameter is less than the preset fifth threshold (in Figure 5 When the fifth threshold is selected as 70%, an enhanced compression mode is triggered; the enhanced compression mode specifically includes: adjusting the convolution stride of the temporal convolutional network (TCN) to a preset sixth threshold (in the case of...). Figure 5 The sixth threshold can be selected as 2) to expand the receptive field of view and increase the downsampling amplitude, thereby enhancing the compression intensity; at the same time, the first threshold corresponding to the spectral clustering operation in the Graph Attention Network (GAT) is increased to 0.95 to further sparsify the topology and reduce redundant edges; the system load parameters are also in Figure 5 The terms "system load" and "load" are abbreviated as "system load" and "load" in Chinese.

[0155] When the data compression ratio is greater than a preset seventh threshold (in Figure 5 The seventh threshold can be set to 50, i.e. ), and the system load parameter is less than the preset eighth threshold (in Figure 5 When the eighth threshold can be set to 40%, spectral clustering based on the adjacency matrix is ​​stopped, and a sparse dependency matrix is ​​generated according to the service call relationship graph to reduce the computational burden and optimize the decoding efficiency described later.

[0156] When the following conditions are met simultaneously: the data compression ratio is greater than or equal to the fourth threshold, the data compression ratio is less than or equal to the preset seventh threshold, the system load parameter is greater than or equal to the preset eighth threshold, and the system load parameter is less than or equal to the preset fifth threshold (i.e.) 50, and 70%), using a preset temporal convolutional network model to extract periodic features of the performance index data, generating a low-dimensional tensor based on the periodic features, and using a preset graph attention network model to process the link tracing data to obtain a sparse dependency matrix, while keeping the parameters of the original temporal convolutional network model and the graph attention network model unchanged.

[0157] It is understood that the system load parameters can characterize resource consumption; please refer to... Figure 2 By using this resource consumption closed-loop control mechanism, the system load and the data compression ratio are obtained in real time. The parameters of the temporal convolutional network model and the graph attention network model are adjusted, the compression strategy is dynamically adjusted, and a discrete symbol sequence corresponding to the data volume is generated, which can optimize the system load and data processing efficiency.

[0158] In some embodiments, the method further includes:

[0159] Get the query request;

[0160] Determine the required query precision based on the query request;

[0161] Given the required query accuracy, the discrete symbol sequence is deconvolutionally decoded to recover the key trend points of the performance index data.

[0162] When the query accuracy requirement is the same as the depth requirement, topology recovery is performed on the discrete symbol sequence to obtain link tracing reconstruction data and / or performance indicator reconstruction data.

[0163] The key trend points of the performance index data may include abrupt changes, fluctuations and / or abnormal inflection points in the performance index data, as well as periodic stable points, extreme points and trend reversal points.

[0164] The link tracing reconstruction data can be reconstruction data that exists in a graph structure;

[0165] In this embodiment, when the query accuracy requirement is a basic requirement, restoring the key trend points of the performance indicator data and preserving the main trend fluctuations and change structures can significantly reduce the amount of output data and improve data processing efficiency. When the query accuracy requirement is a depth requirement, topology recovery is performed on the discrete symbol sequence to obtain link tracing reconstruction data and / or performance indicator reconstruction data, improving the integrity of the reconstructed data. In summary, selecting different data processing methods according to the query request not only improves the flexibility of data processing but also optimizes the data processing flow, reduces unnecessary computational overhead, and thus improves data processing efficiency.

[0166] For example, please see Figure 6 , Figure 6 This is a partial flowchart of a data processing method provided in an embodiment of this application, as shown below. Figure 6 As shown, the method further includes:

[0167] Accept query requests, which may be user-input queries or queries automatically generated by other applications or programs;

[0168] The relevant fragments are located using a scheduler, where the relevant fragments can be the fragments corresponding to the discrete symbol sequence; the scheduler can be a preset scheduler used to locate the relevant fragments.

[0169] Call the decoder;

[0170] Given the aforementioned query accuracy requirement as the basic requirement (i.e.) Figure 6 In the case of basic decoding mode, deconvolution decoding is performed on the discrete symbol sequence to recover the key trend points of the performance index data;

[0171] When the query precision requirement is a depth requirement (i.e.) Figure 6 In the case of medium-depth decoding mode, topology recovery is performed on the discrete symbol sequence to obtain link tracing reconstruction data (i.e., link topology) and / or performance indicator reconstruction data (i.e., critical path indicators).

[0172] For details, please refer to Figure 6 In the basic decoding mode, the decoder can be a deconvolutional network structure of a temporal convolutional network (e.g., Figure 6 A lightweight deconvolutional decoder is used to perform preliminary reconstruction of the discrete symbol sequence. This decoder can consist of three one-dimensional deconvolutional layers, with kernel sizes of 5, 3, and 3 for each layer, corresponding to 64, 32, and 1 output channels, respectively. It also achieves upsampling rates of 2, 2, and 1, thereby gradually recovering key trend points and / or key trends in the performance index data.

[0173] After each deconvolutional layer outputs, keypoint preservation modules are inserted between layers to improve the restoration accuracy of key regions. A first-order difference operation can be performed on each output segment to calculate the rate of change of adjacent points. For specific calculations, please refer to the ninth formula below:

[0174]

[0175] In the ninth formula, It can represent the current time step (e.g., time step). ) and the previous time step (e.g., time step ( The difference in performance metrics data is used to quantify the rate of change of performance metrics data, such as the instantaneous increase in CPU utilization, the fluctuation range of response time, etc. This represents the numerical value of the performance metric data at the current time step, corresponding to a specific performance metric at time [time value missing]. The specific values, for example Real-time monitoring data such as CPU utilization, memory usage, or interface response time; This represents the value of the performance index data at the previous adjacent time step, that is, the same performance index at time ( The value of ) reflects the historical state of the performance index data;

[0176] Statistics of all Standard deviation The absolute value of the difference is greater than The location (i.e., the key trend point); these points often correspond to abrupt changes, fluctuations, or abnormal inflection points in the original indicator.

[0177] In subsequent training, weighted backpropagation is used to enhance the gradient response in the neighborhood of key trend points, guiding the deconvolution decoder to allocate more reconstruction capabilities at these locations.

[0178] The method further includes: performing piecewise linear interpolation on the key trend point and the two sampling points before and after the key trend point as retained points to obtain the interpolation function and the key trend; other positions are reconstructed through linear fitting. The specific interpolation function can be found in the following formula (tenth formula):

[0179]

[0180] In the tenth formula, It can represent the reconstructed value of intermediate interpolation points; and It can be the values ​​of two adjacent retained points. and for and Corresponding time step.

[0181] This approach can significantly reduce the amount of output data to 5% to 10% of performance metrics while still preserving the main trends, fluctuations, and structural changes, meeting the performance requirements of fast querying and edge computing scenarios, and providing an efficient starting point for subsequent deep decoding.

[0182] For details, please refer to Figure 6 In deep decoding mode, a trained graph attention network decoder can be used to perform topological reconstruction on the discrete symbol sequence and output an approximate adjacency matrix. Each element Indicates service node and The intensity of calls between them. To improve the readability and stability of the decoded topology, the call strength can be adjusted. Apply a threshold filtering strategy, setting the edge retention threshold to 30% of the maximum value in the current matrix (or other preset values). For specific calculations, refer to Formula 11 below:

[0183]

[0184] In the eleventh formula, It can represent an approximate adjacency matrix; elements Indicates service node and The intensity of calls between edges (edge ​​weights); Used to represent preset values, optional. That is, setting the edge retention threshold to 30% of the maximum value in the current matrix; It can represent 30% of the maximum value in the current matrix;

[0185] In the eleventh formula, In the case of [a specific event], the weight of the corresponding edge is set to zero to eliminate low-intensity, accidental, or background links; in other cases (i.e., [other events]), the weight of the corresponding edge is set to zero. Under these circumstances, maintain Remain unchanged to preserve the main service paths and enhance the expression of key dependencies.

[0186] After topology restoration is complete, the next step is to correlate indicators and extract the set of nodes from the graph. This represents the microservice instance (i.e., the graph node corresponding to the multiple application nodes) reconstructed by GAT (which can be the graph attention network model in step 102, or other preset GAT models). Subsequently, for each node... Furthermore, it can retrieve three key performance indicators from the results generated during the basic decoding stage: CPU utilization, memory availability, and network bandwidth usage of the host machine where the node resides, and construct a structured node indicator mapping tensor. (For example Figure 6 The indexes in the tensor are associated (each row corresponds to an index vector of a node), and the specific formal representation is as follows:

[0187]

[0188] in, It can represent a node index mapping tensor; and Nodes can be represented separately. Host machine's CPU utilization, memory usage, and network bandwidth utilization These metrics can be correlated with the node load factor during the sharding phase. The parameters remain consistent throughout. This tensor provides a fundamental representation for the fusion of link structure and resource status, supporting subsequent path analysis, bottleneck location, and distributed scheduling optimization.

[0189] To improve the completeness of topology and indicator reconstruction, an error compensation mechanism can also be introduced (e.g., in...). Figure 6 Error compensation repair (for missing information) is used to repair information loss caused by the compression process.

[0190] When an isolated node appears in the reconstructed topology graph, that is, a node without any incoming or outgoing edges, the historical neighbor set of that node can be retrieved from the discrete symbol sequence. This refers to the set of other nodes that the node is connected to before encoding. Then, the average association weight between the node and all nodes in the set is determined. This is used to measure its activity level in the historical topology. Based on this average association weight... To determine the probability of edge completion, please refer to the following twelfth formula for specific calculations:

[0191]

[0192] In the twelfth formula, It can be the normalized average association weight; 0.1 can be the preset basic connection probability; 0.4 can be the preset incentive factor for strong associations; the overall upper limit is controlled at 0.5 to ensure that the structure completion process is executed within a reasonable range; it is worth noting that "0.5", "0.1" and "0.4" are only exemplary numerical parameters, and those skilled in the art can flexibly set them according to actual application scenarios or specific needs.

[0193] Then, using probability Towards Edges are randomly added to the nodes with the highest correlation to restore topological connectivity and avoid graph fragmentation.

[0194] If a graph node cannot be successfully mapped to a performance metric in the TCN decoding result, the container tag of the service to which the node belongs can be queried, and historical metrics of other similar nodes under that tag can be retrieved. Missing values ​​are then smoothly filled using a sliding window mean, ensuring that the constructed graph node and point metric mapping tensors maintain spatial consistency. This mechanism improves the robustness of the decoding process when faced with missing, heterogeneous, or mutated symbol inputs, ensuring an end-to-end integrity closed loop in the data compression and recovery process.

[0195] Specifically, the multiple storage nodes can share a common central repository, on which a global compression model (which can be a large model including temporal convolutional network models and graph attention network models) can be set. Local compression models (such as lightweight compression models) can also be deployed on the multiple storage nodes. The local compression models can be periodically adaptively optimized to improve the decoding success rate in specific scenarios.

[0196] The specific update process can be driven by decoding failure samples. First, all decoding failure input-output pairs within a preset time range (e.g., the past 24 hours) are collected to form a sample set. Then, the corresponding real data is extracted through a log tracing mechanism to form a supervised reference set. Next, calculate the parameters of the current local compression model. For the local loss function on failed samples, please refer to the following formula (number thirteen):

[0197]

[0198] In the thirteenth formula, It can represent the value of a local loss function; It can represent the original input; It can represent the actual output; This represents the prediction result of the current local compression model; It can represent the local compression model parameters of the current local compression model; It can represent the number of failed samples; It can represent the L2 norm error;

[0199] After determining the local loss function value using the thirteen formulas, a gradient update can be performed based on this local loss function value to adjust the local compression model parameters to the new values. For details, please refer to the fourteenth formula below:

[0200]

[0201] In the fourteenth formula, It can represent new values ​​of parameters in a local compression model; It can represent the parameters of the current local compression model; It can represent the learning rate and can be used to control the update magnitude; the specific settings can be configured as needed by those skilled in the art. It can represent the gradient direction of the current local compression model parameters;

[0202] To avoid excessive offset in the local compression model, which could lead to inconsistency with the global compression model, a weight update strategy can be introduced to update the parameters of the local compression model. Global compression model parameters in the central warehouse The models are fused to obtain updated new model parameters, and the updated new model parameters are sent to the global compression model and the local compression model to update their respective model parameters.

[0203] The final updated model parameters can be obtained by referring to the following formula fifteen:

[0204]

[0205] In the fifteenth formula, It can represent the updated model parameters; It can represent global compression model parameters; It can represent new values ​​of parameters in a local compression model; And 0.1 can represent and The corresponding weights are as follows; it is worth noting that “0.9”, “0.1” and “0.4” are merely exemplary numerical parameters, and those skilled in the art can flexibly set them according to actual application scenarios or specific needs.

[0206] This weighted strategy balances the relationship between global generalization ability and local adaptability, enabling the model to maintain overall consistency while dynamically adapting to the business characteristics of each storage node, thereby improving the accuracy and fault tolerance of the compression model for future inputs.

[0207] For example, for the query request, the resource credit value can be determined based on the time range and number of services involved (e.g., the number of application nodes involved). For specific calculations, please refer to the following sixteenth formula:

[0208]

[0209] In the sixteenth formula, The resource credit value can represent the resource consumption of the query request; the time range can represent the time range corresponding to the query request; the number of services within the time range corresponding to the query request can be represented. and These can represent the weights corresponding to the time range and the number of services, respectively.

[0210] More specifically, in basic decoding mode, you can press Allocate resource credit; under deep decoding, dynamically calculate resource credit values ​​according to other rules, and then dynamically allocate resource credit to ensure reasonable allocation and dynamic adjustment of resource usage under different queries.

[0211] It should be noted that the data processing method described above can be executed by an electronic device, that is, all steps included in the above method are executed by the electronic device, which can be a server, computer, cloud service node or mobile phone or other electronic device.

[0212] See Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application, as shown below. Figure 7As shown, the data processing device 700 includes:

[0213] The acquisition module 701 is used to acquire application performance management (APM) data from multiple application nodes, wherein the APM data includes performance metric data and link tracing data.

[0214] The first processing module 702 is used to extract the periodic features of the performance index data using a preset temporal convolutional network model, generate a low-dimensional tensor based on the periodic features, and process the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix.

[0215] The second processing module 703 is used to concatenate the low-dimensional tensor and the sparsed dependency matrix to obtain a joint encoding vector, and use a preset dynamic quantizer to generate a discrete symbol sequence based on the joint encoding vector.

[0216] The third processing module 704 is used to process the discrete symbol sequence to obtain processed data.

[0217] Optionally, generating a low-dimensional tensor based on the periodicity feature includes:

[0218] The periodic features are filtered for noise using the temporal convolutional network model to obtain intermediate features;

[0219] The intermediate features are pooled using the temporal convolutional network model according to the pooling window to obtain a low-dimensional tensor.

[0220] The size of the pooling window is determined by the sampling frequency at which the APM data is acquired.

[0221] Optionally, the step of processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix includes:

[0222] A service call relationship graph is constructed based on the link tracing data, and the graph nodes of the service call relationship graph are constructed based on the multiple application nodes;

[0223] Based on the preset graph attention network model and the service call relationship graph, the adjacency matrix of multiple graph nodes in the service call relationship graph is determined respectively;

[0224] Based on the adjacency matrix, spectral clustering is performed to obtain the similarity scores of the multiple graph nodes.

[0225] The graph nodes whose similarity scores do not exceed the preset first threshold are kept as they are, and the graph nodes whose similarity scores exceed the preset first threshold are merged to obtain a compressed service call relationship graph.

[0226] A sparse dependency matrix is ​​generated based on the compressed service call relationship graph.

[0227] Optionally, processing the discrete symbol sequence to obtain processed data includes:

[0228] Obtain the processor utilization, memory utilization, and network bandwidth usage of multiple storage nodes respectively;

[0229] The load coefficients of the multiple storage nodes are obtained by weighted summation of the processor utilization, memory utilization, and network bandwidth utilization.

[0230] If the load factor of the target storage node is less than a preset second threshold, the discrete symbol sequence is stored in the target storage node in a round-robin sharding manner; if the load factor of the target storage node is greater than or equal to the preset second threshold and less than a preset third threshold, the discrete symbol sequence is stored in the target storage node in a consistent hash sharding manner; if the load factor of the target storage node is greater than or equal to the preset third threshold, the discrete symbol sequence is cut into multiple sub-data blocks according to a preset time window, the sub-data blocks located within the preset time range are stored in the storage nodes with load factors less than the second threshold, and the sub-data blocks located outside the preset time range are migrated to a preset storage device.

[0231] The target storage node is one or more of the plurality of storage nodes.

[0232] Optionally, the step of extracting periodic features from the performance index data using a preset temporal convolutional network model, generating a low-dimensional tensor based on the periodic features, and processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix includes:

[0233] The data compression ratio is determined based on the ratio of the amount of APM data to the amount of discrete symbol sequence data.

[0234] Obtain system load parameters, wherein the system load parameters are the system load parameters corresponding to the device executing the data processing method;

[0235] When the data compression ratio is less than a preset fourth threshold and the system load parameter is greater than a preset fifth threshold: the convolution stride of the temporal convolutional network model is adjusted to a preset sixth threshold to obtain an adjusted temporal convolutional network model. The periodic features of the performance index data are extracted using the adjusted temporal convolutional network model. A low-dimensional tensor is generated based on the periodic features. The first threshold is adjusted to a preset seventh threshold. Graph nodes whose similarity scores do not exceed the preset seventh threshold are kept as they are, and graph nodes whose similarity scores exceed the seventh threshold are merged to obtain a compressed service call relationship graph. A sparse dependency matrix is ​​generated based on the compressed service call relationship graph.

[0236] When the data compression ratio is greater than a preset seventh threshold and the system load parameter is less than a preset eighth threshold: the periodic features of the performance index data are extracted using a preset temporal convolutional network model, and a low-dimensional tensor is generated based on the periodic features. During the process of processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix, the spectral clustering operation based on the adjacency matrix is ​​stopped, and a sparse dependency matrix is ​​generated based on the service call relationship graph.

[0237] When the following conditions are met simultaneously: the data compression ratio is greater than or equal to the fourth threshold, the data compression ratio is less than or equal to the preset seventh threshold, the system load parameter is greater than or equal to the preset eighth threshold, and the system load parameter is less than or equal to the preset fifth threshold: the periodic features of the performance index data are extracted using a preset temporal convolutional network model, a low-dimensional tensor is generated based on the periodic features, and the link tracing data is processed using a preset graph attention network model to obtain a sparse dependency matrix.

[0238] Optionally, the data processing device 700 may also include a fourth processing module;

[0239] The fourth processing module is used to obtain query requests;

[0240] Determine the required query precision based on the query request;

[0241] Given the required query accuracy, the discrete symbol sequence is deconvolutionally decoded to recover the key trend points of the performance index data.

[0242] When the query accuracy requirement is the same as the depth requirement, topology recovery is performed on the discrete symbol sequence to obtain link tracing reconstruction data and / or performance indicator reconstruction data.

[0243] Optionally, the APM data is the APM data after data feature vector annotation;

[0244] The data feature vectors include: timestamp feature vector, service name feature vector, call level feature vector, and host tag feature vector.

[0245] The data processing apparatus 700 is designed to implement the various processes described above in the embodiments of the data processing method. The technical features are one-to-one and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0246] This application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described data processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0247] For details, see Figure 4 This application also provides an electronic device, including a bus 801, a transceiver 802, an antenna 803, a bus interface 804, a processor 805, and a memory 806.

[0248] The transceiver 802 is used to acquire application performance management (APM) data from multiple application nodes, and the APM data includes performance metric data and link tracing data.

[0249] The processor 805 is used to extract periodic features of the performance index data using a preset temporal convolutional network model, generate a low-dimensional tensor based on the periodic features, and process the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix.

[0250] The low-dimensional tensor and the sparsed dependency matrix are concatenated to obtain a joint encoding vector, and a discrete symbol sequence is generated based on the joint encoding vector using a preset dynamic quantizer.

[0251] The discrete symbol sequence is processed to obtain processed data.

[0252] Optionally, generating a low-dimensional tensor based on the periodicity feature includes:

[0253] The periodic features are filtered for noise using the temporal convolutional network model to obtain intermediate features;

[0254] The intermediate features are pooled using the temporal convolutional network model according to the pooling window to obtain a low-dimensional tensor.

[0255] The size of the pooling window is determined by the sampling frequency at which the APM data is acquired.

[0256] Optionally, the step of processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix includes:

[0257] A service call relationship graph is constructed based on the link tracing data, and the graph nodes of the service call relationship graph are constructed based on the multiple application nodes;

[0258] Based on the preset graph attention network model and the service call relationship graph, the adjacency matrix of multiple graph nodes in the service call relationship graph is determined respectively;

[0259] Based on the adjacency matrix, spectral clustering is performed to obtain the similarity scores of the multiple graph nodes.

[0260] The graph nodes whose similarity scores do not exceed the preset first threshold are kept as they are, and the graph nodes whose similarity scores exceed the preset first threshold are merged to obtain a compressed service call relationship graph.

[0261] A sparse dependency matrix is ​​generated based on the compressed service call relationship graph.

[0262] Optionally, processing the discrete symbol sequence to obtain processed data includes:

[0263] Obtain the processor utilization, memory utilization, and network bandwidth usage of multiple storage nodes respectively;

[0264] The load coefficients of the multiple storage nodes are obtained by weighted summation of the processor utilization, memory utilization, and network bandwidth utilization.

[0265] If the load factor of the target storage node is less than a preset second threshold, the discrete symbol sequence is stored in the target storage node in a round-robin sharding manner; if the load factor of the target storage node is greater than or equal to the preset second threshold and less than a preset third threshold, the discrete symbol sequence is stored in the target storage node in a consistent hash sharding manner; if the load factor of the target storage node is greater than or equal to the preset third threshold, the discrete symbol sequence is cut into multiple sub-data blocks according to a preset time window, the sub-data blocks located within the preset time range are stored in the storage nodes with load factors less than the second threshold, and the sub-data blocks located outside the preset time range are migrated to a preset storage device.

[0266] The target storage node is one or more of the plurality of storage nodes.

[0267] Optionally, the step of extracting periodic features from the performance index data using a preset temporal convolutional network model, generating a low-dimensional tensor based on the periodic features, and processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix includes:

[0268] The data compression ratio is determined based on the ratio of the amount of APM data to the amount of discrete symbol sequence data.

[0269] Obtain system load parameters, wherein the system load parameters are the system load parameters corresponding to the device executing the data processing method;

[0270] When the data compression ratio is less than a preset fourth threshold and the system load parameter is greater than a preset fifth threshold: the convolution stride of the temporal convolutional network model is adjusted to a preset sixth threshold to obtain an adjusted temporal convolutional network model. The periodic features of the performance index data are extracted using the adjusted temporal convolutional network model. A low-dimensional tensor is generated based on the periodic features. The first threshold is adjusted to a preset seventh threshold. Graph nodes whose similarity scores do not exceed the preset seventh threshold are kept as they are, and graph nodes whose similarity scores exceed the seventh threshold are merged to obtain a compressed service call relationship graph. A sparse dependency matrix is ​​generated based on the compressed service call relationship graph.

[0271] When the data compression ratio is greater than a preset seventh threshold and the system load parameter is less than a preset eighth threshold: the periodic features of the performance index data are extracted using a preset temporal convolutional network model, and a low-dimensional tensor is generated based on the periodic features. During the process of processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix, the spectral clustering operation based on the adjacency matrix is ​​stopped, and a sparse dependency matrix is ​​generated based on the service call relationship graph.

[0272] When the following conditions are met simultaneously: the data compression ratio is greater than or equal to the fourth threshold, the data compression ratio is less than or equal to the preset seventh threshold, the system load parameter is greater than or equal to the preset eighth threshold, and the system load parameter is less than or equal to the preset fifth threshold: the periodic features of the performance index data are extracted using a preset temporal convolutional network model, a low-dimensional tensor is generated based on the periodic features, and the link tracing data is processed using a preset graph attention network model to obtain a sparse dependency matrix.

[0273] Optionally, the processor 805 is further configured to acquire a query request;

[0274] Determine the required query precision based on the query request;

[0275] Given the required query accuracy, the discrete symbol sequence is deconvolutionally decoded to recover the key trend points of the performance index data.

[0276] When the query accuracy requirement is the same as the depth requirement, topology recovery is performed on the discrete symbol sequence to obtain link tracing reconstruction data and / or performance indicator reconstruction data.

[0277] Optionally, the APM data is the APM data after data feature vector annotation;

[0278] The data feature vectors include: timestamp feature vector, service name feature vector, call level feature vector, and host tag feature vector.

[0279] exist Figure 4 In this document, a bus architecture (represented by bus 801) is used. Bus 801 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 805 and memory represented by memory 806. Bus 801 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 804 provides an interface between bus 801 and transceiver 802. Transceiver 802 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 805 is transmitted over a wireless medium via antenna 803, which further receives data and transmits data to processor 805.

[0280] The processor 805 manages the bus 801 and handles general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory 806 can be used to store data used by the processor 805 during operation.

[0281] Optionally, the processor 805 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0282] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described data processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0283] This application also provides a computer program product, including computer instructions. When executed by a processor, the computer instructions implement the various processes of the above-described data processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0284] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0285] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0286] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Acquire application performance management (APM) data from multiple application nodes, wherein the APM data includes performance metric data and link tracing data; The periodic features of the performance index data are extracted using a preset temporal convolutional network model. A low-dimensional tensor is generated based on the periodic features. The link tracing data is then processed using a preset graph attention network model to obtain a sparse dependency matrix. The low-dimensional tensor and the sparsed dependency matrix are concatenated to obtain a joint encoding vector, and a discrete symbol sequence is generated based on the joint encoding vector using a preset dynamic quantizer. The discrete symbol sequence is processed to obtain processed data.

2. The method according to claim 1, characterized in that, The generation of low-dimensional tensors based on the periodic features includes: The periodic features are filtered for noise using the temporal convolutional network model to obtain intermediate features; The intermediate features are pooled using the temporal convolutional network model according to the pooling window to obtain a low-dimensional tensor. The size of the pooling window is determined by the sampling frequency at which the APM data is acquired.

3. The method according to claim 2, characterized in that, The process of using a preset graph attention network model to process the link tracing data to obtain a sparse dependency matrix includes: A service call relationship graph is constructed based on the link tracing data, and the graph nodes of the service call relationship graph are constructed based on the multiple application nodes; Based on the preset graph attention network model and the service call relationship graph, the adjacency matrix of multiple graph nodes in the service call relationship graph is determined respectively; Based on the adjacency matrix, spectral clustering is performed to obtain the similarity scores of the multiple graph nodes. The graph nodes whose similarity scores do not exceed the preset first threshold are kept as they are, and the graph nodes whose similarity scores exceed the preset first threshold are merged to obtain a compressed service call relationship graph. A sparse dependency matrix is ​​generated based on the compressed service call relationship graph.

4. The method according to claim 3, characterized in that, The process of processing the discrete symbol sequence to obtain processed data includes: Obtain the processor utilization, memory utilization, and network bandwidth usage of multiple storage nodes respectively; The load coefficients of the multiple storage nodes are obtained by weighted summation of the processor utilization, memory utilization, and network bandwidth utilization of the multiple storage nodes. If the load factor of the target storage node is less than a preset second threshold, the discrete symbol sequence is stored in the target storage node in a round-robin sharding manner; if the load factor of the target storage node is greater than or equal to the preset second threshold and less than a preset third threshold, the discrete symbol sequence is stored in the target storage node in a consistent hash sharding manner; if the load factor of the target storage node is greater than or equal to the preset third threshold, the discrete symbol sequence is cut into multiple sub-data blocks according to a preset time window, the sub-data blocks located within the preset time range are stored in the storage nodes with load factors less than the second threshold, and the sub-data blocks located outside the preset time range are migrated to a preset storage device. The target storage node is one or more of the plurality of storage nodes.

5. The method according to claim 4, characterized in that, The process of extracting periodic features from the performance index data using a preset temporal convolutional network model, generating a low-dimensional tensor based on the periodic features, and processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix includes: The data compression ratio is determined based on the ratio of the amount of APM data to the amount of discrete symbol sequence data. Obtain system load parameters, wherein the system load parameters are the system load parameters corresponding to the device executing the data processing method; When the data compression ratio is less than a preset fourth threshold and the system load parameter is greater than a preset fifth threshold: the convolution stride of the temporal convolutional network model is adjusted to a preset sixth threshold to obtain an adjusted temporal convolutional network model. The periodic features of the performance index data are extracted using the adjusted temporal convolutional network model. A low-dimensional tensor is generated based on the periodic features. The first threshold is adjusted to a preset seventh threshold. Graph nodes whose similarity scores do not exceed the preset seventh threshold are kept as they are, and graph nodes whose similarity scores exceed the seventh threshold are merged to obtain a compressed service call relationship graph. A sparse dependency matrix is ​​generated based on the compressed service call relationship graph. When the data compression ratio is greater than a preset seventh threshold and the system load parameter is less than a preset eighth threshold: the periodic features of the performance index data are extracted using a preset temporal convolutional network model, and a low-dimensional tensor is generated based on the periodic features. During the process of processing the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix, the spectral clustering operation based on the adjacency matrix is ​​stopped, and a sparse dependency matrix is ​​generated based on the service call relationship graph. When the following conditions are met simultaneously: the data compression ratio is greater than or equal to the fourth threshold, the data compression ratio is less than or equal to the preset seventh threshold, the system load parameter is greater than or equal to the preset eighth threshold, and the system load parameter is less than or equal to the preset fifth threshold: the periodic features of the performance index data are extracted using a preset temporal convolutional network model, a low-dimensional tensor is generated based on the periodic features, and the link tracing data is processed using a preset graph attention network model to obtain a sparse dependency matrix.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Get the query request; Determine the required query precision based on the query request; Given the required query accuracy, the discrete symbol sequence is deconvolutionally decoded to recover the key trend points of the performance index data. When the query accuracy requirement is the same as the depth requirement, topology recovery is performed on the discrete symbol sequence to obtain link tracing reconstruction data and / or performance indicator reconstruction data.

7. The method according to any one of claims 1 to 3, characterized in that, The APM data is the APM data after data feature vector annotation; The data feature vectors include: timestamp feature vector, service name feature vector, call level feature vector, and host tag feature vector.

8. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire application performance management (APM) data from multiple application nodes. The APM data includes performance metric data and link tracing data. The first processing module is used to extract the periodic features of the performance index data using a preset temporal convolutional network model, generate a low-dimensional tensor based on the periodic features, and process the link tracing data using a preset graph attention network model to obtain a sparse dependency matrix. The second processing module is used to concatenate the low-dimensional tensor and the sparsed dependency matrix to obtain a joint encoding vector, and use a preset dynamic quantizer to generate a discrete symbol sequence based on the joint encoding vector. The third processing module is used to process the discrete symbol sequence to obtain processed data.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7.

Images