A distributed system resource processing method and device and electronic equipment

Abstract

The application discloses a distributed system resource processing method and device and electronic equipment, and relates to the technical field of distributed computing and system architecture. Since a double-layer heterogeneous topology mapping of an index logical tree and a physical resource tree of a distributed system is constructed in a graph database, cross-layer association is realized through a dynamic mapping edge containing an association strength factor. Through real-time feature extraction and information value calculation on an unstructured signal flow, the perception of business mode changes is realized. Based on the perception result, adaptive optimization of the topology structure is driven, and a resource optimization mechanism based on gradient calculation results is triggered. Therefore, the technical problem that in a non-stationary environment, due to topology rigidity, persistent deviation between a monitoring topology and an actual business topology is generated, which is not conducive to guaranteeing the overall performance of the distributed system, can be solved, and a foundation for improving the overall performance of the distributed system is laid.

Description

Technical Field

[0001] This application relates to the field of distributed computing and system architecture technology, and in particular to a distributed system resource processing method, apparatus and electronic device. Background Technology

[0002] With the widespread adoption of cloud-native architectures and microservices, the scale and complexity of modern distributed systems are growing exponentially. A typical enterprise-level distributed system may contain hundreds of microservice instances, thousands of containerized workloads, and heterogeneous physical infrastructure spanning multiple data centers. Therefore, resource management within distributed systems has become a key research focus in order to improve their performance.

[0003] In related technologies, performance monitoring of distributed systems mainly relies on predefined static thresholds and manually configured fixed physical mapping relationships, which can provide basic observability support in scenarios with relatively stable business models.

[0004] However, in non-stationary operating environments where the business logic of a distributed system changes frequently and the data flow characteristics shift over time, the static deployment of monitoring objects and resource topology in a distributed system makes it impossible to automatically adjust the number, type, and connection relationships of monitoring objects according to the evolution of business characteristics. When a new microservice is launched, an old service is taken offline, or the inter-service call relationships change, operations and maintenance personnel need to manually update the monitoring configuration, resulting in a continuous deviation between the monitoring topology and the actual business topology, which is detrimental to ensuring the overall performance of the distributed system. Summary of the Invention

[0005] This application provides a method, apparatus, and electronic device for processing resources in a distributed system, which at least solves the problem in related technologies where a continuous deviation occurs between the monitoring topology and the actual business topology, which is detrimental to ensuring the overall performance of the distributed system.

[0006] This application provides a distributed system resource processing method, including: In a graph database, a distributed system's indicator logic tree and physical resource tree are constructed. The indicator logic tree includes multiple logical nodes representing logical monitoring units, and the physical resource tree includes multiple resource nodes representing target devices. A dynamic mapping edge is provided between the logical nodes and resource nodes, and the dynamic mapping edge includes a correlation strength factor used to characterize the degree of correlation between the logical nodes and resource nodes. Obtain the unstructured signal stream generated by the distributed system; Feature extraction is performed on the unstructured signal stream to obtain a feature vector; Based on the degree of deviation between the current feature distribution, represented by the information value of the feature vector, and the baseline distribution, it is determined whether the business mode of the distributed system has changed. When it is determined that the business model of the distributed system has changed, the topology of the indicator logic tree and physical resource tree is optimized to adapt to the change in the business model and update the indicator logic tree and physical resource tree. When the indicator logic tree reflects that the performance status of the distributed system meets the preset resource optimization conditions, the performance loss of the distributed system relative to the target gradient of the physical resource allocation of the physical resource tree is calculated. Based on the target gradient and the original resource allocation strategy, the target resource optimization strategy of the distributed system is determined; Based on the target resource optimization strategy, the distributed system is optimized for resource allocation.

[0007] This application also provides a distributed system resource processing apparatus, including: A construction module is used to construct an indicator logic tree and a physical resource tree of a distributed system in a graph database. The indicator logic tree includes multiple logical nodes representing logical monitoring units, and the physical resource tree includes multiple resource nodes representing target devices. A dynamic mapping edge is provided between the logical nodes and the resource nodes, and the dynamic mapping edge includes a correlation strength factor used to characterize the degree of correlation between the logical nodes and the resource nodes. An acquisition module is used to acquire the unstructured signal stream generated by the distributed system; The feature extraction module is used to extract features from the unstructured signal stream to obtain feature vectors; The judgment module is used to determine whether the business mode of the distributed system has changed based on the degree of deviation between the current feature distribution represented by the information value of the feature vector and the baseline distribution. The topology optimization module is used to optimize the topology of the indicator logic tree and physical resource tree when it is determined that the business mode of the distributed system has changed, so as to update the indicator logic tree and physical resource tree to adapt to the change in the business mode. The calculation module is used to calculate the target gradient of the performance loss of the distributed system relative to the physical resource allocation of the physical resource tree when the performance status of the distributed system reflected by the indicator logic tree meets the preset resource optimization conditions. The determination module is used to determine the target resource optimization strategy of the distributed system based on the target gradient and the original resource allocation strategy. The optimization module is used to optimize the resources of the distributed system according to the target resource optimization strategy.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described distributed system resource processing methods.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described distributed system resource processing methods.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described distributed system resource processing methods.

[0011] This application achieves cross-layer association by constructing a two-layer heterogeneous topology mapping of the distributed system's indicator logical tree and physical resource tree in a graph database, using dynamic mapping edges containing association strength factors. Through real-time feature extraction and information value calculation of unstructured signal flows, it realizes the perception of changes in business patterns. Based on this perception result, it drives adaptive optimization of the topology and triggers a resource tuning mechanism based on gradient calculation results. Therefore, it can solve the technical problem of persistent deviations between the monitoring topology and the actual business topology due to topology rigidity in non-stationary environments, which is detrimental to ensuring the overall performance of the distributed system, thus laying the foundation for improving the overall performance of distributed systems. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a distributed system resource processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a heterogeneous two-layer topology mapping architecture provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the distributed system resource processing device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0015] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0016] With the widespread adoption of cloud-native architectures and microservices, the scale and complexity of modern distributed systems are growing exponentially. A typical enterprise-level distributed system may contain hundreds of microservice instances, thousands of containerized workloads, and heterogeneous physical infrastructure spanning multiple data centers. In this context, the core challenge for system operation and maintenance has shifted from simply monitoring to intelligently sensing changes and responding autonomously.

[0017] Traditional distributed monitoring systems, such as Prometheus, Zabbix, and Nagios, primarily rely on predefined static thresholds and manually configured fixed physical mappings. While these systems provide basic observability in scenarios with relatively stable business models, they exhibit significant shortcomings in non-stationary operating environments where business logic frequently changes and data flow characteristics shift over time: First, there's the issue of topology rigidity. Existing monitoring systems typically define their monitoring objects and topology statically at deployment time, failing to automatically adjust the number, type, and connection relationships of monitoring logical nodes based on evolving business characteristics. When new microservices are launched, older services are decommissioned, or inter-service call relationships change, operations personnel need to manually update monitoring configurations, leading to a persistent discrepancy between the monitoring topology and the actual business topology. More critically, traditional systems lack the ability to dynamically correlate business logic-level monitoring needs with underlying physical resources, resulting in a disconnect between logical layer monitoring and physical layer resource management.

[0018] Second, there is the issue of perception lag. In real-world production environments, a large amount of critical system status information exists in the form of unstructured logs, event streams, and other similar formats. Traditional monitoring systems have extremely limited processing capabilities for this type of data, typically only able to perform simple regular expression matching or keyword searches. When system operating characteristics undergo gradual drift—the phenomenon defined in statistics as concept drift—traditional methods struggle to quantify the degree and direction of this change in a timely manner. For example, the log pattern of a trading system may gradually deviate from the baseline distribution over several weeks, but alarm mechanisms based on static thresholds only trigger when the problem worsens to exceed a fixed threshold, missing the optimal window for early intervention.

[0019] Third, there is the issue of regulatory gaps. This is the most prominent shortcoming in existing technologies. Current monitoring systems typically stop at the level of problem detection and alerting in their functional chain. After an alert is generated, it still relies on maintenance personnel for manual judgment and adjustment. There is a lack of an automated closed-loop transmission mechanism between high-level business requirements at the monitoring layer (such as reducing transaction latency to within 50ms) and fine-grained control of underlying physical resources (such as CPU core affinity settings, network bandwidth traffic shaping, and disk I / O priority scheduling). This open-loop model of monitoring-alerting-manual handling is no longer sufficient to meet the needs of scenarios with high failure frequency and strict response time requirements.

[0020] Fourth, there is the conflict between transaction pressure and monitoring frequency. In high-frequency monitoring scenarios, the system needs to collect and update monitoring data at a sampling frequency of milliseconds to seconds. If the traditional relational database or graph database's record-by-record synchronous write method is used to maintain the monitoring topology, the high-frequency write transactions will severely conflict with the graph database's own strong consistency guarantee mechanism, leading to transaction lock contention, soaring write latency, and even monitoring link blockage. This technical contradiction between the real-time requirements of high-frequency monitoring signals and the performance overhead of graph database's strong consistency is a key problem that existing technologies have not been able to effectively solve.

[0021] Fifth, the issue of oscillation. Even in existing automated operation and maintenance solutions that achieve a certain degree of automatic resource adjustment, oscillation is a common risk. This means that adjusting the resources of a physical node A may trigger resource stress on node B, and the compensation adjustments to node B, in turn, affect node A, forming a cyclical oscillation loop. In distributed systems, this oscillation can cascade and spread across multiple nodes, ultimately causing a system-wide cascading failure, the damage of which may even exceed the original performance problem itself. Current technologies lack a system-wide perspective and a robust security intervention mechanism with frequency limiting and waterline verification capabilities to effectively suppress such oscillations.

[0022] To address the aforementioned technical problems, embodiments of this application provide a distributed system resource processing method, apparatus, and electronic device. By constructing a two-layer heterogeneous topology mapping of the distributed system's indicator logic tree and physical resource tree in a graph database, cross-layer associations are achieved through dynamic mapping edges containing association strength factors. Real-time feature extraction and information value calculation of unstructured signal flows enable the perception of changes in business models. Based on this perception result, adaptive optimization of the topology structure is driven, triggering a resource tuning mechanism based on gradient calculation results. Therefore, this addresses the technical problem in non-stationary environments where topology rigidity leads to persistent deviations between the monitoring topology and the actual business topology, hindering the overall performance of the distributed system and laying the foundation for improving the overall performance of distributed systems.

[0023] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] This application provides a distributed system resource processing method for adaptive monitoring and resource optimization of a distributed system. The execution subject of this application embodiment is an electronic device, such as a server, desktop computer, laptop computer, tablet computer, or other electronic devices that can be used for distributed system resource processing.

[0025] like Figure 1 The diagram shown is a flowchart illustrating a distributed system resource processing method provided in an embodiment of this application. The method includes: Step 101: Construct the indicator logic tree and physical resource tree of the distributed system in the graph database.

[0026] The indicator logic tree includes multiple logic nodes representing logical monitoring units, and the physical resource tree includes multiple resource nodes representing target devices. There are dynamic mapping edges between the logic nodes and resource nodes, and the dynamic mapping edges include a correlation strength factor used to characterize the degree of correlation between the logic nodes and resource nodes.

[0027] Specifically, in this application embodiment, two interconnected but structurally heterogeneous topological structures are constructed in a graph database (including but not limited to Neo4j, JanusGraph, AmazonNeptune, or TigerGraph). The two are connected by dynamically mapped edges to form a heterogeneous two-layer topological mapping.

[0028] For example, such as Figure 2 The diagram shown illustrates a heterogeneous two-layer topology mapping architecture provided in this application embodiment. The Indicator Tree (IT) is a monitoring topology layer oriented towards business logic. Each node in the IT tree... This represents a logical monitoring unit, whose data structure is defined as follows: Unique Identifier Operator type tags (such as LDA_Transformer, W2V_Encoder, LSTM_Aggregator, etc.), weight vector (in (Feature dimension), state activation function (Sigmoid, tanh, or ReLU family functions can be used), and creation timestamp. and the most recent update timestamp In an IT tree, the edges between nodes represent logical dependencies, including three types: aggregation, derivation, and trigger.

[0029] The Physical Resource Tree (RT) is an infrastructure-oriented resource topology layer. Each node in the RT tree... Representing an actual physical device, virtual machine, or container instance, its data structure includes: IP address, hostname, CPU core list and its real-time utilization, total memory and current quota, current disk I / O throughput and queue depth, network interface bandwidth capacity and real-time traffic, and a comprehensive resource level indicator. (Water Level) is used to characterize the overall resource stress level of the node. The edges between nodes in the RT tree represent physical connections, including three types: ownership (the container belongs to the host machine), network connection (two hosts belong to the same subnet), and storage mount relationship.

[0030] Dynamically mapped edges ) is a connection and Cross-layer association. Each mapping edge Includes a correlation strength factor This indicates the degree of association between a logical monitoring unit and a physical resource node. The value calculation takes into account factors such as historical call frequency, data traffic ratio, and response time correlation.

[0031] Specifically, in one embodiment, in a high-frequency monitoring scenario, the system's monitoring sampling frequency can reach the millisecond level. If each... All value changes are written to the graph database synchronously, which will lead to severe transaction lock contention. This is because, to ensure ACID properties, graph databases require acquiring node locks and edge locks for each write transaction. High-frequency writes will cause a sharp increase in the proportion of lock waiting time, creating a technical contradiction between the real-time requirements of high-frequency monitoring signals and the performance overhead of strong consistency in graph database transactions. This application's embodiment resolves this contradiction through the following asynchronous batch write mechanism: the system maintains a write buffer queue in memory, where all... Value update requests are first written to this queue instead of being directly committed to the database transaction; the system sets a configurable time window. (Default value is 5 seconds) At the end of each time window, all accumulated update operations in the queue are merged. For multiple updates to the same mapped edge within the window, only the final value is retained, and then committed as a single batch graph transaction. Experiments have verified that this mechanism, while ensuring eventual consistency of mapping relationships, reduces the number of write transactions in the graph database by one to two orders of magnitude, and reduces the transaction lock contention rate by more than 90%, effectively resolving the aforementioned technical contradictions.

[0032] Step 102: Obtain the unstructured signal stream generated by the distributed system.

[0033] Among them, unstructured signal streams are the raw signals generated by the distributed system, including system logs, application logs, network packet header information, etc.

[0034] Step 103: Extract features from the unstructured signal stream to obtain feature vectors.

[0035] Specifically, the unstructured signal stream can first be preprocessed (including word segmentation, denoising, and normalization), and then mapped into K-dimensional feature vectors using a configurable vectorization model. This is done to obtain feature vectors. The selection of the vectorization model is based on the logical nodes in the IT tree. of The attributes can be configured to allow different monitoring units to independently select different vectorization strategies based on the data characteristics of their monitored objects. In a preferred embodiment, the optional vectorization models include, but are not limited to: the Latent Dirichlet Allocation (LDA) model suitable for log topic extraction scenarios, the Word2Vec or FastText model suitable for short text semantic embedding scenarios, and the encoder model based on the Transformer architecture (such as BERT's [CLS] vector output) suitable for complex semantic understanding scenarios.

[0036] Step 104: Determine whether the business model of the distributed system has changed based on the degree of deviation between the current feature distribution represented by the information value of the feature vector and the baseline distribution.

[0037] It should be noted that the information value is used to quantify the degree of deviation of the current feature distribution from the baseline distribution.

[0038] Step 105: When it is determined that the business model of the distributed system has changed, the topology of the indicator logic tree and the physical resource tree is optimized to adapt to the change in business model and update the indicator logic tree and the physical resource tree.

[0039] Specifically, when a change in the business model of a distributed system is detected, the following automated topology operations are triggered: First, new logical monitoring nodes are automatically generated in the IT tree through graph database transactions. The operator type for each node is automatically selected based on the current feature change pattern (e.g., if the change is concentrated in a specific topic dimension, the LDA operator is selected; if global semantic offset is involved, the Transformer operator is selected); secondly, each physical node in the RT tree is queried. resource water level Select physical nodes whose water level is below the safety threshold as the mapping targets for new logical nodes, and create new mapping edges. and initialize Value. Conversely, when the IV value remains below the contraction threshold. Reaching the configurable time window Then, the system automatically removes redundant logical nodes and their mapped edges, achieving topology slimming and reducing system overhead.

[0040] Step 106: When the performance status of the distributed system reflected by the indicator logic tree meets the preset resource optimization conditions, calculate the target gradient of the performance loss of the distributed system relative to the physical resource allocation of the physical resource tree.

[0041] Specifically, the performance loss function This function quantifies the deviation between the current system state and the target state to obtain the performance loss of the distributed system. In a preferred embodiment of this application, It can be defined as a weighted combination of multiple performance metrics, for example:

[0042] in, These are configurable weighting coefficients. This represents the actual measured system delay. Indicates the preset target delay. This represents the actual measured system throughput. This represents the preset target system throughput. This represents the actual measured system error rate.

[0043] Specifically, in one embodiment, the performance loss of the distributed system relative to the target gradient of the physical resource allocation of the physical resource tree can be calculated based on the following formula:

[0044] in, Represents the target gradient. This represents the performance loss in a distributed system. , Indicates CPU core quota. Indicates the memory limit. Indicates network bandwidth limit. The physical resource tree represents I / O weights and includes multiple adjustable physical resource parameters, including at least CPU core quotas, memory limits, network bandwidth limits, and I / O weights.

[0045] Specifically, in actual physical resource allocation, some resource parameters are discrete (e.g., the number of CPU cores is a positive integer, and I / O priority has a finite number of levels), making it impossible to directly calculate accurate gradients using classical continuous differentiation methods. This application's embodiments employ the following hybrid gradient estimation strategy to address this problem: for continuous parameters (such as network bandwidth limits), a numerical differentiation method (i.e., a perturbation method) is directly used to apply small perturbations to the parameters. After observing the change in the loss function, the estimated partial derivative is: For discrete parameters (such as the number of CPU cores), the Finite Difference Approximation method is used, that is, the perturbation is... For parameters equal to the smallest adjustable granularity (e.g., one CPU core), the difference in the loss function at this granularity is calculated as an approximation of the gradient. For enumerated parameters (e.g., I / O scheduling categories with only three discrete values: real-time, best-effort, and idle), a heuristic gradient estimation based on a lookup table is used. This involves pre-constructing a lookup table mapping parameter values ​​to performance effects (obtained through offline stress testing or online learning), and directly looking up the table during gradient calculation to determine the optimal adjustment direction. The unified output of all three methods is a scalar gradient component. The complete gradient vector is formed by concatenating vectors. .

[0046] Step 107: Determine the target resource optimization strategy for the distributed system based on the target gradient and the original resource allocation strategy.

[0047] Specifically, in one embodiment, the target resource optimization strategy of the distributed system can be determined based on the following formula, according to the target gradient and the original resource allocation strategy:

[0048] in, This represents the target resource optimization strategy for a distributed system. Indicates the original resource allocation strategy. This indicates the preset learning rate (configurable, default value 0.01), which controls the step size of each tuning iteration. This represents the target gradient.

[0049] Step 108: Optimize the resources of the distributed system according to the target resource optimization strategy.

[0050] Specifically, based on the target optimization strategy, the computing, network, or storage resources of physical nodes (such as servers, containers, and other target devices) in a distributed system can be adjusted to change the runtime state of the system and improve its overall performance.

[0051] Based on the above embodiments, as an implementable approach, in one embodiment, determining whether the business model of the distributed system has changed based on the degree of deviation between the current feature distribution represented by the information values ​​of the feature vector and the baseline distribution includes: Step 1041: Divide the distribution of feature vectors into multiple bins; Step 1042: Determine the information value of the feature vector based on the current distribution proportion and the baseline distribution proportion of each bin; Step 1043: Determine the information threshold based on the historical sequence of information values ​​represented by the baseline distribution; Step 1044: Determine whether the business model of the distributed system has changed based on the degree of deviation between the information value of the feature vector and the information threshold.

[0052] Specifically, the K-dimensional feature space (feature vectors) can be divided into: Each bin, for each bin Calculate the current distribution percentage Proportion of baseline distribution Then calculate IV using the following formula:

[0053] in, Information value, baseline distribution Data is sourced from the system initialization phase or the stabilization period following the last model reload. To avoid zero values ​​in the bins causing abnormal logarithm calculations, the system applies Laplace smoothing to the proportions of all bins, meaning that a small constant is added to the count of each bin before calculation. (default value Then normalize it again.

[0054] Specifically, the system maintains a dynamic threshold. Also known as the information threshold, this threshold is adaptively adjusted based on the statistical characteristics of the historical IV value series. Specifically, the historical IV value series can be smoothed using the Exponentially Weighted Moving Average (EWMA) method, resulting in a dynamic threshold. It is set as the EWMA mean plus a certain number of standard deviations. When the calculated IV value exceeds... When this happens, the system determines that the current feature distribution has changed significantly, meaning that the business model of the distributed system has changed.

[0055] Based on the above embodiments, as an implementable approach, in one embodiment, the method further includes: Step 201: Maintain two sets of parameter buffers and an atomic pointer to the currently active buffer in memory; Step 202: Periodically calculate the population stability index of the feature vector based on the current distribution proportion of each bin and the preset reference distribution proportion; Step 203: When the group stability index exceeds the preset drift threshold, calculate new weight parameters based on new data in the inactive buffer. Step 204: After the new weight parameters are verified, the atomic pointer is switched through an atomic operation to switch the inactive buffer to the active buffer. Subsequent computation requests will use the new weight parameters in the active buffer.

[0056] It should be noted that the Population Stability Index (PSI) measures the overall shift in the characteristic distribution between two time windows. Its calculation formula is formally consistent with the Individual Value Index (IV), but the semantics differ. The PSI compares the actual distributions of two different time periods, rather than the actual distributions with a fixed theoretical baseline.

[0057] in The first in the current time window Distribution ratio of each container (current distribution percentage) For the reference time window The distribution ratio of each bin (preset reference distribution ratio). The system calculates PSI values ​​on a rolling basis at configurable time intervals (default every minute). Laplace smoothing is also applied to ensure numerical stability. The interpretation of PSI values ​​follows the following three-level criteria: PSI < 0.1 indicates stable distribution, no intervention required; 0.1 ≤ PSI < 0.25 indicates slight drift, the system issues an early warning and automatically increases the monitoring frequency (e.g., shortening the PSI calculation interval from 1 minute to 10 seconds); PSI ≥ 0.25 (exceeding the preset drift threshold) indicates significant drift, triggering a hot reload of model parameters.

[0058] Specifically, when the PSI exceeds 0.25 and triggers a reload, the system employs a double-buffering mechanism to ensure business continuity during parameter updates. The core design idea of ​​this mechanism is to decouple parameter calculation from parameter usage in physical memory space, achieving instantaneous switching between the two spaces through atomic operations. The specific implementation is as follows: the system always maintains two sets of parameter buffers in memory, labeled Buffer-A and Buffer-B respectively, with each buffer having an equally weighted size vector. Dimensions Multiply by the corresponding data type's number of bytes. The system also maintains an atomic pointer, which points to the currently active buffer at any given time. During normal operation, all computation requests read parameters from Buffer-A through the atomic pointer. Perform the computation. When the reload is triggered, the system calculates new weights in Buffer-B based on the data from the latest time window in the background training thread. After the calculation is completed and passes basic verifications (such as parameter range verification and numerical stability verification), the pointer is switched from Buffer-A to Buffer-B via a CAS (Compare-And-Swap) atomic operation. Subsequent new calculation requests will automatically use the new parameters. Computation requests that started before the switching time but have not yet been completed will continue to use the old parameters in Buffer-A. Buffer-A releases its reference to the old buffer naturally at the end of its lifecycle. Once all requests holding references to the old buffer have completed, Buffer-A becomes idle and ready for the next hot reload. The entire process requires no downtime, no exclusive locks, and has zero impact on pending requests.

[0059] Based on the above embodiments, as an implementable approach, in one embodiment, the method further includes: Step 301: When the main logic node on the main path in the indicator logic tree fails, the main logic node is designated as the failed node. Step 302: Retrieve the candidate logical node with the highest comprehensive similarity to the faulty node in the indicator logical tree; Step 303: Determine the confidence penalty factor of the candidate logical node based on the comprehensive similarity between the candidate logical node and the faulty node, the fault occurrence time, and the preset time decay coefficient. Step 304: Determine the downgraded output result based on the calculated output of the candidate logical node and the confidence penalty factor of the candidate logical node.

[0060] It should be noted that, unlike the traditional hard failover strategy, which immediately cuts off the faulty path or switches to the backup node upon detecting a fault, this application adopts a flexible strategy: after the primary node (the primary logical node on the primary path) fails, the system does not simply output a zero value or switch completely, but instead retrieves the semantically closest candidate node to provide a degraded but still valuable output result. At the same time, the reliability loss of the output is clearly marked by a confidence penalty factor, enabling downstream systems and administrators to differentiate the output results.

[0061] Specifically, when the main path A certain main logic node When a fault occurs (such as heartbeat timeout, abnormal operator exit, or calculation result verification failure), the graph database's graph traversal and vector indexing capabilities are used to search for the faulty node in the IT tree. The candidate logical node that is closest in semantic space (with the highest overall similarity). The semantic distance metric considers three dimensions: first, operator type similarity, i.e. and First, whether to use vectorized models of the same type or family; second, the cosine similarity of the weight vectors, i.e. Third, the path distance within the IT topology, i.e., from... arrive The shortest path hop count. The three dimensions of the similarity are combined into a single scalar by weighted summation. .

[0062] Among them, the confidence penalty factor is calculated. The formula is as follows:

[0063] in, The overall similarity between the master node and the candidate nodes. The duration (in seconds) of the fault after it occurs. This is a preset time decay coefficient (configurable, default value 0.1). The design intent of this formula is to reflect a two-tiered penalty logic: first, the lower the semantic similarity between the candidate node and the main node, the more likely it is to cause a delay. The smaller the value, the weaker the ability of the backup node to replace the master node; secondly, the longer the fault duration, the weaker the ability of the backup node to replace the master node. The smaller the value due to exponential decay, the worse the timeliness of the data.

[0064] The final output calculation (degraded output result) is as follows: :

[0065] in as a candidate node The system will calculate and output based on its own parameters. The value is appended to the output as metadata tags, which downstream consumers can use to... The value determines whether to accept the result and with what weight. Simultaneously, the system sets a configurable degradation alarm threshold. (Default 0.5), when When this happens, the system automatically sends a degraded operation warning notification to the management team. The notification includes the fault node identifier, the candidate node identifier, and the current status. Values ​​and their changing trends.

[0066] Based on the above embodiments, as an implementable approach, in one embodiment, before performing resource tuning on the distributed system according to the target resource optimization strategy, the method further includes: Step 401: After the resource optimization action is executed, verify whether the resource level of the target resource node and its corresponding upstream and downstream related nodes in the physical resource tree exceeds the safety threshold, and obtain the resource level verification result. Step 402: Verify whether the number of tuning commands issued to the same resource node within the preset time window exceeds the upper limit, and obtain the tuning frequency verification result. Step 403: Detect whether the impact path of resource optimization actions in the physical resource tree forms a loop, and obtain the circular dependency verification result; Step 404: When there are anomalies in the resource level verification results, optimization frequency verification results, or circular dependency verification results, adjust the resource optimization actions to optimize the target resource optimization strategy.

[0067] Specifically, this application embodiment also designs an intervention layer, a security intervention layer used as a key protection mechanism to prevent oscillation and cascading failure. In distributed systems, oscillation is a serious technical failure: when resource adjustments to node A cause resource strain on node B, and then compensation adjustments to node B in turn affect node A, the system will fall into a continuous resource contention cycle, which not only fails to solve the original performance problem but may also trigger a global decline in service quality or even cascading failures. The security intervention layer of this application embodiment suppresses the risk of oscillation through the following triple protection mechanism: The first layer of protection, the Global Water Level Validator: Before converting the gradient calculation results into actual control commands, the security intervention layer queries the target physical node (target resource node) in the RT tree. Resource water level lines of its upstream and downstream related nodes This ensures that after any individual resource optimization action is performed, the utilization rate of any resource on the target node will not exceed the configurable safety threshold. (Default value is 85%). If the gradient calculation suggests a tuning amount that would cause the safety threshold to be exceeded, the system will automatically truncate the tuning amount to a safe boundary value.

[0068] The second layer of protection is the tuning rate limiter: the system optimizes the frequency limiter for each physical node. Maintain a token bucket-style frequency limiter, limiting the frequency to a configurable time window. The maximum number of tuning commands issued to the same physical node within a default period of 30 seconds. (Default 3 times). The purpose of this limiter is to introduce a cooling-off period for tuning, preventing the system from making new adjustment decisions based on incomplete feedback data before the effects of the previous tuning instruction have fully manifested, thereby breaking potential oscillating positive feedback loops.

[0069] The third layer of protection is the Circular Dependency Detector: Before issuing each tuning instruction, the system checks the propagation path of the tuning action in the RT tree. If it detects that the effect of the tuning action may return to the original node that triggered the tuning after several hops (i.e., forming a loop), the tuning action is marked as high risk, its step size is reduced to 1 / 4 of the original value or it is directly rejected, and recorded in the risk audit log.

[0070] Based on the above embodiments, as an implementable approach, in one embodiment, resource optimization of the distributed system is performed according to a target resource optimization strategy, including: Step 1081: Map the resource tuning actions in the target resource optimization strategy to operating system kernel-level resource control instructions; Step 1082: Send the resource control command to the corresponding target resource node for execution; Step 1083: After executing the resource control command on the target resource node, collect feedback data; Step 1084: Update the physical resource tree based on the feedback data.

[0071] Specifically, the resource control instructions mapped to the operating system kernel level can refer to Linux kernel control instructions.

[0072] Specifically, regarding CPU isolation and affinity control, the system restricts specified processes or containers to run on specific CPU cores by writing to the cgroupsv2's cpuset.cpus interface file. (CPU component in the gradient vector) A positive value indicates that the number of cores needs to be increased, and a negative value indicates that the number of cores can be decreased. The updated number of cores... After being rounded down (floor), the core IDs are converted into a specific list and written into the cpuset interface.

[0073] Regarding network traffic shaping, the system configures token bucket filter (TBF) or hierarchical token bucket (HTB) queue rules through the qdisc (Queuing Discipline) interface of the Linux kernel's tc (Traffic Control) subsystem. The gradient vector contains the network bandwidth component. The rate parameter increment / decrement value is converted to TBF / HTB, in Mbps.

[0074] Regarding I / O priority scheduling, the system dynamically adjusts the I / O scheduling priority and scheduling category of a specified process through the ionice tool or by directly calling the ioprio_set system call. (I / O component in the gradient vector) The I / O scheduling is mapped to one of three I / O scheduling categories (real-time class1, best-effort class2, and idle class3) and its intra-class priority level (0-7) using a lookup table method.

[0075] Furthermore, all issued kernel instructions are recorded in the audit log, and the system records the instructions at configurable time intervals after execution. After a default interval of 10 seconds, the actual resource status of the target physical node is re-collected. The latest collected metrics, such as CPU utilization, memory usage, network bandwidth utilization, and I / O queue depth, are written back to the attributes of the corresponding node in the RT tree, completing the physical resource tree status update. The updated RT tree status is then dynamically mapped to edges. The performance penalty function is passed to the logical layer of the IT tree. It can reflect the actual effect of the optimization actions in the next calculation cycle. If If the value increases after optimization (i.e., the optimization effect is negative), the system automatically performs a rollback operation, restoring resource allocation to the state before optimization and marking the anomaly in the audit log. Simultaneously, this anomaly is fed back to the security intervention layer to update its risk model. Through this feedback mechanism, the system forms a complete closed loop of perception → calculation → execution → feedback → re-perception, ensuring that the optimization process converges to the optimal state rather than diverging.

[0076] The functions implemented in each step of the above-described distributed system resource processing method can be represented by functional modules, mainly including a topology storage module, a feature evolution module, a drift reload module, a fault tolerance compensation module, and a kernel execution module. These five modules work together to form a complete adaptive management closed-loop link from signal acquisition → feature quantization → topology evolution → drift perception → closed-loop tuning → execution feedback. Using a graph database as the core data storage and relational reasoning engine, the Linux kernel control interface as the final execution exit, and real-time feedback of execution results as the convergence guarantee of the closed-loop, end-to-end adaptive management from the business logic layer to the physical resource layer is achieved. To facilitate a better understanding of the distributed system resource processing method provided in this application embodiment by those skilled in the art, the following specific examples are also provided in this application embodiment: This paper uses a high-frequency trading monitoring scenario at a securities company as an example to illustrate a specific implementation method. In this scenario, the system needs to monitor dozens of trading servers, hundreds of trading microservice instances, and tens of thousands of trading logs and market data push logs generated per second.

[0077] In the implementation of the topology storage module, Neo4j 5.x is used as the graph database engine. The IT tree is constructed as follows: the top-level node is the logical root node for global transaction health ( , =="Aggregator"), the secondary nodes are divided into stock trading, futures trading, options trading, etc. according to business lines ( =="BizLine_Monitor"), the third-level nodes are the key indicator monitoring units for each business line (such as order delay monitoring, conversion rate monitoring, risk control indicator monitoring, etc.). =="KPI_Tracker"), the lowest level node is the specific operator node (such as the operator used to analyze the topic distribution of transaction logs using the LDA model, =LDA_Transformer, The RT tree is organized hierarchically as follows: data center → server room → rack → server → container. Each server node records its 32-core CPU core list, 128GB memory quota, real-time throughput of the 10 Gigabit network interface, and I / O queue depth of the NVMe disk. Mapping edges associate the underlying LDA operator nodes with the Kubernetes container nodes that host them, with η initialized to 1.0. Asynchronous batch write window. Set to 2 seconds (financial scenarios require higher timeliness of mapping relationships).

[0078] In the implementation of the feature evolution module, the system processes the transaction log stream in real time. It uses a domain-adjusted LDA model (…). (Each topic) will batch convert logs within a 5-second window into a 50-dimensional topic distribution vector. IV values ​​are based on Calculation of equal-width bins, Laplace smoothing constant. Dynamic threshold The initial value is set to 0.3, using EWMA (attenuation coefficient). Adaptive adjustments were made. During a period of sharp market volatility, the topic distribution of the trading logs changed drastically (a large number of previously rare circuit breaker-related logs appeared), and the IV value soared to 0.8, far exceeding the dynamic threshold at the time. The system automatically created a circuit breaker event monitoring logic node in the IT tree. , =LDA_Transformer, After querying the RT tree, a standby server container with a CPU water level of only 40% was selected as the mapping target. Mapping edges were created and initialized. .

[0079] In the implementation of the drift reload module, the system calculates the PSI value every minute. During trading session switching (e.g., from pre-market auction to continuous trading), the log pattern undergoes a structural change: the logs during the pre-market auction phase primarily focus on order queuing and price calculation, while the logs during the continuous trading phase shift to focus on order matching and continuous quotes. The PSI value jumps from 0.05 to 0.35. The system triggers a double-buffered hot reload: the atomic pointer currently points to Buffer-A (which stores the LDA weights trained during the pre-market auction phase). The background training thread retrains the LDA model parameters using the most recent 10 minutes of continuous bidding log data in Buffer-B. The training process takes approximately 3 seconds. After training is complete, the system... The system performs range checks (confirming that all topic probabilities are within the range [0,1] and sum to 1) and numerical stability checks (confirming that there are no NaN or Inf values). After successful verification, the atomic pointer is switched to Buffer-B via a CAS operation. Business monitoring requests before and after the switch are processed normally without any interruption or delay.

[0080] In the implementation of the fault tolerance compensation module, when a certain order delay monitoring node ( When a container is abnormally killed by Kubernetes OOM Kill, the system retrieves the transaction rate monitoring node under the same business line in the IT tree. As a candidate: the two have an operator type similarity of 0.8 (both are KPI_Tracker type), a weight vector cosine similarity of 0.7, and a topological path distance of 2 hops. Overall similarity... After the fault occurred seconds, .because The system automatically sends a downgrade warning to the transaction operation and maintenance management console.

[0081] In the implementation of the kernel execution module, when the system detects that the P99 indicator of transaction latency has increased from 30ms to 80ms, the performance loss function is calculated as follows: (Target latency 50ms). The gradient calculation module performs a difference approximation on the number of CPU cores (discrete parameter, minimum granularity = 1 core): increasing the CPU cores of the trading container from 4 to 5 and observing the performance change, to estimate... (Negative values ​​indicate that adding cores can reduce losses). Security intervention layer verification: The target server's CPU utilization is 60%, and after adding 4 cores, it is estimated to reach 72.5%, which is below the 85% safety threshold, so the verification is passed. Frequency limiter verification: This node has not received any tuning instructions in the past 30 seconds, so the verification is passed. Circular dependency detector verification: The CPU increase operation does not involve reverse dependencies of other nodes, so the verification is passed. The system automatically issues the following kernel instructions: Expand the available CPU cores of the transaction processing process from 4 to 8 via cgroups (write cpuset.cpus="0-7"); Configure HTB queue rules for the transaction network interface via tc, increasing the guaranteed bandwidth of the transaction data stream from 200Mbps to 400Mbps. After 10 seconds of tuning execution, the system collects feedback data: P99 latency drops back to 35ms, and the loss function... The value dropped from 900 to 225, a decrease of 75%. The feedback data was written back to the RT tree to update the resource level of the target server (CPU utilization updated to 72%, bandwidth utilization updated to 65%), completing the closed loop.

[0082] Take a core network monitoring scenario of a telecommunications operator as an example. This scenario requires monitoring the operational status of core network equipment, bearer network equipment, and thousands of base stations spanning multiple provinces. Unstructured data sources include network element alarm logs, signaling trace data, and user complaint texts.

[0083] In this embodiment, JanusGraph is selected as the graph database (due to its advantages in distributed storage for large-scale graphs). The logical nodes of the IT tree are organized at the granularity of network-wide → province → city → base station cluster → single base station. Aggregation operators are used for network-wide and province-level nodes. =="Regional_Aggregator"), the city-level nodes use the Transformer encoder to perform semantic vectorization on the signaling logs ( =="Transformer_Encoder", The base station cluster and single base station level nodes use the lightweight Word2Vec model. =="W2V_Encoder", To adapt to the resource constraints of edge computing. The RT tree reflects the physical network topology: core router → aggregation switch → access device → base station hardware. The η value of the mapped edge comprehensively considers the signaling traffic share (weight 0.6) and geographical proximity (weight 0.4).

[0084] In processing user complaint texts, the system uses the [CLS] vector of a BERT model fine-tuned for the telecommunications domain. The dimension (reduced to 128 dimensions by PCA to match the vector space of the IT tree nodes) is used as the semantic embedding of the complaint text. The IV value quantifies the change in the distribution of complaint topics. When a sudden network failure occurs in a certain area, resulting in a large influx of similar complaints (such as complaints with highly concentrated semantics, such as inability to access the Internet or signal interruption), the IV value surges from 0.15 to 0.9. The system automatically refines the monitoring granularity of the area from the base station group level to the single base station level in the IT tree, creating an independent logical monitoring node for each base station in the failure area.

[0085] In the closed-loop optimization phase, when the PSI detects traffic pattern drift (such as a surge in traffic to five times the normal level for a local base station due to a large concert or sporting event), the network component of the performance loss function increases significantly. Gradient calculation module analysis reveals that the main bottlenecks are insufficient uplink bandwidth at the relevant base stations and insufficient CPU resources in the core network signaling processing process. After the security intervention layer verification passes, the kernel execution module configures TBF (Track-Based Functionality) on the relevant network interfaces via tc, compressing the bandwidth limit for low-priority traffic (such as software background updates) from 100Mbps to 20Mbps, prioritizing user internet access and voice call traffic with the released bandwidth. Simultaneously, ionice is used to upgrade the I / O scheduling class of the core network signaling processing process from best-effort (class 2, priority 4) to real-time (class 1, priority 2), ensuring that signaling processing is not affected by other I / O-intensive processes. After optimization execution, the system collects feedback data within 15 seconds and writes it back to the RT tree to verify the optimization effect: user plane latency decreased from 200ms to 60ms, and signaling processing latency decreased from 50ms to 15ms. Feedback confirmed the optimization was effective, and the closed loop was completed.

[0086] The distributed system resource processing method provided in this application constructs a two-layer heterogeneous topology mapping of the distributed system's indicator logical tree and physical resource tree in a graph database, achieving cross-layer association through dynamic mapping edges containing association strength factors. It realizes the perception of changes in business patterns through real-time feature extraction and information value calculation of unstructured signal flows. Based on this perception result, it drives adaptive optimization of the topology structure and triggers a resource tuning mechanism based on gradient calculation results. Therefore, it can solve the technical problem of persistent deviations between the monitoring topology and the actual business topology due to topology rigidity in non-stationary environments, which is detrimental to ensuring the overall performance of the distributed system, thus laying the foundation for improving the overall performance of the distributed system.

[0087] Furthermore, a two-layer heterogeneous topology structure of indicator logic tree and physical resource tree is constructed in the graph database. Cross-layer association is achieved through dynamic mapping edges containing association strength factors, and an asynchronous batch write mechanism is adopted to resolve the performance contradiction between high-frequency monitoring signals and strong consistency of graph database transactions. The feature evolution module vectorizes the unstructured signal flow and drives the automatic expansion and contraction of the topology based on the information value (IV) index. The drift reload module monitors concept drift through the population stability index (PSI) and uses a double buffering mechanism to achieve zero-downtime hot reload of operator parameters. The fault tolerance compensation module provides flexible degradation output through semantic retrieval and confidence penalty when the main path fails. The kernel execution module uses a hybrid gradient estimation strategy to calculate the optimization direction. After verification by a triple security intervention mechanism (water level verification, frequency limiting, and circular dependency detection), it automatically issues kernel control instructions for CPU affinity, network traffic shaping, and I / O priority, and feeds back the execution results to the physical resource tree to update the water level in real time, forming a complete closed-loop control. This application is applicable to non-stationary operating environments with extremely high requirements for real-time performance and availability, such as financial transactions, telecommunications networks, and industrial IoT.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0089] The embodiments of this application also provide a distributed system resource processing apparatus for executing the distributed system resource processing method provided in the above embodiments.

[0090] like Figure 3 The diagram shown is a structural schematic of a distributed system resource processing device provided in an embodiment of this application. The distributed system resource processing device 30 includes: a construction module 301, an acquisition module 302, a feature extraction module 303, a judgment module 304, a topology optimization module 305, a calculation module 306, a determination module 307, and an optimization module 308.

[0091] The system comprises the following modules: a construction module for building a distributed system's indicator logic tree and physical resource tree in a graph database; the indicator logic tree includes multiple logical nodes representing logical monitoring units, and the physical resource tree includes multiple resource nodes representing target devices; dynamic mapping edges connect the logical nodes and resource nodes, and these dynamic mapping edges include a correlation strength factor characterizing the degree of association between the logical nodes and resource nodes; an acquisition module for acquiring unstructured signal streams generated by the distributed system; a feature extraction module for extracting features from the unstructured signal streams to obtain feature vectors; and a judgment module for judging the deviation between the current feature distribution, represented by the information values ​​of the feature vectors, and the baseline distribution. The system considers whether the business model of the distributed system has changed; the topology optimization module optimizes the topology of the indicator logic tree and physical resource tree to adapt to the change in business model; the calculation module calculates the target gradient of the performance loss of the distributed system relative to the physical resource allocation of the physical resource tree when the performance status of the distributed system reflected by the indicator logic tree meets the preset resource tuning conditions; the determination module determines the target resource optimization strategy of the distributed system based on the target gradient and the original resource allocation strategy; and the tuning module performs resource tuning of the distributed system according to the target resource optimization strategy.

[0092] For a description of the features in the embodiments corresponding to the distributed system resource processing device, please refer to the relevant descriptions in the embodiments corresponding to the distributed system resource processing method, which will not be repeated here.

[0093] Embodiments of this application also provide an electronic device, such as... Figure 4 The diagram shown is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, including a processor 10 and a memory 20. The memory 20 stores a computer program, and the processor 10 is configured to run the computer program to execute the steps in any of the above-described distributed system resource processing method embodiments.

[0094] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the distributed system resource processing method when it is run.

[0095] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0096] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the distributed system resource processing method.

[0097] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the distributed system resource processing method.

[0098] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] The above provides a detailed description of a distributed system resource processing method, apparatus, and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for processing resources in a distributed system, characterized in that, include: In a graph database, a distributed system's indicator logic tree and physical resource tree are constructed. The indicator logic tree includes multiple logical nodes representing logical monitoring units, and the physical resource tree includes multiple resource nodes representing target devices. A dynamic mapping edge is provided between the logical nodes and resource nodes, and the dynamic mapping edge includes a correlation strength factor used to characterize the degree of correlation between the logical nodes and resource nodes. Obtain the unstructured signal stream generated by the distributed system; Feature extraction is performed on the unstructured signal stream to obtain a feature vector; Based on the degree of deviation between the current feature distribution, represented by the information value of the feature vector, and the baseline distribution, it is determined whether the business mode of the distributed system has changed. When it is determined that the business model of the distributed system has changed, the topology of the indicator logic tree and physical resource tree is optimized to adapt to the change in the business model and update the indicator logic tree and physical resource tree. When the indicator logic tree reflects that the performance status of the distributed system meets the preset resource optimization conditions, the performance loss of the distributed system relative to the target gradient of the physical resource allocation of the physical resource tree is calculated. Based on the target gradient and the original resource allocation strategy, the target resource optimization strategy of the distributed system is determined; Based on the target resource optimization strategy, the distributed system is optimized for resource allocation.

2. The distributed system resource processing method according to claim 1, characterized in that, The step of determining whether the business model of the distributed system has changed based on the deviation between the current feature distribution, represented by the information value of the feature vector, and the baseline distribution includes: The distribution of the feature vectors is divided into multiple bins; The information value of the feature vector is determined based on the current distribution proportion and the baseline distribution proportion of each bin; The information threshold is determined based on the historical sequence of information values ​​represented by the baseline distribution; Based on the degree of deviation between the information value of the feature vector and the information threshold, it is determined whether the business mode of the distributed system has changed.

3. The distributed system resource processing method according to claim 2, characterized in that, The method further includes: Maintain two sets of parameter buffers and an atomic pointer to the currently active buffer in memory; The population stability index of the feature vector is calculated periodically based on the current distribution ratio of each bin and the preset reference distribution ratio. When the population stability index exceeds a preset drift threshold, new weight parameters are calculated based on new data in the inactive buffer. After the new weight parameters are verified, the atomic pointer is switched to point to the active buffer through an atomic operation, and subsequent calculation requests will use the new weight parameters in the active buffer.

4. The distributed system resource processing method according to claim 1, characterized in that, The method further includes: When the main logic node on the main path in the indicator logic tree fails, the main logic node is regarded as the fault node. Search the indicator logic tree for the candidate logic node with the highest comprehensive similarity to the fault node; The confidence penalty factor of the candidate logical node is determined based on the comprehensive similarity between the candidate logical node and the faulty node, the fault occurrence time, and the preset time decay coefficient. The downgraded output result is determined based on the calculated output of the candidate logic node and the confidence penalty factor of the candidate logic node.

5. The distributed system resource processing method according to claim 1, characterized in that, When the indicator logic tree reflects the performance state of the distributed system and meets the preset resource optimization conditions, the step of calculating the performance loss of the distributed system relative to the target gradient of the physical resource allocation of the physical resource tree includes: When the indicator logic tree reflects that the performance status of the distributed system meets the preset resource tuning conditions, the target gradient of the performance loss of the distributed system relative to the physical resource allocation of the physical resource tree is calculated based on the following formula: in, Indicates the target gradient. This represents the performance loss of the distributed system. , Indicates CPU core quota. Indicates the memory limit. Indicates network bandwidth limit. The physical resource tree, which represents I / O weights, includes multiple adjustable physical resource parameters, including at least the CPU core quota, memory limit, network bandwidth limit, and I / O weights.

6. The distributed system resource processing method according to claim 1, characterized in that, The step of determining the target resource optimization strategy for the distributed system based on the target gradient and the original resource allocation strategy includes: Based on the following formula, the target resource optimization strategy of the distributed system is determined according to the target gradient and the original resource allocation strategy: in, This represents the target resource optimization strategy for the distributed system. This indicates the original resource allocation strategy. Indicates the preset learning rate, the This represents the target gradient.

7. The distributed system resource processing method according to claim 1, characterized in that, Before performing resource tuning on the distributed system according to the target resource optimization strategy, the method further includes: After the resource optimization action is executed, check whether the resource level of the target resource node and its corresponding upstream and downstream related nodes in the physical resource tree exceeds the safety threshold, and obtain the resource level verification result. Verify whether the number of tuning commands issued to the same resource node within the preset time window exceeds the upper limit, and obtain the tuning frequency verification result; In the physical resource tree, detect whether the impact path of the resource optimization action forms a loop to obtain the circular dependency verification result; When the resource level verification result, optimization frequency verification result, or circular dependency verification result is abnormal, the resource optimization action is adjusted to optimize the target resource optimization strategy.

8. The distributed system resource processing method according to claim 1, characterized in that, The step of optimizing resources in the distributed system according to the target resource optimization strategy includes: The resource tuning actions in the target resource optimization strategy are mapped to resource control instructions at the operating system kernel level. The resource control command is sent to the corresponding target resource node for execution. After the resource control command is executed at the target resource node, feedback data is collected. The physical resource tree is updated based on the feedback data.

9. A distributed system resource processing device, characterized in that, include: A construction module is used to construct an indicator logic tree and a physical resource tree of a distributed system in a graph database. The indicator logic tree includes multiple logical nodes representing logical monitoring units, and the physical resource tree includes multiple resource nodes representing target devices. A dynamic mapping edge is provided between the logical nodes and the resource nodes, and the dynamic mapping edge includes a correlation strength factor used to characterize the degree of correlation between the logical nodes and the resource nodes. An acquisition module is used to acquire the unstructured signal stream generated by the distributed system; The feature extraction module is used to extract features from the unstructured signal stream to obtain feature vectors; The judgment module is used to determine whether the business mode of the distributed system has changed based on the degree of deviation between the current feature distribution represented by the information value of the feature vector and the baseline distribution. The topology optimization module is used to optimize the topology of the indicator logic tree and physical resource tree when it is determined that the business mode of the distributed system has changed, so as to update the indicator logic tree and physical resource tree to adapt to the change in the business mode. The calculation module is used to calculate the target gradient of the performance loss of the distributed system relative to the physical resource allocation of the physical resource tree when the performance status of the distributed system reflected by the indicator logic tree meets the preset resource optimization conditions. The determination module is used to determine the target resource optimization strategy of the distributed system based on the target gradient and the original resource allocation strategy. The optimization module is used to optimize the resources of the distributed system according to the target resource optimization strategy.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the distributed system resource processing method as described in any one of claims 1 to 8 when executing the computer program.

Images