A data analysis-based computing power resource intelligent optimization method

Abstract

The application relates to the technical field of distributed computing power scheduling and big data analysis, and particularly discloses a computing power resource intelligent optimization method based on data analysis, which comprises the following steps: firstly, constructing a multi-source heterogeneous data semantic feature library, and performing vectorization processing on a search request and data; secondly, mapping computing power nodes and search tasks into a graph structure, and generating a task-node matching candidate set through graph neural network clustering; thirdly, adopting Bayesian nonparametric estimation to model resource supply and demand fluctuations, and outputting a supply and demand confidence interval; fourthly, constructing a non-cooperative game model with time delay and utilization rate as the benefits, and solving a Nash equilibrium distribution strategy; and finally, synchronously scheduling the strategy and updating a database index. The application realizes semantic matching between tasks and nodes, reduces cross-node transmission overhead, improves resource utilization and system robustness, and is suitable for green and efficient computing power scheduling in a large-scale heterogeneous data search scene.

Description

Technical Field

[0001] This invention belongs to the field of distributed computing power scheduling and big data analysis technology, specifically involving an intelligent optimization method for computing power resources based on data analysis. Background Technology

[0002] With the expansion of information retrieval databases, the processing of multi-source heterogeneous data places higher demands on computing resource scheduling. In existing technologies, mainstream computing power optimization methods mostly employ static allocation strategies based on task priorities or simple load balancing algorithms. After receiving a retrieval task, the scheduling system distributes the task to computing nodes according to a preset priority queue or the real-time CPU occupancy rate of the node. The database index structure and the computing power scheduling module are typically independent; the index is only responsible for data location, and the scheduling module is only responsible for distributing the computing load. The two lack deep interaction based on data content.

[0003] The aforementioned existing technical solutions have a core technical problem in multi-source heterogeneous big data retrieval scenarios: because the scheduling process does not combine the semantic features of the retrieval task with the inherent relationship of data storage, the retrieval task is often assigned to the computing power node where the target data is not located, causing a large amount of cross-node data transmission and calling, resulting in an imbalance in the allocation of computing power resources and excessive scheduling overhead. Summary of the Invention

[0004] The purpose of this invention is to provide a data analysis-based intelligent optimization method for computing resources, which can effectively solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A data analysis-based intelligent optimization method for computing resources includes: Based on the information retrieval database architecture, a semantic feature library of multi-source heterogeneous data is constructed, and semantic vectors are generated by vectorizing retrieval requests and data to be retrieved. The computing power nodes are mapped to graph structure nodes, the computational requirements of the retrieval task are mapped to edge weights, and a task-node matching candidate set is generated by using graph neural network clustering. By using Bayesian nonparametric estimation, a probability distribution model is performed on the fluctuation of computational resource demand and available resources, and the range of resource demand and supply is output. A non-cooperative game model is constructed using task completion delay and resource utilization rate as payoff functions to solve the computing power resource allocation strategy under Nash equilibrium. The allocation strategy is synchronized to the computing power scheduling, and the clustering results are updated to the database index structure.

[0006] Preferably, in the step of generating semantic vectors, a hierarchical attention mechanism is used to extract deep semantic features from multi-source heterogeneous data, and a sharding structure of an inverted index is dynamically constructed according to the density distribution of semantic features, so that data records with semantic similarity higher than a preset threshold are mapped to the same physical storage node.

[0007] Preferably, in the graph neural network clustering step, similarity calculation of semantic vectors corrected by geodesic distance in manifold learning is introduced to construct a dynamic topological graph structure. Attention weight matrices are added between graph convolutional layers to optimize the connection strength between nodes, thereby improving the accuracy of task and node clustering.

[0008] Preferably, in the Bayesian nonparametric estimation step, a Dirichlet process mixture model is used to fit the nonstationary time series data of resource supply and demand, a hidden Markov model is introduced to predict the transition probability of resource state, and the confidence lower limit of the resource supply interval is dynamically adjusted.

[0009] Preferably, in the solution step of the non-cooperative game model, an evolutionary game mechanism is introduced, the computing power node is modeled as a boundedly rational game participant, and the strategy adjustment process is simulated by replicating dynamic equations, converging to an evolutionarily stable strategy as an approximate solution to the Nash equilibrium.

[0010] Preferably, after constructing the sharding structure of the inverted index, the multi-armed gambler algorithm in reinforcement learning is combined to predict the hot data shards of the next moment based on the principle of locality of historical retrieval requests, and the index metadata of the hot data shards is loaded into the memory cache queue in advance.

[0011] Preferably, after optimizing the node connection strength, an incremental update strategy based on graph sampling aggregation is adopted. When a new computing node or retrieval task is added, only the affected local subgraph structure is updated, without recalculating the global graph topology, thus reducing the time complexity of clustering updates.

[0012] Preferably, when dynamically adjusting the resource supply range, the median absolute deviation method in robust statistics is introduced to remove outliers in the resource fluctuation data, and Gaussian process regression is used to smooth the data after removing outliers, thereby improving the noise resistance of probability distribution modeling.

[0013] Preferably, when solving the dynamic equations using replication, the Lagrange multiplier method is introduced to construct a penalty function containing energy consumption constraints. The upper limit of the energy consumption of the computing node is embedded as a constraint in the revenue function to ensure that the evolutionary stability strategy meets the resource constraints of green computing.

[0014] Preferably, after obtaining the computing power resource allocation strategy, a closed-loop feedback adjustment mechanism is constructed to monitor the throughput index of task execution in real time. The PID control algorithm is used to dynamically adjust the payoff weight coefficient in the game model according to the throughput deviation, so as to realize the adaptive iterative optimization of the resource allocation strategy.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a semantic feature library and performs vectorization processing. It then utilizes graph neural network clustering to match computing power nodes with retrieval tasks at the semantic level, constructing a topological association between tasks and nodes. Based on a non-cooperative game theory model, it solves for the allocation strategy under Nash equilibrium, achieving dynamic adaptation of computing power resources and retrieval tasks. This method reduces cross-node scheduling overhead in heterogeneous data retrieval and avoids the coexistence of local node overload and idleness.

[0016] 2. This invention also models the probability distribution of resource demand and supply using a Bayesian nonparametric estimation algorithm, adapting to dynamic fluctuations in resource status and improving the robustness of resource allocation strategies. By combining evolutionary game theory mechanisms and robust statistical methods, outliers in resource fluctuation data are eliminated, further ensuring the convergence and stability of resource allocation strategies in multi-task competitive environments and optimizing the overall computational efficiency of the information retrieval database. Attached Figure Description

[0017] Figure 1 This is the main flowchart of the intelligent optimization of overall computing resources in this invention; Figure 2 This is a flowchart of the semantic feature extraction and inverted index sharding process of the present invention; Figure 3 This is a flowchart of the graph neural network clustering and dynamic topology construction process of this invention; Figure 4 This is a flowchart of the resource probability modeling and supply range adjustment process of the present invention; Figure 5 This is a flowchart of the non-cooperative / evolutionary game solution process of this invention; Figure 6 This is a flowchart of the closed-loop PID feedback adaptive optimization process of the present invention. Detailed Implementation

[0018] As a preferred embodiment, please refer to the appendix. Figure 1The semantic feature processing module constructs a semantic feature library for multi-source heterogeneous data based on the information retrieval database architecture, and performs vectorization processing on the retrieval request and the data to be retrieved to generate semantic vectors. The information retrieval database architecture is a distributed sharded storage architecture, containing multiple physical storage nodes, each physical storage node corresponding to one or more computing power nodes; the multi-source heterogeneous data includes structured data, semi-structured data, and unstructured data. The process of constructing a semantic feature library is as follows: A unified semantic metadata extraction is performed on multi-source heterogeneous data. For structured data, field names, field value ranges, and field semantic tags are extracted as basic semantic metadata. For semi-structured data, entity tags, attribute key-value pairs, and hierarchical relationships are extracted as basic semantic metadata. For unstructured data, entity, attribute, and relationship triples are obtained through word segmentation, entity recognition, and relationship extraction as basic semantic metadata. All extracted semantic metadata is normalized and mapped to a unified semantic tag system to construct a semantic feature library for multi-source heterogeneous data. Each record in the semantic feature library corresponds to a set of semantic metadata for a piece of data to be retrieved, and is bound to the physical storage address and the identifier of the computing node to which the data belongs.

[0019] The specific process of vectorization is as follows: For the data to be retrieved in the semantic feature library, the corresponding semantic metadata set is input into the pre-trained BERT-base semantic coding model to generate a semantic vector of fixed dimension; For the received user retrieval request, semantic parsing and metadata extraction are performed first to obtain the request semantic metadata of the same system as the semantic feature library, and then input into the same semantic coding model to generate a retrieval request semantic vector of the same dimension as the data to be retrieved. The dimension of the semantic vector is 128-dimensional or 256-dimensional.

[0020] The graph neural network clustering module maps computing power nodes to graph structure nodes and the computational requirements of the retrieval task to edge weights, generating a candidate set of task-node matching using graph neural network clustering. The specific process is as follows: a heterogeneous graph structure is constructed, containing two types of nodes. The first type is computing power nodes, denoted as... ,in The total number of available computing power nodes in the distributed system, and each computing power node The corresponding node attribute vector includes the number of CPU cores, memory capacity, storage I / O bandwidth, current load rate, and semantic feature library shard identifier of the storage node to which it belongs; the second category is retrieval task nodes, denoted as... ,in The total number of search tasks to be scheduled, and each search task node The corresponding task attribute vector includes the semantic vector of the retrieval task, the estimated computational cost, the latency requirement, and the semantic range of the target data.

[0021] An edge is constructed between the computing power node and the retrieval task node. The initial weight of the edge is calculated based on the cosine similarity between the semantic vector of the retrieval task and the central semantic vector of the semantic feature library segment bound to the corresponding computing power node. The calculation formula is as follows: in, To retrieve task nodes With computing nodes The weight of the edges between them For retrieval tasks semantic vectors, For computing power nodes The central semantic vector of the semantic feature library shard corresponding to the storage node. For vector dot product operation, Let L2 be the L2 norm of the vector.

[0022] A two-layer graph convolutional network (GCN) is used to aggregate node features of the above heterogeneous graph structure, generating node embedding vectors. The calculation formula for graph convolution is as follows: in, For the first The node feature matrix of the layer For the initial node attribute matrix, For the first The node feature matrix of the layer The normalized adjacency matrix contains elements that correspond to the edge weights mentioned above. , For the first The trainable weight matrix of the layer, It is the ReLU nonlinear activation function.

[0023] The node embedding vectors output from the second graph convolutional layer are input into the K-Means clustering algorithm to cluster the retrieval task nodes and the computing power nodes with high matching degree into the same cluster. Each cluster corresponds to a set of task-node matching candidate sets, and each candidate set contains at least one retrieval task and at least one computing power node.

[0024] The resource probability modeling module uses Bayesian nonparametric estimation to model the probability distribution of computing resource demand and available resource fluctuations, outputting the resource demand and supply intervals. Specifically, it collects time-series data on computing resource demand and available resource fluctuations. The computing resource demand data includes historical statistics on CPU utilization, memory utilization, and I / O bandwidth utilization for each retrieval task per unit time. The available resource fluctuation data includes real-time monitoring data on the number of available CPU cores, available memory capacity, and available I / O bandwidth for each computing node per unit time. The sampling interval is set to 100ms.

[0025] A Bayesian nonparametric estimation method is used to model the probability distribution of the above time series data, and a probability distribution of resource requirements is constructed for each retrieval task. ,in For retrieval tasks The resource demand vector; for the available resources of each computing node, construct the probability distribution of its resource supply. ,in For computing power nodes The available resource vector is obtained. The distribution characteristics of the data are adaptively fitted using Bayesian nonparametric estimation, and the resource demand confidence interval for each retrieval task is output. And the confidence interval of resource supply for each computing node. The confidence level was set at 95%.

[0026] The game optimization solution module constructs a non-cooperative game model using task completion delay and resource utilization rate as payoff functions, and solves for the computing power resource allocation strategy under Nash equilibrium. Specifically, it constructs the triples of the non-cooperative game model. The set of participants For all pending retrieval tasks, there is a task execution entity. Each participant corresponds to one retrieval task, and the total number of participants is [number missing]. Strategy space In the process, each participant's strategy To select a computing node from the candidate set for task deployment. For retrieval tasks The corresponding task-node matching candidate set of computing power nodes, policy space Cartesian product of the strategy space for all participants Payoff function In the middle, the payoff function of each participant , To exclude participants The specific expression for the payoff function of the strategy combinations of all other participants is as follows: in, For retrieval tasks In strategy The task completion latency includes data reading latency, computation processing latency, and cross-node transmission latency; The overall resource utilization rate of the system is the weighted average of the CPU, memory, and I / O bandwidth utilization rates of all computing nodes. This is the time delay weighting coefficient. This is the resource utilization rate weighting coefficient. , , All are constants greater than 0.

[0027] The iterative optimal response algorithm is used to solve for the Nash equilibrium state of the above non-cooperative game model. The Nash equilibrium is defined as: for any player Arbitrary strategy All satisfy ,in The solution is a Nash equilibrium strategy combination. The specific solution process is as follows: Initialize the strategy of all participants to randomly select computing power nodes from the candidate set; in each iteration, each participant, with the strategies of other participants fixed, selects the optimal strategy that maximizes its own reward function; repeat the iteration process until the strategies of all participants no longer change. The strategy combination at this point is the computing power resource allocation strategy in the Nash equilibrium state. The allocation strategy includes the target computing power node and resource allocation quota corresponding to each retrieval task.

[0028] The computing power scheduling execution module and the index update module synchronize the allocation strategy to the computing power scheduling and update the clustering results to the database index structure. Specifically, the solved computing power resource allocation strategy is sent to the computing power scheduling execution module. Based on the allocation strategy, the computing power scheduling execution module distributes the corresponding retrieval tasks to the target computing power nodes and allocates corresponding resource quotas to the tasks. Simultaneously, the task-node matching clustering results obtained from graph neural network clustering are updated to the database index structure. Specifically, index entries for the data to be retrieved within the same cluster are mapped to the index shards of the corresponding computing power nodes, and the index routing table is updated, so that subsequent retrieval requests with the same semantic range can be directly routed to the computing power nodes of the corresponding clusters.

[0029] As a preferred embodiment, please refer to the appendix. Figure 2 In the step of generating semantic vectors, the semantic feature processing module uses a semantic coding model with a hierarchical attention mechanism to extract deep semantic features from multi-source heterogeneous data. The semantic coding model includes a three-layer structure of word-level attention layer, sentence-level attention layer and document-level attention layer.

[0030] For unstructured text data, word segmentation is first used to obtain a word sequence. A word-level attention layer assigns a corresponding attention weight to each word. The weight calculation is based on the entity importance of the word in the semantic metadata. The formula for calculating the word-level attention weight is: in, For the first Attention weight of each word For the first Word embedding vectors of words, It consists of two fully connected layers, and the summation range includes all words in the current sentence. .

[0031] The word embedding vectors, weighted by word-level attention, are summed to obtain sentence-level embedding vectors. The sentence-level attention layer assigns attention weights to each sentence, calculated based on the number of entity-relation triples contained within the sentence. The calculation logic for sentence-level attention weights is consistent with that of the word-level attention layer. The input is the sentence-level embedding vector, and the output is the sentence-level attention weights. After weighted summation, the document-level embedding vector is obtained. The document-level attention layer performs global semantic encoding on the document-level embedding vectors, combining the type labels from multi-source heterogeneous data to output the final 256-dimensional semantic vector. For structured and semi-structured data, corresponding to field-level and entity-level attention layers respectively, the same attention weight calculation method is used to extract deep semantic features and generate semantic vectors of uniform dimension.

[0032] After obtaining the semantic vectors of all data to be retrieved, the sharding structure of the inverted index is dynamically constructed based on the density distribution of semantic features. Specifically, the kernel density estimation method is used to calculate the probability density distribution of all semantic vectors in the semantic space, obtaining high-density and low-density regions in the semantic space. For high-density regions, data records with semantic similarity higher than a preset threshold are mapped to the same physical storage node. The preset threshold is set to 0.85, and the semantic similarity is calculated using cosine similarity. For low-density regions, data records with close semantic vector distances are merged and mapped to the same physical storage node, ensuring that the semantic similarity of data within the index shards corresponding to each physical storage node meets the preset requirements. The sharding structure of the inverted index corresponds one-to-one with the physical storage nodes. Each shard's inverted index entry contains the data's semantic vector, physical storage address, and the identifier of the computing power node to which it belongs. The center vector of the semantic vectors of all data records within the same shard is bound to the computing power node corresponding to that shard for subsequent graph structure edge weight calculation.

[0033] As a preferred embodiment, please refer to the appendix. Figure 3In the step of generating a node matching candidate set using graph neural network clustering, the graph neural network clustering module introduces similarity calculation of semantic vectors corrected by geodesic distance from manifold learning to construct a dynamic topological graph structure. Specifically, it constructs a k-nearest neighbor graph based on the semantic vectors of all nodes. Taking 15, in the k-nearest neighbor graph, the geodesic distance between two nodes is the length of the shortest path connecting the two nodes, which is solved using Dijkstra's algorithm. The similarity calculation formula based on the geodesic distance correction is: in, For the corrected retrieval task node With computing nodes Similarity between them For retrieving the semantic vector of the task semantic vector of the shard center corresponding to the computing power node Geodesic distance between them This is the distance scaling factor, which is the average of all geodesic distances.

[0034] Based on the corrected similarity Constructing a dynamic topology graph structure involves setting a similarity threshold, and only when... When the value exceeds a preset threshold, an edge is constructed between the retrieval task node and the computing power node, with an edge weight of . When the semantic vector of a node is updated, the similarity of the corresponding edges is recalculated, the topology of the graph is dynamically adjusted, edges with similarity below the threshold are deleted, and edges with similarity above the threshold are added.

[0035] An attention weight matrix is ​​added between graph convolutional layers to optimize the connection strength between nodes. The graph neural network adopts a two-layer graph attention network (GAT) structure, and the calculation formula for each graph attention convolutional layer is as follows: in, For the first Layer nodes The output feature vector, For nodes The set of adjacent nodes, For the first Layer nodes With adjacent nodes Attention weights between them For the first The feature transformation weight matrix of the layer, This is a non-linear activation function. Attention weights. The calculation formula is: in, For attention weight vectors, This is a vector concatenation operation. It is a non-linear activation function with a negative slope set to 0.2.

[0036] By using the attention weight matrix between layers, the contribution of different neighboring nodes to the central node is adaptively adjusted, strengthening the connection strength between nodes with high matching degree and weakening the connection influence of nodes with low matching degree. The nodes output from the second graph attention convolutional layer are embedded into the vector input spectral clustering algorithm to obtain the task-node matching candidate set.

[0037] As a preferred embodiment, please refer to the appendix. Figure 4 In the step of modeling the probability distribution of computational resource demand and available resource fluctuations through Bayesian nonparametric estimation, the resource probability modeling module uses a Dirichlet process mixture model to fit the non-stationary time series data of resource supply and demand. The expression of the Dirichlet process mixture model is as follows: in, For the Dirichlet process, For lumped parameters, As the base distribution, For the random distribution sampled from the Dirichlet process, For the first The mixture composition parameters corresponding to each observed sample For the first A single observation sample, namely time-series data of resource demand or resource supply. The distribution function of the observed samples is a Gaussian distribution.

[0038] By fitting non-stationary resource time series data using a Dirichlet process mixture model, without pre-setting the number of mixture components, the model adaptively matches the distribution pattern of the data to obtain a mixture Gaussian distribution model of resource demand and resource supply.

[0039] Hidden Markov Models (HMMs) are introduced to predict the transition probabilities of resource states. The hidden states of the HMM represent the resource supply status of computing nodes, categorized into high availability, medium availability, and low availability. The observed states are the real-time available resource monitoring data of the computing nodes. The triples of the HMM are... ,in Let be the initial state probability vector. Here is the state transition probability matrix. This represents the observation probability matrix. The parameters of a Hidden Markov Model (HMM) are obtained by training historical resource time-series data using the Baum-Welch algorithm, and the trained model is used to predict future trends. The resource state transition probability within each time step yields the probability that each computing node will be in a different available state at future times.

[0040] Based on the predicted state transition probabilities, the confidence lower bound of the resource supply range is dynamically adjusted. Specifically, when the predicted low availability state transition probability of a computing node in the future is higher than a preset threshold, the confidence lower bound of the resource supply range is increased, and the range of the resource supply range is narrowed; when the predicted high availability state transition probability is higher than a preset threshold, the confidence lower bound is decreased, and the range of the resource supply range is expanded.

[0041] As a preferred embodiment, please refer to the appendix. Figure 5 In the step of constructing a non-cooperative game model using task completion delay and resource utilization rate as payoff functions, the game optimization solution module introduces an evolutionary game mechanism, modeling computing nodes as boundedly rational game participants to construct the evolutionary game model. Specifically, the participants in the evolutionary game model are all computing nodes in the distributed system. Each computing node's strategy space consists of accepting or rejecting resource allocation requests for the corresponding retrieval task. The strategy set is {cooperate, non-cooperate}, where the cooperative strategy is to accept the allocation of the retrieval task and allocate the corresponding resources; the non-cooperative strategy is to reject the allocation of the retrieval task and reserve resources for subsequent tasks.

[0042] Construct a revenue matrix for participants, for computing power nodes. With retrieval task The profit matrix is ​​as follows: when the computing power node Select a cooperation strategy and retrieve tasks. When a match is successful, the computing node's revenue is: ,in The resource rewards after the task is completed. Resource costs for executing tasks; when computing nodes Choose a non-cooperation strategy to retrieve tasks. When a match fails, the computing node's revenue is... When computing nodes When a cooperation strategy is chosen, but resources are insufficient to meet task requirements, the benefit is... The strategy adjustment process is simulated using a replicated dynamic equation. The specific expression of the replicated dynamic equation is: in, The proportion of computing power nodes among all participants is used to select the cooperation strategy. To determine the expected returns of choosing a cooperation strategy, The average expected return for all participants, for Rate of change over time. Expected returns from choosing a cooperation strategy. Average expected return Substituting this into the replication dynamic equation, we get: By solving the above differential equations, the evolutionarily stable policy ESS is obtained. The evolutionarily stable policy satisfies: when When the policy value is reached, the derivative of the replication dynamic equation is less than 0, the policy is resistant to disturbances, and converges to the evolutionary stable policy. This policy is used as an approximate solution to the Nash equilibrium solution, and the corresponding computing resource allocation policy is obtained.

[0043] As a preferred embodiment, after constructing the shard structure of the inverted index, the multi-armed gambler algorithm in reinforcement learning is combined to predict the hot data shards for the next time step based on the principle of locality of reference of historical retrieval requests, and the index metadata of the hot data shards is loaded into the memory cache queue in advance. Specifically, a multi-armed gambler model is constructed using the UCB1 algorithm, where each arm of the multi-armed gambler model corresponds to an inverted index shard, and the reward of each arm is the number of retrieval request hits of the corresponding index shard per unit time. At each decision time... Calculate each index shard The UCB value is calculated using the following formula: in, To the time So far, index sharding Average hit payout, To the time So far, index sharding The number of times it was selected This represents the current total number of decisions.

[0044] The index shard with the largest UCB value is selected as the hot data shard for the next time step. The index metadata of this hot data shard, including semantic vectors, inverted items, and physical storage addresses, is pre-loaded into a memory cache queue. The memory cache queue uses an LRU eviction mechanism; when the cache queue is full, the least recently used index metadata is evicted. Based on the temporal and spatial locality of historical retrieval requests, the calculation of the UCB value is optimized by assigning additional weight coefficients to index shards with high hit counts in the last 10 time steps.

[0045] As a preferred embodiment, after optimizing node connection strength, an incremental update strategy based on graph sampling aggregation is adopted. When a new computing power node or retrieval task is added, only the affected local subgraph structure is updated, without recalculating the global graph topology. Specifically: when a new computing power node is added... Upon addition, the node attribute vector and the central semantic vector of the corresponding shard of the newly added computing power node are first extracted, and the corrected similarity between them and the existing retrieval task nodes is calculated. Only nodes with similarity higher than a preset threshold are connected to form a first-order local subgraph centered on the newly added computing power node.

[0046] Performing graph attention convolution on this local subgraph yields the node embedding vector of the newly added computing power node and the updated embedding vectors of the affected adjacent retrieval task nodes, without needing to re-convolve all nodes in the global graph. When a new retrieval task is added... Similarly, when adding, calculate the corrected similarity between it and the existing computing power nodes, construct a first-order local subgraph, and perform convolution operation only on this local subgraph to obtain the node embedding vector of the new retrieval task and the updated embedding vector of the affected neighboring computing power nodes.

[0047] During clustering updates, only the updated node embedding vectors are added to the original clustering results. An incremental K-Means algorithm is used to assign new nodes to the nearest cluster, or to generate a new cluster when the distance between a new node and any existing cluster is greater than a threshold. This eliminates the need to re-cluster all nodes globally, reducing the time complexity of clustering updates from... Reduce to ,in This represents the number of nodes in the local subgraph. much smaller .

[0048] As a preferred embodiment, when dynamically adjusting the resource supply range, the median absolute deviation method from robust statistics is introduced to remove outliers from resource fluctuation data. Specifically, for time series data of resource fluctuations... First, calculate the median of the data. Then, calculate the median absolute deviation of each data point from the median, i.e., the median absolute deviation (MAD): For each data point Calculate its standardized deviation value: when When the value exceeds a preset threshold of 3, the data point is identified as an outlier and removed. Gaussian process regression is then used to smooth the data after outlier removal. The model expression for Gaussian process regression is: in, This is the smoothed resource time series function. For Gaussian processes, The mean function is set to 0. The covariance function is the squared exponential covariance function: in, For signal variance, This is the length scale parameter.

[0049] By fitting the resource time series data after outlier removal using Gaussian process regression, a smoothed continuous resource fluctuation curve is obtained, eliminating noise interference in the data.

[0050] As a preferred embodiment, when solving the dynamic equations using replication, the Lagrange multiplier method is introduced to construct a penalty function containing energy consumption constraints, embedding the upper limit of the computing node's energy consumption as a constraint into the revenue function. Specifically, the energy consumption of the computing node is linearly positively correlated with CPU utilization. Real-time energy consumption The calculation formula is: in, This refers to the standby power consumption of the computing node. This represents the full-load power consumption of the computing node. This refers to the CPU utilization rate of the computing node. The energy consumption limit for the computing node is set to [value missing]. The corresponding constraints are: By embedding the above constraints into the payoff function using the Lagrange multiplier method, a payoff function with a penalty term is constructed: in, Let be the payoff function with a penalty term. For the original payoff function, Let be a Lagrange multiplier, and be a penalty coefficient greater than 0. This is a penalty for violating energy consumption constraints.

[0051] Substitute the payoff function with a penalty term into the replication dynamic equation, and resolve the evolutionary stable strategy to ensure that the final evolutionary stable strategy satisfies that the energy consumption of the computing node does not exceed the preset upper limit.

[0052] As a preferred embodiment, please refer to the appendix. Figure 6 After obtaining the computing resource allocation strategy, a closed-loop feedback adjustment mechanism is constructed to monitor the throughput index of task execution in real time. A PID control algorithm is used to dynamically adjust the payoff weight coefficients in the game theory model based on the throughput deviation, achieving adaptive iterative optimization of the resource allocation strategy. Specifically, the system's throughput index is the number of successfully completed retrieval tasks per unit time, and the target throughput value is set to... The actual throughput value obtained from real-time monitoring is Calculate throughput deviation .

[0053] The incremental PID control algorithm is used to calculate the correction amount of the revenue weight coefficient. The expression for the incremental PID control algorithm is as follows: in, The increment is used to adjust the weight at the current moment. This is the proportionality coefficient. The integral coefficient is... The differential coefficients are... The throughput deviation at the current moment. This represents the throughput deviation from the previous time step. This represents the throughput deviation between the two previous time points.

[0054] Weight-adjusted increments Dynamically adjust the time delay weight coefficient in the payoff function of the game theory model. Weighting coefficient of resource utilization rate The specific correction method is as follows: in, The time delay weighting coefficient for the current moment. The time delay weighting coefficient of the previous time step ensures and The sum is always 1, and all are in the range of 1. The range of values.

[0055] By substituting the corrected weight coefficients into the game model, resolving the Nash equilibrium, updating the computing resource allocation strategy, and realizing closed-loop adaptive optimization based on throughput feedback.

[0056] The technical features in all the above embodiments can be combined arbitrarily without conflict, and the combined technical solutions all fall within the protection scope of this invention. For those skilled in the art, any modifications, equivalent substitutions, or detailed optimizations made to the above embodiments without departing from the principle of this invention should be considered within the protection scope of this invention.

Claims

1. A method for intelligent optimization of computing resources based on data analysis, characterized in that, include: Based on the information retrieval database architecture, a semantic feature library of multi-source heterogeneous data is constructed, and semantic vectors are generated by vectorizing retrieval requests and data to be retrieved. The computing power nodes are mapped to graph structure nodes, the computational requirements of the retrieval task are mapped to edge weights, and a task-node matching candidate set is generated by clustering using graph neural networks. By using Bayesian nonparametric estimation, a probability distribution model is performed on the fluctuation of computational resource demand and available resources, and the range of resource demand and supply is output. A non-cooperative game model is constructed using task completion delay and resource utilization rate as payoff functions to solve the computing power resource allocation strategy under Nash equilibrium. The allocation strategy is synchronized to the computing power scheduling, and the clustering results are updated to the database index structure.

2. The method according to claim 1, characterized in that, In the step of generating semantic vectors, a hierarchical attention mechanism is used to extract deep semantic features from multi-source heterogeneous data, and a sharding structure of inverted index is dynamically constructed according to the density distribution of semantic features, so that data records with semantic similarity higher than a preset threshold are mapped to the same physical storage node.

3. The method according to claim 1, characterized in that, In the graph neural network clustering step, similarity calculation of semantic vectors corrected by geodesic distance in manifold learning is introduced to construct a dynamic topological graph structure. Attention weight matrices are added between graph convolutional layers to optimize the connection strength between nodes.

4. The method according to claim 1, characterized in that, In the Bayesian nonparametric estimation step, a Dirichlet process mixture model is used to fit the nonstationary time series data of resource supply and demand, a hidden Markov model is introduced to predict the transition probability of resource state, and the confidence lower limit of the resource supply interval is dynamically adjusted.

5. The method according to claim 1, characterized in that, In the solution steps of the non-cooperative game model, an evolutionary game mechanism is introduced, which models the computing nodes as boundedly rational game participants and uses the replication dynamic equation to simulate the strategy adjustment process, converging to an evolutionarily stable strategy as an approximate solution to the Nash equilibrium.

6. The method according to claim 2, characterized in that, After constructing the sharded structure of the inverted index, the multi-armed gambler algorithm in reinforcement learning is combined to predict the hot data shards in the next moment based on the principle of locality of historical retrieval requests, and the index metadata of the hot data shards is loaded into the memory cache queue in advance.

7. The method according to claim 3, characterized in that, After optimizing node connection strength, an incremental update strategy based on graph sampling aggregation is adopted. When a new computing node or retrieval task is added, only the affected local subgraph structure is updated, without recalculating the global graph topology.

8. The method according to claim 4, characterized in that, When dynamically adjusting the resource supply range, the median absolute deviation method from robust statistics is introduced to remove outliers in the resource fluctuation data. Gaussian process regression is then used to smooth the data after removing outliers, thereby improving the noise resistance of probability distribution modeling.

9. The method according to claim 5, characterized in that, When solving the dynamic equations using replication, the Lagrange multiplier method is introduced to construct a penalty function containing energy consumption constraints, and the energy consumption limit of the computing node is embedded as a constraint condition into the revenue function.

10. The method according to claim 9, characterized in that, After obtaining the computing power resource allocation strategy, a closed-loop feedback adjustment mechanism is constructed to monitor the throughput indicators of task execution in real time, and the payoff weight coefficients in the game model are dynamically adjusted based on the throughput deviation using a PID control algorithm.

Images