Intelligent heterogeneous metadata scheduling optimization method and system based on reinforcement learning algorithm

Abstract

The invention provides a reinforcement learning algorithm-based heterogeneous metadata intelligent scheduling optimization method and system, and relates to the technical field of heterogeneous data, and the method comprises the steps: constructing a multi-level dynamic feature extraction mechanism, carrying out the feature decomposition of multi-source heterogeneous metadata through a self-attention mechanism, and generating a feature matrix set; and feature fusion is carried out through a transformer encoder. And training a dual-channel neural network scheduling strategy model based on the fusion feature vector, realizing hierarchical optimization in combination with deep reinforcement learning, generating a scheduling action candidate set, and evaluating the performance of the scheduling action candidate set. The method effectively improves system load balance, access delay, resource utilization rate and data locality, and has remarkable technical advantages.

Description

Technical Field

[0001] The present invention relates to heterogeneous data technology, and in particular to a heterogeneous metadata intelligent scheduling optimization method and system based on a reinforcement learning algorithm. Background Art

[0002] In modern computing environments, scheduling optimization of heterogeneous metadata systems has become an important research direction for improving system performance and resource utilization. With the surge in data volume and the diversification of application scenarios, traditional scheduling methods are facing more and more challenges. Existing technologies usually rely on static rules or simple heuristic algorithms, which are difficult to adapt to dynamically changing loads and resource conditions.

[0003] First, traditional scheduling methods lack the ability to dynamically analyze real-time metadata and cannot respond to changes in system status in a timely manner, resulting in delayed scheduling decisions. Second, existing technologies often rely on fixed feature sets in feature extraction and fail to fully explore the potential information of multi-source heterogeneous metadata, affecting the accuracy of scheduling strategies. Finally, existing optimization mechanisms usually only focus on a single goal and fail to comprehensively consider multi-dimensional factors such as system load balancing, access latency, and resource utilization, resulting in unsatisfactory scheduling results. Summary of the invention

[0004] The embodiments of the present invention provide a heterogeneous metadata intelligent scheduling optimization method and system based on a reinforcement learning algorithm, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention, Provides a heterogeneous metadata intelligent scheduling optimization method based on reinforcement learning algorithm, including: Construct a multi-level dynamic feature extraction mechanism, obtain real-time metadata information in a multi-source heterogeneous metadata system through a distributed data acquisition engine, perform feature decomposition on the real-time metadata information using a self-attention mechanism, and obtain a feature matrix set of metadata, wherein the feature matrix set includes a type feature matrix, a timing feature matrix, a relationship feature matrix, and a load feature matrix; input the feature matrix set into a transformer encoder for feature fusion to generate a fused feature vector, and construct a feature vector space through an adaptive dimensionality reduction algorithm; train a scheduling strategy model of a dual-channel neural network structure based on the feature vector space, wherein a graph convolutional neural network channel extracts spatial features based on the type feature matrix and the relationship feature matrix, and a gated recurrent unit network channel extracts timing features based on the timing feature matrix and the load feature matrix, and outputs initial scheduling strategy parameters; According to the initial scheduling strategy parameters, a hierarchical optimization mechanism based on deep reinforcement learning is established, and the fused feature vector is used as the state input to generate a candidate set of scheduling actions; a multi-objective evaluation function is designed to evaluate the candidate actions, and the system load balance score, access delay score, resource utilization score and data locality score are calculated respectively through an adaptive weight system, and the system load balance, the access delay score, the resource utilization score and the data locality score are weightedly combined to obtain a multi-dimensional system performance comprehensive score, and a comprehensive reward function is constructed based on the multi-dimensional system performance comprehensive score; a dual-objective Q network structure is used for strategy optimization, wherein the online network updates the strategy parameters in real time based on the comprehensive reward function, the target network maintains training stability through a soft update mechanism, and the optimized strategy parameters are output as a scheduling decision sequence; The scheduling decision sequence is input into a distributed scheduling execution engine, and metadata scheduling operations are performed through a two-phase submission protocol; an adaptive monitoring threshold is set based on the fused feature vector, and a distributed monitoring network is deployed to collect system operation status data in real time; when the monitoring data deviates from the adaptive monitoring threshold, a difference analysis is performed between the monitoring data and the original fused feature vector, and a hierarchical scheduling optimization strategy is started according to the degree of difference, an incremental adjustment method is adopted for local anomalies, and the global anomaly triggering strategy is re-optimized; the optimized scheduling effect data is used as a new training sample, and distributed incremental training is performed on the dual-channel neural network and the dual-objective Q network through a federated learning framework to achieve continuous optimization of the scheduling strategy model.

[0006] According to the initial scheduling strategy parameters, a hierarchical optimization mechanism based on deep reinforcement learning is established, and the fused feature vector is used as the state input to generate a candidate set of scheduling actions; a multi-objective evaluation function is designed to evaluate the candidate actions, and the system load balance score, access delay score, resource utilization score and data locality score are calculated respectively through an adaptive weight system, including: A three-level linkage optimization architecture is constructed based on the initial scheduling strategy parameters, and the three-level linkage optimization architecture includes a strategy layer and an execution layer; the strategy layer has a built-in deep deterministic policy gradient network with a dual exploration mechanism, and the execution layer has a built-in self-correcting four-dimensional performance evaluation model; the deep deterministic policy gradient network adopts a four-layer fully connected structure, the number of nodes in the input layer is the same as the dimension of the fusion feature vector, and the number of nodes in the output layer is the same as the dimension of the action space; The fused feature vector is input into the deep deterministic policy gradient network, and the parameter distribution of the deep deterministic policy gradient network is modeled by hierarchical Bayesian modeling to obtain a hierarchical network weight distribution. A Thompson sampling method is used to obtain multiple groups of network parameter samples from each layer of the hierarchical network weight distribution to form a parameter sample set, and Gaussian noise perturbation is introduced into the parameter sample set; the parameter sample sets after adding the Gaussian noise perturbation are respectively loaded into the deep deterministic policy gradient network to generate multiple policy network instances, and the multiple policy network instances independently generate scheduling action candidate sets based on the initial scheduling policy parameters; The scheduling action candidate set is input into the four-dimensional performance evaluation model of the execution layer. The four-dimensional performance evaluation model sets a self-correction threshold for the evaluation result of each dimension, and triggers a re-evaluation mechanism when the evaluation result exceeds the self-correction threshold. The load balancing score, access delay score, resource utilization score and data locality score are calculated respectively through the four-dimensional performance evaluation model.

[0007] A comprehensive reward function is constructed based on the comprehensive score of the multi-dimensional system performance; a dual-objective Q network structure is used for policy optimization, wherein the online network updates the policy parameters in real time based on the comprehensive reward function, and the target network maintains the training stability through a soft update mechanism, including: A comprehensive reward function is constructed based on the multi-dimensional system performance comprehensive score, and the difference between the multi-dimensional system performance comprehensive score at the current moment and the previous moment is calculated to obtain a performance change value; the multi-dimensional system performance comprehensive score is set as a basic reward item, and the performance change value is set as a priority reward item; based on the accumulated reward and average reward during the training process, the combined weights of the basic reward item and the priority reward item are adaptively adjusted to obtain the comprehensive reward function; A dual-objective Q network structure is used for strategy optimization to construct an online network and a target network with a shared network structure; the current system state is input into the online network to obtain the current action value, and the next moment system state is input into the target network to obtain the target state value; the current action value, the target state value and the comprehensive reward function are substituted into the Bellman equation to calculate the temporal difference error; Calculating sample priority based on the absolute value of the temporal difference error; storing historical interaction samples and corresponding sample priorities in an experience replay pool; performing importance sampling according to the sample priorities to obtain training sample batches; inputting the training sample batches into the online network, and updating the strategy parameters of the online network in real time based on the temporal difference error through a back propagation algorithm; The soft update mechanism is adopted to update the policy parameters of the target network, and the updated policy parameters of the online network and the policy parameters of the target network are weighted averaged to obtain the updated parameters of the target network; wherein the weight of the weighted average is determined by a preset soft update coefficient, and the smoothness of the target network parameter update is maintained by the soft update coefficient.

[0008] Calculating sample priority based on the absolute value of the temporal difference error; storing historical interaction samples and corresponding sample priorities in an experience replay pool; performing importance sampling according to the sample priorities to obtain training sample batches; inputting the training sample batches into the online network, and updating the strategy parameters of the online network in real time based on the temporal difference error through a back propagation algorithm, including: The absolute value of the timing difference error is added to the priority reference value and then an exponential operation is performed to obtain the sample priority, wherein the priority reference value is used to ensure that all samples have a non-zero sampling probability; The historical interaction samples and the sample priorities form a sample item and store it in an experience replay pool of a priority queue structure, wherein the priority queue structure includes leaf nodes and non-leaf nodes, the leaf nodes store the sample items, and the non-leaf nodes store the priority sum of the subtrees; Calculate the sampling probability based on the sample priority, the sampling probability monotonically increases with the sample priority, and perform stratified sampling according to the sampling probability to obtain a training sample batch; Calculating the importance weight of each sample in the training sample batch, wherein the importance weight is used to compensate for the sample distribution deviation caused by priority sampling, and the importance weight is calculated by an adjustable importance sampling index; Inputting the training sample batches into the online network, calculating a weighted temporal difference loss function based on the importance weights, and calculating the gradient of the loss function with respect to the online network parameters by a back propagation algorithm; The gradient calculation formula of the network parameters is as follows: ; Among them, ∇ θ L is the gradient of the loss function with respect to the parameter θ, N is the size of the experience replay pool, and w i is the importance weight of the i-th sample, y i is the target Q value of the i-th sample, Q(s i ,a i ;θ) is the state-action pair of the online network (s i ,a i )’s predicted Q value, ∇ θ Q is the gradient of the Q value with respect to the parameter θ.

[0009] Based on the fused feature vector, an adaptive monitoring threshold is set, and a distributed monitoring network is deployed to collect system operation status data in real time; when the monitoring data deviates from the adaptive monitoring threshold, the monitoring data and the original fused feature vector are analyzed for differences, and a hierarchical scheduling optimization strategy is started according to the degree of difference, and an incremental adjustment method is adopted for local anomalies. The global anomaly triggering strategy is re-optimized, including: Establishing a dynamic baseline based on the fused feature vector, calculating an adaptive adjustment factor in combination with system operating environment parameters, and multiplying the dynamic baseline by the adaptive adjustment factor to obtain an adaptive monitoring threshold; Deploy a three-layer distributed monitoring network to collect real-time operating status data of the system. The three-layer distributed monitoring network consists of a local monitoring agent, a regional monitoring node, and a global monitoring center. The device-level status data collected by the local monitoring agent is aggregated by the regional monitoring node and transmitted to the global monitoring center; The global monitoring center performs feature extraction and fusion processing on the received real-time operating status data of the system to generate a real-time fusion feature vector, and compares the real-time fusion feature vector with the adaptive monitoring threshold; When the real-time fused feature vector deviates from the adaptive monitoring threshold, calculating a standardized difference value of the real-time fused feature vector relative to the initial fused feature vector, and calculating a comprehensive difference degree based on the standardized difference value; The comprehensive difference degree is graded according to a preset difference degree threshold. When the comprehensive difference degree is less than the difference degree threshold, it is determined as a local abnormality and triggers an incremental adjustment strategy. When the comprehensive difference degree is greater than the difference degree threshold or equal to the difference degree threshold, it is determined as a global abnormality and triggers a re-optimization strategy.

[0010] The optimized scheduling effect data is used as a new training sample, and the dual-channel neural network and the dual-objective Q network are subjected to distributed incremental training through a federated learning framework to achieve continuous optimization of the scheduling strategy model, including: Collecting scheduling effect evaluation data as training samples, the scheduling effect evaluation data includes system state vectors before and after scheduling execution, actually executed scheduling action sequences, resource utilization data and energy consumption data, and performing normalization preprocessing on the scheduling effect evaluation data to obtain standardized training samples; Distributing the standardized training samples to local training nodes through a federated learning framework, and the local training nodes construct local model copies including a dual-channel neural network and a dual-objective Q network; At the local training node, inputting the state feature vector in the standardized training sample into the state representation network of the dual-channel neural network to obtain a state code, and generating a new scheduling action based on the state code by the action generation network of the dual-channel neural network; Comparing the new scheduled action with the actual executed action in the standardized training sample, calculating the policy gradient and updating the parameters of the action generation network; Inputting the state code and the new scheduling action into the dual-objective Q network, calculating a resource utilization Q value and an energy efficiency Q value, weighting the resource utilization Q value and the energy efficiency Q value according to the performance indicator value in the standardized training sample to obtain a comprehensive Q value, and updating the parameters of the dual-objective Q network based on the comprehensive Q value; Periodically triggering a federated averaging algorithm to collect model parameters of the local training nodes for global aggregation to obtain global model parameters, and distributing the global model parameters to the local training nodes to update the local model copies; New scheduling effect evaluation data is collected as incremental training samples, and the dual-channel neural network and the dual-objective Q network are continuously optimized through the federated learning framework.

[0011] Inputting the state code and the new scheduling action into the dual-objective Q network, calculating the resource utilization Q value and the energy efficiency Q value, weighting the resource utilization Q value and the energy efficiency Q value according to the performance indicator value in the standardized training sample to obtain a comprehensive Q value, and updating the dual-objective Q network parameters based on the comprehensive Q value includes: The state code and the new scheduling action are concatenated to obtain a state action vector, and the state action vector is used as a common input of a resource utilization evaluation network and an energy efficiency evaluation network; Inputting the state action vector into the resource utilization evaluation network, performing nonlinear transformation through the first hidden layer and the second hidden layer of the resource utilization evaluation network to obtain resource utilization characteristics, and calculating the resource utilization Q value based on the resource utilization characteristics; d. The resource utilization Q value calculation formula is as follows: ; Among them, Q r (s t ,a t ) is the resource utilization Q value, s t is the state code at time t, a t is the action at time t, W3 is the weight matrix of the output layer, h2 is the resource utilization feature, and b3 is the bias scalar of the output layer; Inputting the state action vector into the energy efficiency evaluation network, performing nonlinear transformation through the first hidden layer and the second hidden layer of the energy efficiency evaluation network to obtain energy efficiency characteristics, and calculating the energy efficiency Q value based on the energy efficiency characteristics; The energy efficiency Q value calculation formula is as follows: ; Among them, Q e (s t ,a t ) is the energy efficiency Q value, W3 e is the weight matrix of the output layer of the energy efficiency evaluation network, h2 e is the energy efficiency characteristic, b3 e The bias scalar of the output layer of the network for energy efficiency evaluation; Calculate the relative change of resource utilization based on the performance indicators in the current batch of training samples, input the relative change of resource utilization into the adaptive weight calculation unit, and calculate the resource utilization weight and energy efficiency weight by the adaptive weight calculation unit; Multiplying the resource utilization rate Q value by the resource utilization rate weight to obtain a weighted resource utilization rate Q value, multiplying the energy efficiency Q value by the energy efficiency weight to obtain a weighted energy efficiency Q value, and adding the weighted resource utilization rate Q value and the weighted energy efficiency Q value to obtain a comprehensive Q value; A temporal difference loss function is constructed based on the comprehensive Q value, and network parameters of the resource utilization evaluation network and the energy efficiency evaluation network are simultaneously updated by minimizing the temporal difference loss function.

[0012] According to a second aspect of the embodiments of the present invention, Provides a heterogeneous metadata intelligent scheduling optimization system based on reinforcement learning algorithms, including: The first unit is used to construct a multi-level dynamic feature extraction mechanism, obtain real-time metadata information in a multi-source heterogeneous metadata system through a distributed data acquisition engine, and perform feature decomposition on the real-time metadata information using a self-attention mechanism to obtain a feature matrix set of metadata, wherein the feature matrix set includes a type feature matrix, a timing feature matrix, a relationship feature matrix, and a load feature matrix; the feature matrix set is input into a transformer encoder for feature fusion to generate a fused feature vector, and a feature vector space is constructed through an adaptive dimensionality reduction algorithm; a scheduling strategy model of a dual-channel neural network structure is trained based on the feature vector space, wherein a graph convolutional neural network channel extracts spatial features based on the type feature matrix and the relationship feature matrix, and a gated recurrent unit network channel extracts timing features based on the timing feature matrix and the load feature matrix, and outputs initial scheduling strategy parameters; The second unit is used to establish a hierarchical optimization mechanism based on deep reinforcement learning according to the initial scheduling strategy parameters, use the fused feature vector as the state input, and generate a candidate set of scheduling actions; design a multi-objective evaluation function to evaluate the candidate actions, calculate the system load balance score, access delay score, resource utilization score and data locality score respectively through an adaptive weight system, and perform a weighted combination of the system load balance, the access delay score, the resource utilization score and the data locality score to obtain a multi-dimensional system performance comprehensive score, and construct a comprehensive reward function based on the multi-dimensional system performance comprehensive score; adopt a dual-objective Q network structure to perform strategy optimization, wherein the online network updates the strategy parameters in real time based on the comprehensive reward function, the target network maintains training stability through a soft update mechanism, and outputs the optimized strategy parameters as a scheduling decision sequence; The third unit is used to input the scheduling decision sequence into a distributed scheduling execution engine, and perform metadata scheduling operations through a two-phase submission protocol; set an adaptive monitoring threshold based on the fused feature vector, and deploy a distributed monitoring network to collect system operation status data in real time; when the monitoring data deviates from the adaptive monitoring threshold, perform a difference analysis between the monitoring data and the original fused feature vector, start a hierarchical scheduling optimization strategy according to the degree of difference, adopt an incremental adjustment method for local anomalies, and re-optimize the global anomaly triggering strategy; use the optimized scheduling effect data as a new training sample, and perform distributed incremental training on the dual-channel neural network and the dual-objective Q network through a federated learning framework to achieve continuous optimization of the scheduling strategy model.

[0013] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0014] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0015] The beneficial effects of this application are as follows: 1. Improve scheduling efficiency: By building a multi-level dynamic feature extraction mechanism and an adaptive dimensionality reduction algorithm, it is possible to acquire and fuse multi-source heterogeneous metadata features in real time, thereby optimizing the scheduling strategy and significantly improving the scheduling efficiency of the system.

[0016] 2. Enhance system performance: Use a multi-objective evaluation function to evaluate scheduling actions, comprehensively consider multiple dimensions such as system load balancing, access latency, resource utilization, and data locality, which can effectively improve the overall system performance and ensure the rational use of resources.

[0017] 3. Achieve continuous optimization: Through the federated learning framework, the scheduling strategy model is trained incrementally in a distributed manner, which can continuously optimize the scheduling strategy based on real-time monitoring data and scheduling effect data, and achieve adaptive adjustment and continuous improvement of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a process of a heterogeneous metadata intelligent scheduling optimization method based on a reinforcement learning algorithm according to an embodiment of the present invention; Figure 2 It is a structural diagram of a heterogeneous metadata intelligent scheduling optimization system based on a reinforcement learning algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0021] Figure 1 FIG. 1 is a flow chart of a heterogeneous metadata intelligent scheduling optimization method based on a reinforcement learning algorithm according to an embodiment of the present invention. Figure 1 As shown, the method includes: S11. Construct a multi-level dynamic feature extraction mechanism, obtain real-time metadata information in a multi-source heterogeneous metadata system through a distributed data acquisition engine, perform feature decomposition on the real-time metadata information using a self-attention mechanism, and obtain a feature matrix set of metadata, wherein the feature matrix set includes a type feature matrix, a timing feature matrix, a relationship feature matrix, and a load feature matrix; input the feature matrix set into a transformer encoder for feature fusion, generate a fused feature vector, and construct a feature vector space through an adaptive dimensionality reduction algorithm; train a scheduling strategy model of a dual-channel neural network structure based on the feature vector space, wherein a graph convolutional neural network channel extracts spatial features based on the type feature matrix and the relationship feature matrix, and a gated recurrent unit network channel extracts timing features based on the timing feature matrix and the load feature matrix, and outputs initial scheduling strategy parameters; S12. According to the initial scheduling strategy parameters, a hierarchical optimization mechanism based on deep reinforcement learning is established, and the fused feature vector is used as the state input to generate a candidate set of scheduling actions; a multi-objective evaluation function is designed to evaluate the candidate actions, and the system load balance score, access delay score, resource utilization score and data locality score are calculated respectively through an adaptive weight system, and the system load balance, the access delay score, the resource utilization score and the data locality score are weightedly combined to obtain a multi-dimensional system performance comprehensive score, and a comprehensive reward function is constructed based on the multi-dimensional system performance comprehensive score; a dual-objective Q network structure is used for strategy optimization, wherein the online network updates the strategy parameters in real time based on the comprehensive reward function, and the target network maintains training stability through a soft update mechanism, and the optimized strategy parameters are output as a scheduling decision sequence; S13. Input the scheduling decision sequence into the distributed scheduling execution engine, and execute the metadata scheduling operation through the two-stage submission protocol; set an adaptive monitoring threshold based on the fused feature vector, and deploy a distributed monitoring network to collect system operation status data in real time; when the monitoring data deviates from the adaptive monitoring threshold, perform a difference analysis between the monitoring data and the original fused feature vector, start a hierarchical scheduling optimization strategy according to the degree of difference, adopt an incremental adjustment method for local anomalies, and re-optimize the global anomaly triggering strategy; use the optimized scheduling effect data as a new training sample, and perform distributed incremental training on the dual-channel neural network and dual-objective Q network through a federated learning framework to achieve continuous optimization of the scheduling strategy model.

[0022] In an optional implementation, according to the initial scheduling strategy parameters, a hierarchical optimization mechanism based on deep reinforcement learning is established, the fused feature vector is used as a state input, and a candidate set of scheduling actions is generated; a multi-objective evaluation function is designed to evaluate the candidate actions, and the system load balance score, access delay score, resource utilization score and data locality score are calculated respectively through an adaptive weight system, including: A three-level linkage optimization architecture is constructed based on the initial scheduling strategy parameters, and the three-level linkage optimization architecture includes a strategy layer and an execution layer; the strategy layer has a built-in deep deterministic policy gradient network with a dual exploration mechanism, and the execution layer has a built-in self-correcting four-dimensional performance evaluation model; the deep deterministic policy gradient network adopts a four-layer fully connected structure, the number of nodes in the input layer is the same as the dimension of the fusion feature vector, and the number of nodes in the output layer is the same as the dimension of the action space; The fused feature vector is input into the deep deterministic policy gradient network, and the parameter distribution of the deep deterministic policy gradient network is modeled by hierarchical Bayesian modeling to obtain a hierarchical network weight distribution. A Thompson sampling method is used to obtain multiple groups of network parameter samples from each layer of the hierarchical network weight distribution to form a parameter sample set, and Gaussian noise perturbation is introduced into the parameter sample set; the parameter sample sets after adding the Gaussian noise perturbation are respectively loaded into the deep deterministic policy gradient network to generate multiple policy network instances, and the multiple policy network instances independently generate scheduling action candidate sets based on the initial scheduling policy parameters; The scheduling action candidate set is input into the four-dimensional performance evaluation model of the execution layer. The four-dimensional performance evaluation model sets a self-correction threshold for the evaluation result of each dimension, and triggers a re-evaluation mechanism when the evaluation result exceeds the self-correction threshold. The load balancing score, access delay score, resource utilization score and data locality score are calculated respectively through the four-dimensional performance evaluation model.

[0023] First, a comprehensive reward function is constructed based on the comprehensive score of multi-dimensional system performance. The performance change value is obtained by calculating the difference between the comprehensive score of system performance at the current moment and the previous moment. For example, if the comprehensive performance score at the current moment is 0.85 and the previous moment is 0.78, the performance change value is 0.07. The comprehensive performance score is used as the basic reward item, and the performance change value is used as the priority reward item. In practical applications, the initial basic reward item weight is set to 0.6, and the priority reward item weight is set to 0.4. During the training process, when the cumulative reward exceeds 1000 and the average reward is greater than 0.8, the weight ratio is dynamically adjusted, the basic reward item weight is adjusted to 0.5, and the priority reward item weight is adjusted to 0.5 to obtain a comprehensive reward function.

[0024] Then, the dual-objective Q network structure is used for policy optimization. The online network and target network with the same network structure are constructed, including an input layer, three hidden layers, and an output layer. The input layer receives the current system state information, including resource utilization, load distribution and other characteristics. The current state is input into the online network to obtain the action value evaluation result. For example, the evaluation value of a certain scheduling action is 0.92. At the same time, the state at the next moment is input into the target network to obtain the target state value of 0.88. The temporal difference error is calculated by combining the above action value, state value and comprehensive reward.

[0025] Then, the sample priority is calculated based on the absolute value of the time difference error. For example, if the absolute value of the time difference error is 0.15, the corresponding sample priority is 0.85. The historical interaction samples and priority information are stored in the experience replay pool, and the replay pool capacity is set to 10,000. Importance sampling is performed according to the sample priority. The higher the priority, the greater the probability of being sampled. The training sample batch of size 256 is obtained by sampling and input into the online network for training. The online network parameters are updated through the back propagation algorithm so that the predicted value gradually approaches the target value.

[0026] Finally, the soft update mechanism is used to update the target network parameters. The soft update coefficient is set to 0.01, and the updated online network parameters and the target network parameters are weighted averaged. In the specific implementation, the new target network parameter value is equal to 0.99 times the original target network parameter value plus 0.01 times the online network parameter value. A smaller soft update coefficient ensures that the target network parameters are updated smoothly and improves training stability.

[0027] The solution of this application can: In terms of algorithm performance: by constructing a comprehensive reward function and dynamically adjusting weights, the model's ability to adapt to environmental changes is improved; the dual-objective Q network structure and soft update mechanism are adopted to significantly enhance training stability and avoid policy shocks. In terms of system efficiency: the priority-based experience replay mechanism improves sample utilization efficiency; importance sampling ensures that high-value samples can be fully trained; the soft update mechanism reduces the volatility of parameter updates. In terms of practicality: the algorithm design fully considers actual application needs and has strong scalability; the parameter configuration is flexible and adjustable, which is convenient for optimization in different scenarios; the training process is stable and controllable, suitable for engineering practice deployment.

[0028] In an optional implementation, a comprehensive reward function is constructed based on the comprehensive score of the multi-dimensional system performance; a dual-objective Q network structure is used for policy optimization, wherein the online network updates the policy parameters in real time based on the comprehensive reward function, and the target network maintains training stability through a soft update mechanism, including: A comprehensive reward function is constructed based on the multi-dimensional system performance comprehensive score, and the difference between the multi-dimensional system performance comprehensive score at the current moment and the previous moment is calculated to obtain a performance change value; the multi-dimensional system performance comprehensive score is set as a basic reward item, and the performance change value is set as a priority reward item; based on the accumulated reward and average reward during the training process, the combined weights of the basic reward item and the priority reward item are adaptively adjusted to obtain the comprehensive reward function; A dual-objective Q network structure is used for strategy optimization to construct an online network and a target network with a shared network structure; the current system state is input into the online network to obtain the current action value, and the next moment system state is input into the target network to obtain the target state value; the current action value, the target state value and the comprehensive reward function are substituted into the Bellman equation to calculate the temporal difference error; Calculating sample priority based on the absolute value of the temporal difference error; storing historical interaction samples and corresponding sample priorities in an experience replay pool; performing importance sampling according to the sample priorities to obtain training sample batches; inputting the training sample batches into the online network, and updating the strategy parameters of the online network in real time based on the temporal difference error through a back propagation algorithm; The soft update mechanism is adopted to update the policy parameters of the target network, and the updated policy parameters of the online network and the policy parameters of the target network are weighted averaged to obtain the updated parameters of the target network; wherein the weight of the weighted average is determined by a preset soft update coefficient, and the smoothness of the target network parameter update is maintained by the soft update coefficient.

[0029] First, the basis for building a comprehensive reward function is a comprehensive score of multi-dimensional system performance. This score is obtained by evaluating the performance of the system in different dimensions. The score of each dimension can be quantified by set standards, such as response time, resource utilization, and system stability. By comparing the comprehensive score at the current moment with the previous moment, the performance change value is calculated. This change value reflects the improvement or decline of system performance.

[0030] Next, the comprehensive score of multi-dimensional system performance is used as the basic reward item, and the performance change value is used as the priority reward item. The basic reward item provides basic feedback for the system, while the priority reward item emphasizes the dynamic changes in performance. By analyzing the cumulative reward and average reward during the training process, the combined weights of these two reward items are dynamically adjusted to ensure that the comprehensive reward function can adapt to different training stages and environmental changes.

[0031] In terms of policy optimization, a dual-objective Q network structure is adopted. The online network and the target network share the same network structure. The online network is responsible for real-time processing of the current system state and outputting the value of the current action, while the target network is used to evaluate the value of the system state at the next moment. By combining the current action value, the target state value and the comprehensive reward function, the Bellman equation is used to calculate the temporal difference error. This error provides an important basis for subsequent policy updates.

[0032] In the calculation of sample priority, the importance of the sample is determined based on the absolute value of the time difference error. The historical interaction samples and their corresponding priorities are stored in the experience replay pool for subsequent training. Importance sampling is performed according to the sample priority to obtain training sample batches. These samples are input into the online network, and the policy parameters are updated in real time through the back propagation algorithm to improve the learning efficiency of the network.

[0033] Finally, a soft update mechanism is used to update the policy parameters of the target network. The updated parameters of the online network are weighted averaged with the parameters of the target network to ensure a smooth parameter update process for the target network. The weight of the weighted average is determined by the preset soft update coefficient. This mechanism effectively avoids drastic fluctuations in the parameter update process, thereby improving the stability of training.

[0034] The solution of this application can: Through the dynamic adjustment of the comprehensive reward function, the system can better adapt to different environmental changes, improve learning efficiency and policy optimization effects. The application of the dual-objective Q network structure enhances the stability of policy optimization, reduces fluctuations during training, and makes the system more reliable in complex environments. The introduction of the soft update mechanism ensures the smooth update of the target network parameters, further improving the overall performance and responsiveness of the system.

[0035] In an optional implementation, the sample priority is calculated based on the absolute value of the temporal difference error; the historical interaction samples and the corresponding sample priorities are stored in an experience replay pool; importance sampling is performed according to the sample priorities to obtain training sample batches; the training sample batches are input into the online network, and the strategy parameters of the online network are updated in real time based on the temporal difference error through a back propagation algorithm, including: The absolute value of the timing difference error is added to the priority reference value and then an exponential operation is performed to obtain the sample priority, wherein the priority reference value is used to ensure that all samples have a non-zero sampling probability; The historical interaction samples and the sample priorities form a sample item and store it in an experience replay pool of a priority queue structure, wherein the priority queue structure includes leaf nodes and non-leaf nodes, the leaf nodes store the sample items, and the non-leaf nodes store the priority sum of the subtrees; Calculate the sampling probability based on the sample priority, the sampling probability monotonically increases with the sample priority, and perform stratified sampling according to the sampling probability to obtain a training sample batch; Calculating the importance weight of each sample in the training sample batch, wherein the importance weight is used to compensate for the sample distribution deviation caused by priority sampling, and the importance weight is calculated by an adjustable importance sampling index; Inputting the training sample batches into the online network, calculating a weighted temporal difference loss function based on the importance weights, and calculating the gradient of the loss function with respect to the online network parameters by a back propagation algorithm; The gradient calculation formula of the network parameters is as follows: ; Among them, ∇ θ L is the gradient of the loss function with respect to the parameter θ, N is the size of the experience replay pool, and w i is the importance weight of the i-th sample, y i is the target Q value of the i-th sample, Q(s i ,a i ;θ) is the state-action pair of the online network (s i ,a i )’s predicted Q value, ∇ θ Q is the gradient of the Q value with respect to the parameter θ.

[0036] First, collect historical interaction samples and calculate the temporal difference error of each sample. The temporal difference error refers to the difference between the current state and the next state, reflecting the performance of the model under the current decision. By processing the absolute values ​​of these errors, the priority of each sample can be obtained. The priority is calculated by adding the absolute value to a preset priority benchmark value and then performing an exponential operation to ensure that all samples have a non-zero sampling probability.

[0037] Next, the historical interaction samples and the calculated sample priorities are combined into sample items and stored in the experience replay pool. The experience replay pool uses a priority queue structure, where leaf nodes are used to store sample items, while non-leaf nodes store the sum of the priorities of the subtrees. This structure can effectively manage the priority of samples, so that samples with higher importance can be selected first when sampling.

[0038] After the samples are stored, the sampling probability is calculated based on the sample priority. The sampling probability refers to the probability of selecting a sample from the experience replay pool. As the sample priority increases, the sampling probability will also increase monotonically. In this way, stratified sampling can be achieved to ensure that important samples are used more frequently during training.

[0039] Then, the importance weight of each sample in the training sample batch is calculated. The importance weight is used to compensate for the sample distribution deviation caused by priority sampling. Through the adjustable importance sampling index, the weight calculation method can be flexibly adjusted to adapt to different training needs.

[0040] After the training sample batches are input into the online network, the weighted temporal difference loss function is calculated based on the importance weights. The loss function reflects the difference between the model prediction and the actual target. Through the back propagation algorithm, the gradient of the loss function with respect to the online network parameters can be calculated. This process can effectively update the model parameters, thereby improving the learning effect of the model.

[0041] The solution of this application can: Improve sample utilization efficiency: Through priority sampling, important samples are used more frequently, thereby accelerating the model's learning process. Enhance model stability: The introduction of importance weights can effectively compensate for the deviation of sample distribution and reduce fluctuations during training. Optimize training results: The weighted loss function allows the model to pay more attention to important samples when updating parameters, thereby improving the overall prediction accuracy.

[0042] In an optional implementation, an adaptive monitoring threshold is set based on the fused feature vector, and a distributed monitoring network is deployed to collect system operation status data in real time; when the monitoring data deviates from the adaptive monitoring threshold, the monitoring data and the original fused feature vector are analyzed for differences, and a hierarchical scheduling optimization strategy is started according to the degree of difference, and an incremental adjustment method is adopted for local anomalies, and the global anomaly triggering strategy is re-optimized, including: Establishing a dynamic baseline based on the fused feature vector, calculating an adaptive adjustment factor in combination with system operating environment parameters, and multiplying the dynamic baseline by the adaptive adjustment factor to obtain an adaptive monitoring threshold; Deploy a three-layer distributed monitoring network to collect real-time operating status data of the system. The three-layer distributed monitoring network consists of a local monitoring agent, a regional monitoring node, and a global monitoring center. The device-level status data collected by the local monitoring agent is aggregated by the regional monitoring node and transmitted to the global monitoring center; The global monitoring center performs feature extraction and fusion processing on the received real-time operating status data of the system to generate a real-time fusion feature vector, and compares the real-time fusion feature vector with the adaptive monitoring threshold; When the real-time fused feature vector deviates from the adaptive monitoring threshold, calculating a standardized difference value of the real-time fused feature vector relative to the initial fused feature vector, and calculating a comprehensive difference degree based on the standardized difference value; The comprehensive difference degree is graded according to a preset difference degree threshold. When the comprehensive difference degree is less than the difference degree threshold, it is determined as a local abnormality and triggers an incremental adjustment strategy. When the comprehensive difference degree is greater than the difference degree threshold or equal to the difference degree threshold, it is determined as a global abnormality and triggers a re-optimization strategy.

[0043] First, the foundation of the fusion feature vector is established. A comprehensive feature vector is formed by extracting multi-dimensional features from the system operation status data. The feature vector contains information such as the equipment's operating parameters, environmental factors, and historical data, ensuring its comprehensiveness and accuracy.

[0044] Next, set the adaptive monitoring threshold. Based on the fused feature vector and the system's operating environment parameters, a dynamic baseline is calculated. The dynamic baseline is set based on the normal operating status of the device and can reflect the performance changes of the device in different environments. The adaptive adjustment factor is adjusted according to the fluctuation of the real-time monitoring data to ensure that the monitoring threshold can reflect the actual operating status of the device in real time.

[0045] Then, a three-layer distributed monitoring network is deployed. The network consists of local monitoring agents, regional monitoring nodes, and a global monitoring center. The local monitoring agent is responsible for collecting device-level status data and transmitting the data to the regional monitoring node. The regional monitoring node aggregates and preliminarily analyzes the data from multiple local monitoring agents, and finally transmits the processed data to the global monitoring center.

[0046] At the global monitoring center, the received real-time system operation status data will be subjected to feature extraction and fusion processing to generate a real-time fusion feature vector. This vector will be compared with the adaptive monitoring threshold to determine whether the operation status of the device is normal.

[0047] When the real-time fused feature vector deviates from the adaptive monitoring threshold, the standardized difference value between it and the initial fused feature vector needs to be calculated. This difference value reflects the degree of deviation between the current state and the normal state. Based on this difference value, the comprehensive difference degree is further calculated to evaluate the abnormality of the device.

[0048] The comprehensive difference degree is graded according to the preset difference degree threshold. When the comprehensive difference degree is less than the difference degree threshold, it is judged as a local abnormality, triggering the incremental adjustment strategy, and making a small range of parameter adjustments to restore the normal state of the equipment; when the comprehensive difference degree is greater than or equal to the difference degree threshold, it is judged as a global abnormality, triggering the re-optimization strategy, and performing comprehensive system optimization and adjustment.

[0049] The solution of this application can: The real-time monitoring capability of the system is improved, and abnormal conditions of the equipment can be discovered and handled in time, reducing the failure rate. The flexibility and adaptability of the monitoring system are ensured by setting adaptive monitoring thresholds, and it can adapt to the operating status in different environments. The implementation of the hierarchical scheduling optimization strategy makes the handling of local and global abnormalities more efficient, improving the overall operating efficiency of the system.

[0050] In an optional implementation, the optimized scheduling effect data is used as a new training sample, and distributed incremental training is performed on the dual-channel neural network and the dual-objective Q network through a federated learning framework to achieve continuous optimization of the scheduling strategy model, including: Collecting scheduling effect evaluation data as training samples, the scheduling effect evaluation data includes system state vectors before and after scheduling execution, actually executed scheduling action sequences, resource utilization data and energy consumption data, and performing normalization preprocessing on the scheduling effect evaluation data to obtain standardized training samples; Distributing the standardized training samples to local training nodes through a federated learning framework, and the local training nodes construct local model copies including a dual-channel neural network and a dual-objective Q network; At the local training node, inputting the state feature vector in the standardized training sample into the state representation network of the dual-channel neural network to obtain a state code, and generating a new scheduling action based on the state code by the action generation network of the dual-channel neural network; Comparing the new scheduled action with the actual executed action in the standardized training sample, calculating the policy gradient and updating the parameters of the action generation network; Inputting the state code and the new scheduling action into the dual-objective Q network, calculating a resource utilization Q value and an energy efficiency Q value, weighting the resource utilization Q value and the energy efficiency Q value according to the performance indicator value in the standardized training sample to obtain a comprehensive Q value, and updating the parameters of the dual-objective Q network based on the comprehensive Q value; Periodically triggering a federated averaging algorithm to collect model parameters of the local training nodes for global aggregation to obtain global model parameters, and distributing the global model parameters to the local training nodes to update the local model copies; New scheduling effect evaluation data is collected as incremental training samples, and the dual-channel neural network and the dual-objective Q network are continuously optimized through the federated learning framework.

[0051] First, the scheduling effect evaluation data is collected as training samples. The data includes the system state vector before and after the scheduling execution, the actual scheduling action sequence, resource utilization data and energy consumption data. By normalizing and preprocessing these data, standardized training samples are obtained. The purpose of normalization is to eliminate the influence of different dimensions and ranges on model training, so that the data can be compared and learned under the same standard.

[0052] Next, the standardized training samples are distributed to local training nodes through the federated learning framework. Each local training node builds a local model copy containing a dual-channel neural network and a dual-objective Q network. This step ensures that each node can independently perform model training while protecting data privacy.

[0053] In the local training node, the state feature vector in the standardized training sample is input into the state representation network of the dual-channel neural network to obtain the state code. Based on the state code, the action generation network of the dual-channel neural network generates a new scheduling action. This process optimizes the current scheduling decision by learning the historical scheduling effect.

[0054] Subsequently, the generated new scheduling actions are compared with the actual execution actions in the standardized training samples, the policy gradient is calculated and the parameters of the action generation network are updated. In this way, the model can continuously adjust and optimize its decision-making strategy to improve the scheduling effect.

[0055] Next, the state code and the new scheduling action are input into the dual-objective Q network to calculate the resource utilization Q value and the energy efficiency Q value. According to the performance indicator values ​​in the standardized training samples, the resource utilization Q value and the energy efficiency Q value are weighted to obtain the comprehensive Q value. Based on the comprehensive Q value, the parameters of the dual-objective Q network are updated to ensure that the model achieves the best balance between resource utilization and energy efficiency.

[0056] The federated averaging algorithm is triggered regularly to collect the model parameters of local training nodes for global aggregation to obtain global model parameters. The global model parameters are distributed to local training nodes and local model copies are updated. This process ensures that the models of each node can share learning results and improve the performance of the overall model.

[0057] Finally, new scheduling effect evaluation data is collected as incremental training samples, and the dual-channel neural network and dual-objective Q network are continuously optimized through the federated learning framework. Through continuous iteration and updating, the model can adapt to new scheduling environments and requirements and maintain efficient scheduling strategies.

[0058] The solution of this application can: Improve the accuracy and efficiency of scheduling strategies, optimize resource utilization and energy efficiency, and reduce energy consumption. Protect data privacy through federated learning and avoid security risks caused by centralized data storage. Achieve continuous optimization of the model, adapt to the dynamically changing scheduling environment, and improve the flexibility and responsiveness of the system. In an optional implementation, the state code and the new scheduling action are input into the dual-objective Q network, a resource utilization Q value and an energy efficiency Q value are calculated, the resource utilization Q value and the energy efficiency Q value are weighted according to the performance indicator value in the standardized training sample to obtain a comprehensive Q value, and updating the dual-objective Q network parameters based on the comprehensive Q value includes: The state code and the new scheduling action are concatenated to obtain a state action vector, and the state action vector is used as a common input of a resource utilization evaluation network and an energy efficiency evaluation network; Inputting the state action vector into the resource utilization evaluation network, performing nonlinear transformation through the first hidden layer and the second hidden layer of the resource utilization evaluation network to obtain resource utilization characteristics, and calculating the resource utilization Q value based on the resource utilization characteristics; d. The resource utilization Q value calculation formula is as follows: ; Among them, Q r (s t ,a t ) is the resource utilization Q value, s t is the state code at time t, a t is the action at time t, W3 is the weight matrix of the output layer, h2 is the resource utilization feature, and b3 is the bias scalar of the output layer; Inputting the state action vector into the energy efficiency evaluation network, performing nonlinear transformation through the first hidden layer and the second hidden layer of the energy efficiency evaluation network to obtain energy efficiency characteristics, and calculating the energy efficiency Q value based on the energy efficiency characteristics; The energy efficiency Q value calculation formula is as follows: ; Among them, Q e (s t ,a t ) is the energy efficiency Q value, W3 e is the weight matrix of the output layer of the energy efficiency evaluation network, h2 e is the energy efficiency characteristic, b3 e The bias scalar of the output layer of the network for energy efficiency evaluation; Calculate the relative change of resource utilization based on the performance indicators in the current batch of training samples, input the relative change of resource utilization into the adaptive weight calculation unit, and calculate the resource utilization weight and energy efficiency weight by the adaptive weight calculation unit; Multiplying the resource utilization rate Q value by the resource utilization rate weight to obtain a weighted resource utilization rate Q value, multiplying the energy efficiency Q value by the energy efficiency weight to obtain a weighted energy efficiency Q value, and adding the weighted resource utilization rate Q value and the weighted energy efficiency Q value to obtain a comprehensive Q value; A temporal difference loss function is constructed based on the comprehensive Q value, and network parameters of the resource utilization evaluation network and the energy efficiency evaluation network are simultaneously updated by minimizing the temporal difference loss function.

[0059] First, the state encoding and the new scheduling action are processed. The state encoding vector and the scheduling action vector are concatenated to generate a state action vector. For example, the state encoding dimension is 128, the scheduling action dimension is 64, and the concatenation results in a state action vector with a dimension of 192. This vector is used as a common input feature of the resource utilization evaluation network and the energy efficiency evaluation network.

[0060] For the resource utilization evaluation network, a three-layer neural network structure is used. The first hidden layer contains 256 neurons, and the ReLU activation function is used for nonlinear transformation; the second hidden layer contains 128 neurons, and the ReLU activation function is also used to obtain the resource utilization feature vector. The feature vector is input into the output layer with 1 neuron to obtain the resource utilization Q value evaluation result. In practical applications, it is assumed that the resource utilization Q value of a certain state-action combination is 0.85.

[0061] The energy efficiency evaluation network uses the same network architecture. The energy efficiency feature vector is obtained through nonlinear transformation of the two hidden layers, and the energy efficiency Q value is obtained by the input and output layers. For example, the energy efficiency Q value of the same state-action combination is 0.72. The two evaluation networks share input layer parameters, but have independent hidden layer and output layer parameters.

[0062] Calculate the relative change in resource utilization based on the current batch of training samples. Assuming that the average resource utilization of the current batch of samples is 75% and that of the previous batch is 70%, the relative change is 7.14%. Input the change into the adaptive weight calculation unit, and calculate the weight coefficients of the two networks according to the preset weight adjustment rules. When the resource utilization rate is significantly improved, increase the weight of the resource utilization evaluation. For example, the resource utilization weight is calculated to be 0.6 and the energy efficiency weight is 0.4.

[0063] The resource utilization Q value is multiplied by the corresponding weight to obtain a weighted result of 0.51, and the energy efficiency Q value is multiplied by the weight to obtain a weighted result of 0.29. The two weighted Q values ​​are added together to obtain a comprehensive Q value of 0.80. This comprehensive Q value reflects both resource utilization efficiency and energy consumption level.

[0064] Finally, a temporal difference loss function is constructed, and the difference between the current comprehensive Q value and the target Q value is used as the optimization target. The gradient is calculated by the back propagation algorithm, and the parameters of the two evaluation networks are updated at the same time. The Adam optimizer is used for parameter optimization, and the learning rate is set to 0.001.

[0065] The solution of this application can: In terms of optimization effect: the dual-objective Q network structure realizes the joint optimization of resource utilization and energy efficiency; the adaptive weight mechanism ensures the dynamic balance of the two optimization objectives; the shared input layer design reduces the redundant calculation of feature extraction. In terms of algorithm performance: the use of deep neural networks improves the ability to extract state action features; multi-level nonlinear transformation enhances the expression ability of the model; independent evaluation networks ensure the professionalism of different optimization objectives. In terms of engineering practicality: the network structure is reasonably designed and easy to deploy; the weight adaptive mechanism can be flexibly adjusted according to actual needs; the selection of evaluation indicators meets the needs of actual application scenarios.

[0066] Figure 2 Schematic diagram of the structure of the heterogeneous metadata intelligent scheduling optimization system based on the reinforcement learning algorithm according to the embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to construct a multi-level dynamic feature extraction mechanism, obtain real-time metadata information in a multi-source heterogeneous metadata system through a distributed data acquisition engine, and perform feature decomposition on the real-time metadata information using a self-attention mechanism to obtain a feature matrix set of metadata, wherein the feature matrix set includes a type feature matrix, a timing feature matrix, a relationship feature matrix, and a load feature matrix; the feature matrix set is input into a transformer encoder for feature fusion to generate a fused feature vector, and a feature vector space is constructed through an adaptive dimensionality reduction algorithm; a scheduling strategy model of a dual-channel neural network structure is trained based on the feature vector space, wherein a graph convolutional neural network channel extracts spatial features based on the type feature matrix and the relationship feature matrix, and a gated recurrent unit network channel extracts timing features based on the timing feature matrix and the load feature matrix, and outputs initial scheduling strategy parameters; The second unit is used to establish a hierarchical optimization mechanism based on deep reinforcement learning according to the initial scheduling strategy parameters, use the fused feature vector as the state input, and generate a candidate set of scheduling actions; design a multi-objective evaluation function to evaluate the candidate actions, calculate the system load balance score, access delay score, resource utilization score and data locality score respectively through an adaptive weight system, and perform a weighted combination of the system load balance, the access delay score, the resource utilization score and the data locality score to obtain a multi-dimensional system performance comprehensive score, and construct a comprehensive reward function based on the multi-dimensional system performance comprehensive score; adopt a dual-objective Q network structure to perform strategy optimization, wherein the online network updates the strategy parameters in real time based on the comprehensive reward function, the target network maintains training stability through a soft update mechanism, and outputs the optimized strategy parameters as a scheduling decision sequence; The third unit is used to input the scheduling decision sequence into a distributed scheduling execution engine, and perform metadata scheduling operations through a two-phase submission protocol; set an adaptive monitoring threshold based on the fused feature vector, and deploy a distributed monitoring network to collect system operation status data in real time; when the monitoring data deviates from the adaptive monitoring threshold, perform a difference analysis between the monitoring data and the original fused feature vector, start a hierarchical scheduling optimization strategy according to the degree of difference, adopt an incremental adjustment method for local anomalies, and re-optimize the global anomaly triggering strategy; use the optimized scheduling effect data as a new training sample, and perform distributed incremental training on the dual-channel neural network and the dual-objective Q network through a federated learning framework to achieve continuous optimization of the scheduling strategy model.

[0067] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0068] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0069] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A heterogeneous metadata intelligent scheduling optimization method based on reinforcement learning algorithm, characterized in that: include: Construct a multi-level dynamic feature extraction mechanism, obtain real-time metadata information in a multi-source heterogeneous metadata system through a distributed data acquisition engine, perform feature decomposition on the real-time metadata information using a self-attention mechanism, and obtain a feature matrix set of metadata, wherein the feature matrix set includes a type feature matrix, a time series feature matrix, a relationship feature matrix, and a load feature matrix; input the feature matrix set into a transformer encoder for feature fusion, generate a fused feature vector, and construct a feature vector space through an adaptive dimensionality reduction algorithm; A scheduling strategy model of a dual-channel neural network structure is trained based on the feature vector space, wherein a graph convolutional neural network channel extracts spatial features based on the type feature matrix and the relationship feature matrix, and a gated recurrent unit network channel extracts timing features based on the timing feature matrix and the load feature matrix, and outputs initial scheduling strategy parameters; According to the initial scheduling strategy parameters, a hierarchical optimization mechanism based on deep reinforcement learning is established, and the fused feature vector is used as a state input to generate a candidate set of scheduling actions; a multi-objective evaluation function is designed to evaluate the candidate actions, and the system load balance score, access delay score, resource utilization score and data locality score are calculated respectively through an adaptive weight system, and the system load balance, the access delay score, the resource utilization score and the data locality score are weightedly combined to obtain a multi-dimensional system performance comprehensive score, and a comprehensive reward function is constructed based on the multi-dimensional system performance comprehensive score; A dual-objective Q network structure is used for policy optimization, wherein the online network updates the policy parameters in real time based on the comprehensive reward function, the target network maintains training stability through a soft update mechanism, and outputs the optimized policy parameters as a scheduling decision sequence; Inputting the scheduling decision sequence into a distributed scheduling execution engine, and executing metadata scheduling operations through a two-phase commit protocol; Setting an adaptive monitoring threshold based on the fused feature vector and deploying a distributed monitoring network to collect system operation status data in real time; When the monitoring data deviates from the adaptive monitoring threshold, a difference analysis is performed on the monitoring data and the original fused feature vector, and a hierarchical scheduling optimization strategy is started according to the degree of difference. An incremental adjustment method is used for local anomalies, and the global anomaly triggering strategy is re-optimized. The optimized scheduling effect data is used as a new training sample, and distributed incremental training is performed on the dual-channel neural network and the dual-objective Q network through a federated learning framework to achieve continuous optimization of the scheduling strategy model.

2. The method according to claim 1, characterized in that According to the initial scheduling strategy parameters, a hierarchical optimization mechanism based on deep reinforcement learning is established, and the fused feature vector is used as the state input to generate a candidate set of scheduling actions; a multi-objective evaluation function is designed to evaluate the candidate actions, and the system load balance score, access delay score, resource utilization score and data locality score are calculated respectively through an adaptive weight system, including: A three-level linkage optimization architecture is constructed based on the initial scheduling strategy parameters, and the three-level linkage optimization architecture includes a strategy layer and an execution layer; the strategy layer has a built-in deep deterministic policy gradient network with a dual exploration mechanism, and the execution layer has a built-in self-correcting four-dimensional performance evaluation model; the deep deterministic policy gradient network adopts a four-layer fully connected structure, the number of nodes in the input layer is the same as the dimension of the fusion feature vector, and the number of nodes in the output layer is the same as the dimension of the action space; The fused feature vector is input into the deep deterministic policy gradient network, and the parameter distribution of the deep deterministic policy gradient network is modeled by hierarchical Bayesian modeling to obtain a hierarchical network weight distribution. A Thompson sampling method is used to obtain multiple groups of network parameter samples from each layer of the hierarchical network weight distribution to form a parameter sample set, and Gaussian noise perturbation is introduced into the parameter sample set; the parameter sample sets after adding the Gaussian noise perturbation are respectively loaded into the deep deterministic policy gradient network to generate multiple policy network instances, and the multiple policy network instances independently generate scheduling action candidate sets based on the initial scheduling policy parameters; The scheduling action candidate set is input into the four-dimensional performance evaluation model of the execution layer. The four-dimensional performance evaluation model sets a self-correction threshold for the evaluation result of each dimension, and triggers a re-evaluation mechanism when the evaluation result exceeds the self-correction threshold. The load balancing score, access delay score, resource utilization score and data locality score are calculated respectively through the four-dimensional performance evaluation model.

3. The method according to claim 1, characterized in that: A comprehensive reward function is constructed based on the comprehensive score of the multi-dimensional system performance; a dual-objective Q network structure is used for policy optimization, wherein the online network updates the policy parameters in real time based on the comprehensive reward function, and the target network maintains the training stability through a soft update mechanism, including: A comprehensive reward function is constructed based on the multi-dimensional system performance comprehensive score, and the difference between the multi-dimensional system performance comprehensive score at the current moment and the previous moment is calculated to obtain a performance change value; the multi-dimensional system performance comprehensive score is set as a basic reward item, and the performance change value is set as a priority reward item; based on the accumulated reward and average reward during the training process, the combined weights of the basic reward item and the priority reward item are adaptively adjusted to obtain the comprehensive reward function; A dual-objective Q network structure is used for strategy optimization to construct an online network and a target network with a shared network structure; the current system state is input into the online network to obtain the current action value, and the next moment system state is input into the target network to obtain the target state value; the current action value, the target state value and the comprehensive reward function are substituted into the Bellman equation to calculate the temporal difference error; Calculating sample priority based on the absolute value of the temporal difference error; storing historical interaction samples and corresponding sample priorities in an experience replay pool; performing importance sampling according to the sample priorities to obtain training sample batches; inputting the training sample batches into the online network, and updating the strategy parameters of the online network in real time based on the temporal difference error through a back propagation algorithm; The soft update mechanism is adopted to update the policy parameters of the target network, and the updated policy parameters of the online network and the policy parameters of the target network are weighted averaged to obtain the updated parameters of the target network; wherein the weight of the weighted average is determined by a preset soft update coefficient, and the stability of the target network parameter update is maintained by a smaller soft update coefficient.

4. The method according to claim 3, characterized in that Calculating the sample priority based on the absolute value of the temporal difference error; storing the historical interaction samples and the corresponding sample priorities in the experience replay pool; Performing importance sampling according to the sample priorities to obtain a training sample batch; Inputting the training sample batches into the online network, and updating the strategy parameters of the online network in real time based on the temporal difference error by a back propagation algorithm includes: The absolute value of the timing difference error is added to the priority reference value and then an exponential operation is performed to obtain the sample priority, wherein the priority reference value is used to ensure that all samples have a non-zero sampling probability; The historical interaction samples and the sample priorities form a sample item and store it in an experience replay pool of a priority queue structure, wherein the priority queue structure includes leaf nodes and non-leaf nodes, the leaf nodes store the sample items, and the non-leaf nodes store the priority sum of the subtrees; Calculate the sampling probability based on the sample priority, the sampling probability monotonically increases with the sample priority, and perform stratified sampling according to the sampling probability to obtain a training sample batch; Calculating the importance weight of each sample in the training sample batch, wherein the importance weight is used to compensate for the sample distribution deviation caused by priority sampling, and the importance weight is calculated by an adjustable importance sampling index; Inputting the training sample batches into the online network, calculating a weighted temporal difference loss function based on the importance weights, and calculating the gradient of the loss function with respect to the online network parameters by a back propagation algorithm; The gradient calculation formula of the network parameters is as follows: ； Among them, ∇ θ L is the gradient of the loss function with respect to the parameter θ, N is the size of the experience replay pool, and w i is the importance weight of the i-th sample, y i is the target Q value of the i-th sample, Q(s i ,a i ;θ) is the state-action pair of the online network (s i ,a i )’s predicted Q value, ∇ θ Q is the gradient of the Q value with respect to the parameter θ.

5. The method according to claim 1, characterized in that: Based on the fused feature vector, an adaptive monitoring threshold is set, and a distributed monitoring network is deployed to collect system operation status data in real time; when the monitoring data deviates from the adaptive monitoring threshold, the monitoring data and the original fused feature vector are analyzed for differences, and a hierarchical scheduling optimization strategy is started according to the degree of difference, and an incremental adjustment method is adopted for local anomalies. The global anomaly triggering strategy is re-optimized, including: Establishing a dynamic baseline based on the fused feature vector, calculating an adaptive adjustment factor in combination with system operating environment parameters, and multiplying the dynamic baseline by the adaptive adjustment factor to obtain an adaptive monitoring threshold; Deploy a three-layer distributed monitoring network to collect real-time operating status data of the system. The three-layer distributed monitoring network consists of a local monitoring agent, a regional monitoring node, and a global monitoring center. The device-level status data collected by the local monitoring agent is aggregated by the regional monitoring node and transmitted to the global monitoring center; The global monitoring center performs feature extraction and fusion processing on the received real-time operating status data of the system to generate a real-time fusion feature vector, and compares the real-time fusion feature vector with the adaptive monitoring threshold; When the real-time fused feature vector deviates from the adaptive monitoring threshold, calculating a standardized difference value of the real-time fused feature vector relative to the initial fused feature vector, and calculating a comprehensive difference degree based on the standardized difference value; The comprehensive difference degree is graded according to a preset difference degree threshold. When the comprehensive difference degree is less than the difference degree threshold, it is determined as a local abnormality and triggers an incremental adjustment strategy. When the comprehensive difference degree is greater than the difference degree threshold or equal to the difference degree threshold, it is determined as a global abnormality and triggers a re-optimization strategy.

6. The method according to claim 1, characterized in that The optimized scheduling effect data is used as a new training sample, and the dual-channel neural network and the dual-objective Q network are subjected to distributed incremental training through a federated learning framework to achieve continuous optimization of the scheduling strategy model, including: Collecting scheduling effect evaluation data as training samples, the scheduling effect evaluation data includes system state vectors before and after scheduling execution, actually executed scheduling action sequences, resource utilization data and energy consumption data, and performing normalization preprocessing on the scheduling effect evaluation data to obtain standardized training samples; Distributing the standardized training samples to local training nodes through a federated learning framework, and the local training nodes construct local model copies including a dual-channel neural network and a dual-objective Q network; At the local training node, inputting the state feature vector in the standardized training sample into the state representation network of the dual-channel neural network to obtain a state code, and generating a new scheduling action based on the state code by the action generation network of the dual-channel neural network; Comparing the new scheduled action with the actual executed action in the standardized training sample, calculating the policy gradient and updating the parameters of the action generation network; Inputting the state code and the new scheduling action into the dual-objective Q network, calculating a resource utilization Q value and an energy efficiency Q value, weighting the resource utilization Q value and the energy efficiency Q value according to the performance indicator value in the standardized training sample to obtain a comprehensive Q value, and updating the parameters of the dual-objective Q network based on the comprehensive Q value; Periodically triggering a federated averaging algorithm to collect model parameters of the local training nodes for global aggregation to obtain global model parameters, and distributing the global model parameters to the local training nodes to update the local model copies; New scheduling effect evaluation data is collected as incremental training samples, and the dual-channel neural network and the dual-objective Q network are continuously optimized through the federated learning framework.

7. The method according to claim 6, characterized in that Inputting the state code and the new scheduling action into the dual-objective Q network, calculating the resource utilization Q value and the energy efficiency Q value, weighting the resource utilization Q value and the energy efficiency Q value according to the performance indicator value in the standardized training sample to obtain a comprehensive Q value, and updating the dual-objective Q network parameters based on the comprehensive Q value includes: The state code and the new scheduling action are concatenated to obtain a state action vector, and the state action vector is used as a common input of a resource utilization evaluation network and an energy efficiency evaluation network; Inputting the state action vector into the resource utilization evaluation network, performing nonlinear transformation through the first hidden layer and the second hidden layer of the resource utilization evaluation network to obtain resource utilization characteristics, and calculating the resource utilization Q value based on the resource utilization characteristics; The resource utilization Q value calculation formula is as follows: ； Among them, Q r (s t ,a t ) is the resource utilization Q value, s t is the state code at time t, a t is the action at time t, W3 is the weight matrix of the output layer, h2 is the resource utilization feature, and b3 is the bias scalar of the output layer; Inputting the state action vector into the energy efficiency evaluation network, performing nonlinear transformation through the first hidden layer and the second hidden layer of the energy efficiency evaluation network to obtain energy efficiency characteristics, and calculating the energy efficiency Q value based on the energy efficiency characteristics; The energy efficiency Q value calculation formula is as follows: ； Among them, Q e (s t ,a t ) is the energy efficiency Q value, W3 e is the weight matrix of the output layer of the energy efficiency evaluation network, h2 e is the energy efficiency characteristic, b3 e The bias scalar of the output layer of the network for energy efficiency evaluation; Calculate the relative change of resource utilization based on the performance indicators in the current batch of training samples, input the relative change of resource utilization into the adaptive weight calculation unit, and calculate the resource utilization weight and energy efficiency weight by the adaptive weight calculation unit; Multiplying the resource utilization rate Q value by the resource utilization rate weight to obtain a weighted resource utilization rate Q value, multiplying the energy efficiency Q value by the energy efficiency weight to obtain a weighted energy efficiency Q value, and adding the weighted resource utilization rate Q value and the weighted energy efficiency Q value to obtain a comprehensive Q value; A temporal difference loss function is constructed based on the comprehensive Q value, and network parameters of the resource utilization evaluation network and the energy efficiency evaluation network are simultaneously updated by minimizing the temporal difference loss function.

8. A heterogeneous metadata intelligent scheduling optimization system based on a reinforcement learning algorithm, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to construct a multi-level dynamic feature extraction mechanism, obtain real-time metadata information in a multi-source heterogeneous metadata system through a distributed data acquisition engine, perform feature decomposition on the real-time metadata information using a self-attention mechanism, and obtain a feature matrix set of metadata, wherein the feature matrix set includes a type feature matrix, a time series feature matrix, a relationship feature matrix, and a load feature matrix; the feature matrix set is input into a transformer encoder for feature fusion to generate a fused feature vector, and a feature vector space is constructed through an adaptive dimensionality reduction algorithm; A scheduling strategy model of a dual-channel neural network structure is trained based on the feature vector space, wherein a graph convolutional neural network channel extracts spatial features based on the type feature matrix and the relationship feature matrix, and a gated recurrent unit network channel extracts timing features based on the timing feature matrix and the load feature matrix, and outputs initial scheduling strategy parameters; The second unit is used to establish a hierarchical optimization mechanism based on deep reinforcement learning according to the initial scheduling strategy parameters, use the fused feature vector as a state input, and generate a candidate set of scheduling actions; design a multi-objective evaluation function to evaluate the candidate actions, calculate the system load balance score, access delay score, resource utilization score and data locality score respectively through an adaptive weight system, perform a weighted combination of the system load balance, the access delay score, the resource utilization score and the data locality score to obtain a multi-dimensional system performance comprehensive score, and construct a comprehensive reward function based on the multi-dimensional system performance comprehensive score; A dual-objective Q network structure is used for policy optimization, wherein the online network updates the policy parameters in real time based on the comprehensive reward function, the target network maintains training stability through a soft update mechanism, and outputs the optimized policy parameters as a scheduling decision sequence; A third unit is used to input the scheduling decision sequence into a distributed scheduling execution engine and perform metadata scheduling operations through a two-phase commit protocol; Setting an adaptive monitoring threshold based on the fused feature vector and deploying a distributed monitoring network to collect system operation status data in real time; When the monitoring data deviates from the adaptive monitoring threshold, a difference analysis is performed on the monitoring data and the original fused feature vector, and a hierarchical scheduling optimization strategy is started according to the degree of difference. An incremental adjustment method is used for local anomalies, and the global anomaly triggering strategy is re-optimized. The optimized scheduling effect data is used as a new training sample, and distributed incremental training is performed on the dual-channel neural network and the dual-objective Q network through a federated learning framework to achieve continuous optimization of the scheduling strategy model.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images