Multi-heterogeneous equipment configuration method and device based on multi-target graph reinforcement learning

Abstract

The invention discloses a multi-heterogeneous equipment configuration method and device based on multi-target graph reinforcement learning, and the method comprises the steps: converting an equipment configuration problem into a multi-target combination optimization problem model through modeling, and setting a plurality of evaluation indexes; constructing an enhanced heterogeneous graph attention network by stacking a plurality of heterogeneous graph attention layers; taking the full-connection deep neural network as a decoder, and generating an equipment configuration scheme through a multi-step non-autoregression strategy; substituting the preference vector into a decoding process, and training a single neural network model based on a preference condition to realize preference multi-objective optimization; training the multi-objective combinatorial optimization problem model by setting a reinforcement learning algorithm; and inputting the equipment parameter information into the trained multi-target combination optimization problem model, and outputting a multi-heterogeneous equipment configuration optimization scheme. According to the method, the configuration problems with different scales and distribution can be effectively processed, the operation efficiency is improved, and the resource consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment configuration, and in particular to a multi-heterogeneous equipment configuration method and device based on multi-objective graph reinforcement learning. Background Art

[0002] Equipment configuration refers to the selection of appropriate equipment combinations to meet established needs and goals in specific tasks or scenarios. This issue involves multiple fields, including industry, logistics, emergency management, etc. Its core is to optimize resource allocation and ensure the effectiveness and efficiency of equipment. Equipment configuration emphasizes the high degree of synergy between multi-level heterogeneous equipment to achieve greater flexibility and effectiveness. How to achieve equipment configuration optimization with high quality and efficiency is a hot topic in various engineering fields. The realization of equipment configuration is by no means easy. From a simple "chain" structure to an interconnected "net" structure, it illustrates the transition from a single, isolated action to a more comprehensive and interconnected resource configuration method. Equipment elements in multiple fields often interact in complex and unpredictable ways. Therefore, the challenge is not only how to master these complex networks of configuration elements, but also how to effectively coordinate cross-domain collaboration of multiple equipment to achieve the best resource allocation effect.

[0003] At present, most researchers abstract the equipment configuration optimization problem as a combinatorial optimization problem, the essence of which is to make decisions on the best combination of multiple equipment under certain constraints to maximize the effectiveness of the goal. The corresponding research methods can be divided into precise optimization methods, approximate optimization methods, swarm intelligence methods and their hybrid methods. These traditional algorithms are effective in some problem scenarios, but their application in the equipment configuration optimization problem we studied still has many shortcomings. Precise optimization algorithms are usually highly sensitive to problem scale and modeling, and are suitable for small-scale, linear problems; while approximate optimization methods and swarm intelligence methods have faster running speeds, but their operator design is complex and the quality of the solution is difficult to guarantee. Considering the characteristics of equipment configuration optimization problems, such as large-scale, unstructured, heterogeneous, complex and dynamic, the efficiency of traditional methods is difficult to meet the requirements. In particular, due to the high requirements of equipment configuration problems for real-time decision-making, traditional iteration-based optimization methods are no longer applicable. Therefore, how to effectively complete large-scale equipment configuration in a short time has become an urgent challenge today. In recent years, various artificial intelligence methods represented by deep reinforcement learning methods have been widely used in combinatorial optimization problems, such as traveling salesman problems, vehicle routing problems and flow shop scheduling problems. The powerful learning and generalization capabilities of deep reinforcement learning methods make them well suited for optimization of problems with hundreds of nodes or even larger scales. In addition, since deep reinforcement learning methods adopt the strategy of "offline training, online optimization", once the model is pre-trained, its decision speed is usually in seconds, which is crucial for dynamic optimization scenarios and emergency decision-making tasks.

[0004] Therefore, how to invent a multi-heterogeneous equipment configuration method to effectively handle configuration problems with different scales and distributions, improve computing efficiency, and reduce resource consumption has become an urgent problem to be solved. Summary of the invention

[0005] To this end, the present invention provides a multi-heterogeneous equipment configuration method and device based on multi-objective graph reinforcement learning, which improves the algorithm's solution effect on large-scale problems and different examples through the powerful generalization ability of deep reinforcement learning; at the same time, it systematically proposes five evaluation indicators in terms of comprehensive cost, benefit, efficiency, reliability and risk, and all indicators are designed as matrix operations, which exponentially improves the algorithm's calculation speed.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning, comprising:

[0007] The equipment configuration problem is converted into a multi-objective combination optimization problem model through modeling, and several evaluation indicators are set;

[0008] By stacking several heterogeneous graph attention layers, an enhanced heterogeneous graph attention network is constructed; and the features of heterogeneous nodes and edges are processed by the enhanced heterogeneous graph attention network;

[0009] A fully connected deep neural network is used as a decoder, and the decoder is used to generate a probability vector; the decoder generates an equipment configuration scheme through a multi-step non-autoregressive strategy;

[0010] Substitute the preference vector into the decoding process, train a single neural network model based on preference conditions, and achieve preference multi-objective optimization;

[0011] The multi-objective combined optimization problem model is trained by setting a reinforcement learning algorithm to obtain a trained multi-objective combined optimization problem model;

[0012] The equipment parameter information is input into the trained multi-objective combined optimization problem model, and processed by the multi-objective combined optimization problem model to output a multi-heterogeneous equipment configuration optimization solution.

[0013] As a preferred solution of the multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning, the expression of the multi-objective combination optimization problem model is:

[0014]

[0015] In the formula, F(X) is a number of optimization objectives; M is the number of optimization objectives; X is the set of decision variables; x i (k)is the i-th node of the k-th type of equipment; K is the number of equipment types; n k is the total number of equipment of type k.

[0016] As a preferred solution for the multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning, the several evaluation indicators set include: equipment utilization rate, effectiveness generation time, solution redundancy, task completion and solution risk.

[0017] As a preferred solution of the multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning, in the process of constructing the enhanced heterogeneous graph attention network by stacking a plurality of the heterogeneous graph attention layers, each of the heterogeneous graph attention layers includes a precoding module, an attention module, a message passing module and an aggregation module;

[0018] The expression of the precoding module is:

[0019]

[0020] In the formula, K i (t) is the Key vector; Q i (t) is the Query vector; V i (t) is the Value vector; i is the attention head; t is the node; τ(t) is the node type; l is the number of attention layers; is the linear mapping of the (l-1)th layer;

[0021] The expression of the attention module is:

[0022]

[0023] Where N(t) is the predecessor node of target node t; a i (s,e,t) is the attention vector of the i-th attention head; h is the number of attention heads; d is the embedding dimension; s is the predecessor node; is a learnable weight matrix;

[0024] The expression of the message passing module is:

[0025] V(s,e,t)=|| i∈[1,h] V i (s,e,t)

[0026]

[0027] Where V(s,e,t) is the value information vector; V i (s,e,t) is the value information vector of the i-th attention head; is a weight matrix related to the edge type, used to parameterize heterogeneous edges;

[0028] The expression of the aggregation module is:

[0029]

[0030] In the formula, H (l) (t) is the final context embedding of node t; is the updated context embedding of node t; σ is the scaling factor; A-Linear is the linear projection layer.

[0031] As a preferred solution of the multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning, in the process of generating the probability vector through the decoder, the expression of the probability vector is:

[0032]

[0033] In the formula, p i is the probability vector; x i is a node; H (L) (x i ) is the node x i Embedding vector; is the weight matrix; is the bias vector.

[0034] As a preferred solution of the multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning, in the process of substituting the preference vector into decoding, the expression of the preference vector is:

[0035]

[0036] Where λ is the preference vector and M is the number of targets.

[0037] As a preferred solution of the multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning, in the process of training the multi-objective combined optimization problem model by the set reinforcement learning algorithm, the parameters of the multi-objective combined optimization problem model are trained by the REINFORCE algorithm; the expression of the loss function is:

[0038]

[0039] Where θ is a learnable model parameter; π~p θ (.|r,λ) is the solution sampled according to the policy; b(r,λ) is the baseline of the expected cost of reducing the gradient variance; r is the instance; λ is the preference vector; f tch (π|r,λ) is a single objective function under preference conditions.

[0040] The present invention also provides a multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning, based on the above multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning, including:

[0041] The target combination optimization problem model conversion unit is used to convert the equipment configuration problem into a multi-target combination optimization problem model through modeling and set several evaluation indicators;

[0042] An enhanced heterogeneous graph attention network construction and processing unit, used to construct an enhanced heterogeneous graph attention network by stacking a plurality of heterogeneous graph attention layers; and to process the features of heterogeneous nodes and edges through the enhanced heterogeneous graph attention network;

[0043] A decoder construction and processing unit, for using a fully connected deep neural network as a decoder, the decoder is used to generate a probability vector; the decoder generates an equipment configuration scheme through a multi-step non-autoregressive strategy;

[0044] The preference multi-objective optimization unit is used to substitute the preference vector into the decoding process, train a single neural network model based on the preference condition, and realize the preference multi-objective optimization;

[0045] A multi-objective combined optimization problem model training unit, used to train the multi-objective combined optimization problem model by setting a reinforcement learning algorithm to obtain the trained multi-objective combined optimization problem model;

[0046] The multi-heterogeneous equipment configuration optimization solution acquisition unit is used to input equipment parameter information into the trained multi-objective combined optimization problem model, process it through the multi-objective combined optimization problem model, and output the multi-heterogeneous equipment configuration optimization solution.

[0047] As a preferred solution of the multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning, in the objective combination optimization problem model conversion unit, the expression of the multi-objective combination optimization problem model is:

[0048]

[0049] In the formula, F(X) is a number of optimization objectives; M is the number of optimization objectives; X is the set of decision variables; is the i-th node of the k-th type of equipment; K is the number of equipment types; n k is the total number of equipment of type k.

[0050] As a preferred solution for a multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning, the several evaluation indicators set in the objective combination optimization problem model conversion unit include: equipment utilization rate, performance generation time, solution redundancy, task completion and solution risk.

[0051] As a preferred solution of the multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning, in the enhanced heterogeneous graph attention network construction and processing unit, in the process of constructing the enhanced heterogeneous graph attention network by stacking a plurality of the heterogeneous graph attention layers, each of the heterogeneous graph attention layers includes a precoding module, an attention module, a message passing module and an aggregation module;

[0052] The expression of the precoding module is:

[0053]

[0054] In the formula, K i (t) is the Key vector; Q i (t) is the Query vector; V i (t) is the Value vector; i is the attention head; t is the node; τ(t) is the node type; l is the number of attention layers; is the linear mapping of the (l-1)th layer;

[0055] The expression of the attention module is:

[0056]

[0057] Where N(t) is the predecessor node of target node t; a i (s,e,t) is the attention vector of the i-th attention head; h is the number of attention heads; d is the embedding dimension; s is the predecessor node; is a learnable weight matrix;

[0058] The expression of the message passing module is:

[0059] V(s,e,t)=|| i∈[1,h] V i (s,e,t)

[0060]

[0061] Where V(s,e,t) is the value information vector; V i (s,e,t) is the value information vector of the i-th attention head; is a weight matrix related to the edge type, used to parameterize heterogeneous edges;

[0062] The expression of the aggregation module is:

[0063]

[0064] In the formula, H (l) (t) is the final context embedding of node t; is the updated context embedding of node t; σ is the scaling factor; A-Linear is the linear projection layer.

[0065] As a preferred solution of the multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning, in the decoder construction and processing unit, in the process of generating the probability vector through the decoder, the expression of the probability vector is:

[0066]

[0067] In the formula, p i is the probability vector; x i is a node; H (L) (x i ) is the node x i Embedding vector; is the weight matrix; is the bias vector.

[0068] As a preferred solution of the multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning, in the preference multi-objective optimization unit, when substituting the preference vector into the decoding process, the expression of the preference vector is:

[0069]

[0070] Where λ is the preference vector and M is the number of targets.

[0071] As a preferred solution of the multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning, in the multi-objective combination optimization problem model training unit, in the process of training the multi-objective combination optimization problem model by the set reinforcement learning algorithm, the parameters of the multi-objective combination optimization problem model are trained by the REINFORCE algorithm; the expression of the loss function is:

[0072]

[0073] Where θ is a learnable model parameter; π~p θ (.|r,λ) is the solution sampled according to the policy; b(r,λ) is the baseline of the expected cost of reducing the gradient variance; r is the instance; λ is the preference vector; f tch (π|r,λ) is a single objective function under preference conditions.

[0074] The present invention has the following advantages: the present invention converts the equipment configuration problem into a multi-objective combination optimization problem model through modeling, and sets several evaluation indicators; by stacking several heterogeneous graph attention layers, an enhanced heterogeneous graph attention network is constructed; the features of heterogeneous nodes and edges are processed by the enhanced heterogeneous graph attention network; a fully connected deep neural network is used as a decoder, and the decoder is used to generate a probability vector; the decoder generates an equipment configuration scheme through a multi-step non-autoregressive strategy; the preference vector is substituted into the decoding process, and a single neural network model based on preference conditions is trained to achieve preference multi-objective optimization; the multi-objective combination optimization problem model is trained by setting a reinforcement learning algorithm to obtain the trained multi-objective combination optimization problem model; the equipment parameter information is input into the trained multi-objective combination optimization problem model, and the multi-objective combination optimization problem model is processed to output a multi-heterogeneous equipment configuration optimization scheme. The present invention proposes a method for realizing equipment configuration using a multi-objective heterogeneous graph reinforcement learning algorithm to solve the equipment configuration optimization problem. The framework can effectively utilize the powerful generalization ability of deep reinforcement learning and improve the algorithm's solution effect on large-scale problems and different examples. It can fully learn the basic characteristics of the problem and effectively handle configuration problems with different scales and distributions within a few seconds. The present invention systematically proposes five evaluation indicators for the configuration problem of multi-heterogeneous equipment, integrating indicators of cost, benefit, efficiency, reliability and risk, and all indicators are designed as matrix operations, which exponentially increases the algorithm operation speed. The present invention designs a single-model deep reinforcement learning optimization framework based on preference conditions, which allows the pre-trained model to approximate the Pareto frontier with any number of preferences, greatly improving the speed of generating solutions and effectively reducing resource consumption compared to traditional multi-model optimization frameworks. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0076] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0077] Figure 1This is a schematic diagram of the process flow of the multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning provided in Example 1 of the present invention;

[0078] Figure 2 This is a schematic diagram of a specific implementation process of the multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning provided in Example 1 of the present invention;

[0079] Figure 3 A schematic diagram of a process for solving a multi-heterogeneous equipment configuration problem in a multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning provided in Example 1 of the present invention;

[0080] Figure 4 A schematic diagram of a heterogeneous graph attention neural network in a multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning provided in Example 1 of the present invention;

[0081] Figure 5 This is a schematic diagram of the architecture of a multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0082] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] Example 1

[0084] See also Figure 1 and Figure 2 Embodiment 1 of the present invention provides a multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning, comprising the following steps:

[0085] S1. Convert the equipment configuration problem into a multi-objective combination optimization problem model through modeling, and set several evaluation indicators;

[0086] S2. By stacking several heterogeneous graph attention layers, an enhanced heterogeneous graph attention network is constructed; and the features of heterogeneous nodes and edges are processed by the enhanced heterogeneous graph attention network;

[0087] S3, using a fully connected deep neural network as a decoder, the decoder is used to generate a probability vector; the decoder generates an equipment configuration plan through a multi-step non-autoregressive strategy;

[0088] S4, substituting the preference vector into the decoding process, training a single neural network model based on the preference condition, and realizing the preference multi-objective optimization;

[0089] S5. Training the multi-objective combined optimization problem model by setting a reinforcement learning algorithm to obtain a trained multi-objective combined optimization problem model;

[0090] S6. Input the equipment parameter information into the trained multi-objective combined optimization problem model, process it through the multi-objective combined optimization problem model, and output a multi-heterogeneous equipment configuration optimization solution.

[0091] In this embodiment, equipment configuration refers to selecting the appropriate equipment combination to meet the given needs and goals in a specific task or scenario. This problem involves multiple fields, including industry, logistics, emergency management, etc. Its core is to optimize resource allocation and ensure the effectiveness and efficiency of equipment. The problem-solving process is as follows: Figure 3 shown.

[0092] In this embodiment, in step S1, the equipment configuration problem is converted into a multi-objective combined optimization problem model through modeling, and several evaluation indicators are set;

[0093] Specifically, the equipment configuration problem is modeled as a multi-objective combination optimization problem, requiring the decision maker to select appropriate nodes and construct an equipment configuration plan to optimize multiple objectives. The expression of the multi-objective combination optimization problem model is:

[0094]

[0095] In the formula, F(X) is a number of optimization objectives; M is the number of optimization objectives; X is the set of decision variables; is the i-th node of the k-th type of equipment; K is the number of equipment types; n k is the total number of equipment of type k.

[0096] In this embodiment, the evaluation indicators set include equipment utilization rate, effectiveness generation time, solution redundancy, task completion and solution risk.

[0097] Equipment utilization rate: Taking into account the equipment cost, the percentage of non-target equipment nodes selected when building the equipment configuration plan is used as one of the evaluation indicators, which is called equipment utilization rate. This optimization goal should be minimized, aiming to use as few equipment resources as possible.

[0098] Effectiveness generation time: mainly describes the time required from generating an equipment configuration plan to completing the execution of the task. It is defined as the average time cost of all effective equipment configuration plans. Each effective equipment configuration plan needs to consider reconnaissance time, decision time, equipment preparation time and execution time. This optimization goal should be minimized to improve the system's response speed.

[0099] Solution redundancy: mainly describes the diversity of available equipment means for the task. It is defined as the average number of effective equipment configuration solutions that can be formed in the configuration solution for each task goal. This optimization goal should be maximized to enhance the reliability of the overall configuration solution.

[0100] Task completion: mainly refers to the degree of completion of a given task. It is defined as the proportion of target tasks that are included in at least one valid equipment configuration solution. This optimization goal should be maximized to improve the task completion.

[0101] Solution risk: mainly describes the possible impact of equipment failure in equipment configuration. It is defined as the maximum node degree of all equipment nodes in the configuration solution. This optimization goal should be minimized to improve the flexibility and reliability of the equipment configuration solution.

[0102] In this embodiment, in step S2, an enhanced heterogeneous graph attention network is constructed by stacking several heterogeneous graph attention layers; the features of heterogeneous nodes and edges are processed by the enhanced heterogeneous graph attention network;

[0103] Specifically, an enhanced heterogeneous graph attention neural network is constructed as an encoder to process the features of heterogeneous nodes and edges; the encoder structure stacks L layers of heterogeneous graph attention layers to achieve highly contextualized representation. Figure 4 As shown, each graph attention layer consists of a pre-encoding module, an attention module, a message passing module, and an aggregation module.

[0104] Among them, the precoding module:

[0105] For the node t with attention head i in the l-th layer of heterogeneous graph attention layer (HGT), the key vector K is first generated i (t), Query vector Q i (t) and the Value vector V i (t). These linear projections map vectors from d dimensions to dimensional, where d is the embedding dimension and h is the number of attention heads. Each of these linear projection layers is indexed by node type τ(t) and i.

[0106]

[0107] In the formula, K i (t) is the Key vector; Q i (t) is the Query vector; V i (t) is the Value vector; i is the attention head; t is the node; τ(t) is the node type; l is the number of attention layers; is the linear mapping of the (l-1)th layer.

[0108] Attention Module:

[0109] The function of the attention module is to assign different levels of attention to different parts of the input, thereby identifying potential relationships that are critical to the analysis. For any target node t and its predecessor node s, the attention a i (s,e,t) is defined as the Softmax normalized form of the concatenation of all h attention vectors:

[0110]

[0111] Where N(t) is the predecessor node of target node t; a i (s,e,t) is the attention vector of the i-th attention head; h is the number of attention heads; d is the embedding dimension; s is the predecessor node; is the learnable weight matrix.

[0112] Messaging Module:

[0113] Similar to the construction of the attention vector, the value information vector V(s,e,t) is also constructed by concatenating the value information vectors under h attention heads:

[0114] V(s,e,t)=|| i∈[1,h] V i (s,e,t)

[0115]

[0116] Where V(s,e,t) is the value information vector; V i (s,e,t) is the value information vector of the i-th attention head; It is a weight matrix related to the edge type and is used to parameterize heterogeneous edges.

[0117] Aggregation module:

[0118] To generate the contextual embedding of a target node, we need to aggregate the information of all its source nodes. Note that the vector a(s,e,t)|s∈N(t) is used as a weight to average the value vector V(s,e,t)|s∈N(t). The updated contextual embedding of node t is calculated as follows:

[0119]

[0120] The linear projection layer will To do further mapping:

[0121]

[0122] In the formula, H (l) (t) is the final context embedding of node t; is the updated context embedding of node t; σ is the scaling factor; A-Linear is the linear projection layer.

[0123] By stacking L HGT layers as described above, we can obtain the final context embedding H of node t (l) (t).

[0124] In this embodiment, in step S3, a fully connected deep neural network is used as a decoder, and the decoder is used to generate a probability vector; the decoder generates an equipment configuration scheme through a multi-step non-autoregressive strategy;

[0125] Specifically, a fully connected deep neural network is used as a decoder, and a multi-step non-autoregressive method is used to generate equipment configuration solutions.

[0126] The decoder designs a multi-step non-autoregressive method to generate the probability vector. First, a fully connected layer is used to embed the encoded node into the vector H (L) (x i ) is linearly mapped to a low-dimensional space, and then the Softmax normalization method is used to obtain the node x i The selection probability vector p i :

[0127]

[0128] In the formula, p i is the probability vector; x i is a node; H (L) (x i ) is the node x i Embedding vector; is the weight matrix; is the bias vector.

[0129] Due to the inherent connectivity between heterogeneous nodes, selecting all nodes at once can easily lead to the generation of infeasible solutions, which results in inefficient model training. Therefore, the present invention adopts a multi-step non-autoregressive decoding strategy. Specifically, nodes are decoded step by step according to their node type index, and unconnected nodes are shielded according to the node connection relationship during decoding.

[0130] In this embodiment, in step S4, the preference vector is substituted into the decoding process, and a single neural network model based on the preference condition is trained to achieve preference multi-objective optimization;

[0131] Specifically, the preference vector is incorporated into the decoding process to train a single preference condition-based neural network model to achieve multi-objective optimization;

[0132] First, a problem decomposition strategy is adopted to decompose the multi-objective problem into a series of weighted single-objective sub-problems. Then, the preference is introduced into neural learning, and the weighted Chebyshev aggregation method is used to construct the preference conditional single objective. In the decoding stage, the preference vector is encoded as a weight matrix of the context vector to help the neural model learn how to generate preference conditional solutions. For an instance of M objectives, the preference vector is expressed as:

[0133]

[0134] Use a multilayer perceptron to map the preference vector to d 2 dimensional weight vector, where d is the dimension of the node embedding. This multilayer perceptron has three 256-dimensional hidden layers activated by ReLU. The weight vector is reconstructed into a weight matrix After that, the probability vector in the decoding stage becomes:

[0135]

[0136] Then, the equipment configuration scheme can be generated by the probability sampling method. Since the traditional reinforcement learning method updates parameters based on a single reward function, the present invention uses the weighted Chebyshev method to aggregate multiple optimization objective functions into one reward function.

[0137]

[0138] Where π is the solution obtained; is the target i (π) is the reference value.

[0139] In this embodiment, in step S5, the multi-objective combined optimization problem model is trained by setting a reinforcement learning algorithm to obtain the trained multi-objective combined optimization problem model;

[0140] Specifically, a gradient-based reinforcement learning method, the REINFORCE algorithm, is used to train the model parameters of the multi-objective combinatorial optimization problem. It adjusts the policy parameters based on the reward function gradient in the direction of improving the expected return without the need for an explicit value function. Given an instance r and a preference vector λ, the loss function is defined as follows:

[0141]

[0142] Where θ is a learnable model parameter; π~p θ (.|r,λ) is the solution sampled according to the strategy; f tch(π|r,λ) is a single objective function under preference conditions; b(r,λ) is a baseline of the expected cost of reducing the gradient variance, which is estimated by a critic network (Critic) using a multi-layer perceptron with 256 hidden layers and 1 output layer, and the activation function is Relu. b(r,λ) is calculated as follows:

[0143]

[0144] Therefore, the gradient of the loss is calculated as:

[0145]

[0146] Where Λ and R are the distribution of preference vectors and instances, respectively. Since it is not possible to exhaust all possible cases, Monte Carlo sampling is used to estimate the gradient. For each update step, N instances with preference vectors are sampled.

[0147] In this embodiment, in step S6, the equipment parameter information is input into the trained multi-objective combined optimization problem model, and the multi-objective combined optimization problem model is used to process and output a multi-heterogeneous equipment configuration optimization solution.

[0148] In summary, the present invention converts the equipment configuration problem into a multi-objective combination optimization problem model through modeling, and sets several evaluation indicators; by stacking several heterogeneous graph attention layers, an enhanced heterogeneous graph attention network is constructed; the features of heterogeneous nodes and edges are processed by the enhanced heterogeneous graph attention network; a fully connected deep neural network is used as a decoder, and the decoder is used to generate a probability vector; the decoder generates an equipment configuration scheme through a multi-step non-autoregressive strategy; the preference vector is substituted into the decoding process, and a single neural network model based on preference conditions is trained to achieve preference multi-objective optimization; the multi-objective combination optimization problem model is trained by setting a reinforcement learning algorithm to obtain the trained multi-objective combination optimization problem model; the equipment parameter information is input into the trained multi-objective combination optimization problem model, and the multi-objective combination optimization problem model is processed to output a multi-heterogeneous equipment configuration optimization scheme. The present invention proposes a method for realizing equipment configuration using a multi-objective heterogeneous graph reinforcement learning algorithm to solve the equipment configuration optimization problem. The framework can effectively utilize the powerful generalization ability of deep reinforcement learning and improve the algorithm's solution effect on large-scale problems and different examples. It can fully learn the basic characteristics of the problem and effectively handle configuration problems with different scales and distributions within a few seconds. The present invention systematically proposes five evaluation indicators for the configuration problem of multi-heterogeneous equipment, integrating indicators of cost, benefit, efficiency, reliability and risk, and all indicators are designed as matrix operations, which exponentially increases the algorithm operation speed. The present invention designs a single-model deep reinforcement learning optimization framework based on preference conditions, which allows the pre-trained model to approximate the Pareto frontier with any number of preferences, greatly improving the speed of generating solutions and effectively reducing resource consumption compared to traditional multi-model optimization frameworks.

[0149] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0150] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] Example 2

[0152] See also Figure 5 Embodiment 2 of the present invention further provides a multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning, comprising:

[0153] The target combination optimization problem model conversion unit 001 is used to convert the equipment configuration problem into a multi-target combination optimization problem model through modeling, and set a number of evaluation indicators;

[0154] The enhanced heterogeneous graph attention network construction and processing unit 002 is used to construct an enhanced heterogeneous graph attention network by stacking a plurality of heterogeneous graph attention layers; and the features of heterogeneous nodes and edges are processed by the enhanced heterogeneous graph attention network;

[0155] Decoder construction and processing unit 003, used to use a fully connected deep neural network as a decoder, the decoder is used to generate a probability vector; the decoder generates an equipment configuration plan through a multi-step non-autoregressive strategy;

[0156] The preference multi-objective optimization unit 004 is used to substitute the preference vector into the decoding process, train a single neural network model based on the preference condition, and realize the preference multi-objective optimization;

[0157] A multi-objective combined optimization problem model training unit 005 is used to train the multi-objective combined optimization problem model by setting a reinforcement learning algorithm to obtain the trained multi-objective combined optimization problem model;

[0158] The multi-heterogeneous equipment configuration optimization solution acquisition unit 006 is used to input equipment parameter information into the trained multi-objective combined optimization problem model, process it through the multi-objective combined optimization problem model, and output the multi-heterogeneous equipment configuration optimization solution.

[0159] In this embodiment, in the target combination optimization problem model conversion unit 001, the expression of the multi-objective combination optimization problem model is:

[0160]

[0161] In the formula, F(X) is a number of optimization objectives; M is the number of optimization objectives; X is the set of decision variables; is the i-th node of the k-th type of equipment; K is the number of equipment types; n k is the total number of equipment of type k.

[0162] In this embodiment, in the target combination optimization problem model conversion unit 001, the several evaluation indicators set include: equipment utilization rate, effectiveness generation time, solution redundancy, task completion and solution risk.

[0163] In this embodiment, in the enhanced heterogeneous graph attention network construction and processing unit 002, in the process of constructing the enhanced heterogeneous graph attention network by stacking a plurality of the heterogeneous graph attention layers, each of the heterogeneous graph attention layers includes a precoding module, an attention module, a message passing module and an aggregation module;

[0164] The expression of the precoding module is:

[0165]

[0166] In the formula, K i (t) is the Key vector; Q i (t) is the Query vector; V i (t) is the Value vector; i is the attention head; t is the node; τ(t) is the node type; l is the number of attention layers; is the linear mapping of the (l-1)th layer;

[0167] The expression of the attention module is:

[0168]

[0169] Where N(t) is the predecessor node of target node t; a i (s,e,t) is the attention vector of the i-th attention head; h is the number of attention heads; d is the embedding dimension; s is the predecessor node; is a learnable weight matrix;

[0170] The expression of the message passing module is:

[0171] V(s,e,t)=|| i∈[1,h] V i (s,e,t)

[0172]

[0173] Where V(s,e,t) is the value information vector; V i (s,e,t) is the value information vector of the i-th attention head; is a weight matrix related to the edge type, used to parameterize heterogeneous edges;

[0174] The expression of the aggregation module is:

[0175]

[0176] In the formula, H (l) (t) is the final context embedding of node t; is the updated context embedding of node t; σ is the scaling factor; A-Linear is the linear projection layer.

[0177] In this embodiment, in the decoder construction and processing unit 003, in the process of generating the probability vector by the decoder, the expression of the probability vector is:

[0178]

[0179] In the formula, p i is the probability vector; x i is a node; H (L) (x i ) is the node x i Embedding vector; is the weight matrix; is the bias vector.

[0180] In this embodiment, in the preference multi-objective optimization unit 004, when substituting the preference vector into the decoding process, the expression of the preference vector is:

[0181]

[0182] Where λ is the preference vector and M is the number of targets.

[0183] In this embodiment, in the multi-objective combined optimization problem model training unit 005, in the process of training the multi-objective combined optimization problem model by using the set reinforcement learning algorithm, the parameters of the multi-objective combined optimization problem model are trained by the REINFORCE algorithm; the expression of the loss function is:

[0184]

[0185] Where θ is a learnable model parameter; π~p θ (.|r,λ) is the solution sampled according to the policy; b(r,λ) is the baseline of the expected cost of reducing the gradient variance; r is the instance; λ is the preference vector; f tch (π|r,λ) is a single objective function under preference conditions.

[0186] It should be noted that the information interaction, execution process and other contents between the various units of the above-mentioned system are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and will not be repeated here.

[0187] Example 3

[0188] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which a program code of a multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning is stored, and the program code includes instructions for executing the multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning of embodiment 1 or any possible implementation thereof.

[0189] The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0190] Example 4

[0191] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0192] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning of Example 1 or any possible implementation thereof.

[0193] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor implemented by reading software codes stored in a memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0194] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0195] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing system, they can be concentrated on a single computing system, or distributed on a network composed of multiple computing systems, and optionally, they can be implemented by a program code executable by a computing system, so that they can be stored in a storage system and executed by the computing system, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0196] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning, characterized in that: include: The equipment configuration problem is converted into a multi-objective combination optimization problem model through modeling, and several evaluation indicators are set; By stacking several heterogeneous graph attention layers, an enhanced heterogeneous graph attention network is constructed; Processing the features of heterogeneous nodes and edges through the enhanced heterogeneous graph attention network; Using a fully connected deep neural network as a decoder, the decoder is used to generate a probability vector; The decoder generates an equipment configuration scheme through a multi-step non-autoregressive strategy; Substitute the preference vector into the decoding process, train a single neural network model based on preference conditions, and achieve preference multi-objective optimization; The multi-objective combined optimization problem model is trained by setting a reinforcement learning algorithm to obtain a trained multi-objective combined optimization problem model; The equipment parameter information is input into the trained multi-objective combined optimization problem model, and processed by the multi-objective combined optimization problem model to output a multi-heterogeneous equipment configuration optimization solution.

2. The multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning according to claim 1 is characterized in that: The expression of the multi-objective combined optimization problem model is: min F(X)=(f1(X),...,f M (X)), In the formula, F(X) is a number of optimization objectives; M is the number of optimization objectives; X is the set of decision variables; is the i-th node of the k-th type of equipment; K is the number of equipment types; n k is the total number of equipment of type k.

3. The multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning according to claim 2 is characterized in that: The several evaluation indicators set include: equipment utilization rate, effectiveness generation time, solution redundancy, task completion and solution risk.

4. The multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning according to claim 3 is characterized in that: In the process of constructing the enhanced heterogeneous graph attention network by stacking a plurality of the heterogeneous graph attention layers, each of the heterogeneous graph attention layers comprises a precoding module, an attention module, a message passing module and an aggregation module; The expression of the precoding module is: In the formula, K i (t) is the Key vector; Q i (t) is the Query vector; V i (t) is the Value vector; i is the attention head; t is the node; τ(t) is the node type; l is the number of attention layers; is the linear mapping of the (l-1)th layer; The expression of the attention module is: Where N(t) is the predecessor node of target node t; a i (s,e,t) is the attention vector of the i-th attention head; h is the number of attention heads; d is the embedding dimension; s is the predecessor node; is a learnable weight matrix; The expression of the message passing module is: V(s,e,t)=|| i∈[1,h] V i (s,e,t) Where V(s,e,t) is the value information vector; V i (s,e,t) is the value information vector of the i-th attention head; is a weight matrix related to the edge type, used to parameterize heterogeneous edges; The expression of the aggregation module is: In the formula, H (l) (t) is the final context embedding of node t; is the updated context embedding of node t; σ is the scaling factor; A-Linear is the linear projection layer.

5. The multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning according to claim 4 is characterized in that: In the process of generating the probability vector by the decoder, the expression of the probability vector is: In the formula, p i is the probability vector; x i is a node; H (L) (x i ) is the node x i Embedding vector; is the weight matrix; is the bias vector.

6. The multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning according to claim 5 is characterized in that: When substituting the preference vector into the decoding process, the expression of the preference vector is: Where λ is the preference vector and M is the number of targets.

7. The multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning according to claim 6 is characterized in that: In the process of training the multi-objective combined optimization problem model by setting the reinforcement learning algorithm, the parameters of the multi-objective combined optimization problem model are trained by the REINFORCE algorithm; the expression of the loss function is: Where θ is a learnable model parameter; π~p θ (.|r,λ) is the solution sampled according to the policy; b(r,λ) is the baseline of the expected cost of reducing the gradient variance; r is the instance; λ is the preference vector; f tch (π|r,λ) is a single objective function under preference conditions.

8. A multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning, adopting a multi-heterogeneous equipment configuration method based on multi-objective graph reinforcement learning according to any one of claims 1 to 7, characterized in that: include: The target combination optimization problem model conversion unit is used to convert the equipment configuration problem into a multi-target combination optimization problem model through modeling and set several evaluation indicators; An enhanced heterogeneous graph attention network construction and processing unit, which is used to construct an enhanced heterogeneous graph attention network by stacking several heterogeneous graph attention layers; Processing the features of heterogeneous nodes and edges through the enhanced heterogeneous graph attention network; A decoder construction and processing unit, configured to use a fully connected deep neural network as a decoder, wherein the decoder is configured to generate a probability vector; The decoder generates an equipment configuration scheme through a multi-step non-autoregressive strategy; The preference multi-objective optimization unit is used to substitute the preference vector into the decoding process, train a single neural network model based on the preference condition, and realize the preference multi-objective optimization; A multi-objective combined optimization problem model training unit, used to train the multi-objective combined optimization problem model by setting a reinforcement learning algorithm to obtain the trained multi-objective combined optimization problem model; The multi-heterogeneous equipment configuration optimization solution acquisition unit is used to input equipment parameter information into the trained multi-objective combined optimization problem model, process it through the multi-objective combined optimization problem model, and output the multi-heterogeneous equipment configuration optimization solution.

9. The multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning according to claim 8, characterized in that: In the target combination optimization problem model conversion unit, the expression of the multi-objective combination optimization problem model is: min F(X)=(f1(X),...f M (X)), In the formula, F(X) is a number of optimization objectives; M is the number of optimization objectives; X is the set of decision variables; is the i-th node of the k-th type of equipment; K is the number of equipment types; n k is the total number of equipment of type k.

10. The multi-heterogeneous equipment configuration device based on multi-objective graph reinforcement learning according to claim 9, characterized in that: In the target combination optimization problem model conversion unit, the several evaluation indicators set include: equipment utilization rate, effectiveness generation time, solution redundancy, task completion and solution risk.

Images