Deep reinforcement learning method and device based on graph vector update and neighborhood search

Abstract

The invention relates to the technical field of multi-objective optimization, in particular to a deep reinforcement learning method and device based on graph vector updating and neighborhood search, and the method comprises the steps: obtaining training data and a test problem, and training a deep reinforcement learning model through the training data; a test problem is coded by using the trained deep reinforcement learning model, a graph vector irrelevant to a weight vector is generated, decoder parameters are dynamically generated according to the weight vector, solving strategies for different optimization preferences are constructed according to the decoder parameters, and the graph vector is updated according to the solving strategies; and dividing a test problem into a plurality of sub-problems with different weight vectors, migrating graph vectors between the sub-problems corresponding to the adjacent weight vectors, generating an initial solution set of the sub-problems from a plurality of different starting points by using the trained deep reinforcement learning model, and performing neighborhood search on the initial solution set to obtain a solving result. Therefore, the problems of insufficient scale expandability, data distribution sensitivity defects and the like of related technical problems are solved.

Description

Technical Field

[0001] This application relates to the field of multi-objective optimization technology, and in particular to a deep reinforcement learning method and apparatus based on graph vector update and neighborhood search. Background Technology

[0002] Multi-objective combinatorial optimization problems are an important research direction in operations research and computer science. They aim to find the optimal or near-optimal solution set among multiple conflicting objectives and are widely used in production scheduling, logistics distribution, resource allocation, and other fields. Due to the conflicting objectives and the complexity of the decision space, these problems are difficult to solve. Traditional methods include exact algorithms, approximate algorithms, and heuristic algorithms, but they generally suffer from high computational cost, poor scalability, and reliance on expert knowledge.

[0003] In recent years, deep reinforcement learning has provided a new approach to multi-objective combinatorial optimization. It extracts state features through deep neural networks and combines them with reinforcement learning strategies for decision-making, offering advantages such as fast solution speed, data-driven approach, and strong generalization ability. Compared to traditional methods, deep reinforcement learning can generate solutions end-to-end, reducing reliance on manual rules and adapting to different optimization scenarios.

[0004] However, existing deep reinforcement learning-based methods face two major challenges in solving multi-objective combinatorial optimization problems: the scalability of the problem and the impact of changes in data distribution. When the test problem differs significantly from the training data in terms of scale or distribution, the model performance will drop significantly, limiting its application in real-world complex scenarios. Summary of the Invention

[0005] This application provides a deep reinforcement learning method and apparatus based on graph vector update and neighborhood search to solve problems such as insufficient scalability and defects in sensitivity to data distribution in related technologies.

[0006] The first aspect of this application provides a deep reinforcement learning method based on graph vector update and neighborhood search, comprising the following steps: acquiring training data and a test problem; training a deep reinforcement learning model using the training data; encoding the test problem using the trained deep reinforcement learning model to generate graph vectors independent of weight vectors; dynamically generating decoder parameters based on the weight vectors; constructing solution strategies for different optimization preferences based on the decoder parameters; updating the graph vectors based on the solution strategies; dividing the test problem into multiple sub-problems with different weight vectors; transferring graph vectors between sub-problems corresponding to adjacent weight vectors; generating initial solution sets for the sub-problems from multiple different starting points using the trained deep reinforcement learning model; and performing neighborhood search on the initial solution sets to obtain the solution results.

[0007] Optionally, a deep reinforcement learning model is trained using the training data, including: extracting solution strategies and problem features from the training data; and training the deep reinforcement learning model to learn the solution strategies and problem features.

[0008] Optionally, the decoder parameters are dynamically generated based on the weight vector, including: obtaining the hypernetwork parameters of the trained decoder; and calculating the decoder parameters based on the hypernetwork parameters and the weight vector.

[0009] Optionally, updating the graph vector according to the solution strategy includes: constructing effective solutions sequentially according to the solution strategy; calculating the probability distribution of the effective solutions and calculating the approximate estimated gradient of the probability distribution; and updating the graph vector according to the approximate estimated gradient.

[0010] Optionally, the transfer of graph vectors between subproblems corresponding to adjacent weight vectors includes: identifying the update state of the graph vector of the previous subproblem during the solution process of the subproblem corresponding to the weight vector; if the update state of the graph vector of the previous subproblem is updated, then the updated graph vector is used as the initial vector input of the current subproblem; if the update state of the graph vector of the previous subproblem is not updated, then the encoder is used to extract features from the test problem to obtain the initial graph vector, and the initial graph vector is adjusted by using reinforcement learning.

[0011] Optionally, a neighborhood search is performed on the initial solution set to obtain the solution result, including: optimizing each solution trajectory in the initial solution set using a local search algorithm, wherein the local search algorithm includes swapping the order of some nodes in the path of each solution trajectory to obtain multiple candidate solutions, selecting the target solution from the multiple candidate solutions to form an elite solution set; and determining the solution result based on the elite solution set.

[0012] A second aspect of this application provides a deep reinforcement learning apparatus based on graph vector update and neighborhood search, comprising: a training module for acquiring training data and a test problem, and training a deep reinforcement learning model using the training data; a generation module for encoding the test problem using the trained deep reinforcement learning model, generating graph vectors independent of weight vectors, dynamically generating decoder parameters based on the weight vectors, constructing solution strategies for different optimization preferences based on the decoder parameters, and updating the graph vectors based on the solution strategies; and a partitioning module for dividing the test problem into multiple sub-problems with different weight vectors, transferring graph vectors between sub-problems corresponding to adjacent weight vectors, generating initial solution sets for the sub-problems from multiple different starting points using the trained deep reinforcement learning model, and performing neighborhood search on the initial solution sets to obtain the solution results.

[0013] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the deep reinforcement learning method based on graph vector update and neighborhood search as described in the above embodiments.

[0014] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a deep reinforcement learning method based on graph vector update and neighborhood search as described in the above embodiments.

[0015] A fifth aspect of this application provides a computer program product, which, when executed, is used to implement the deep reinforcement learning method based on graph vector update and neighborhood search as described in the above embodiments.

[0016] Therefore, this application has the following beneficial effects:

[0017] This application's embodiments train a deep reinforcement learning model using training data, encode the test problem using the trained deep reinforcement learning model, generate graph vectors independent of weight vectors, dynamically generate decoder parameters based on the weight vectors, construct solution strategies for different optimization preferences based on the decoder parameters, update the graph vectors according to the solution strategies, divide the test problem into multiple sub-problems with different weight vectors, transfer graph vectors between sub-problems corresponding to adjacent weight vectors, generate initial solution sets for the sub-problems from multiple different starting points using the trained deep reinforcement learning model, and obtain the solution results by performing neighborhood search on the initial solution sets. This constructs a more general and robust end-to-end framework. Since the training phase is based only on data from a single distribution, it can improve the solution accuracy and stability of the model on different scales and spatial distributions, thus better suiting the complex environments of unknown or frequently changing distributions in practical applications. Therefore, it solves the problems of insufficient scalability and deficiencies in sensitivity to data distribution in related technologies.

[0018] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. Attached Figure Description

[0019] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 This is a flowchart illustrating a deep reinforcement learning method based on graph vector update and neighborhood search according to an embodiment of this application.

[0021] Figure 2 This is a schematic diagram of a solution method provided according to an embodiment of this application;

[0022] Figure 3 This is a flowchart of a multi-starting-point neighborhood search in a problem-solving example provided according to an embodiment of this application;

[0023] Figure 4 This is a block diagram illustrating a deep reinforcement learning device based on graph vector update and neighborhood search according to an embodiment of this application.

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0026] The following describes a deep reinforcement learning method and apparatus based on graph vector update and neighborhood search, according to embodiments of this application, with reference to the accompanying drawings. Addressing the issues of insufficient scalability and sensitivity to data distribution mentioned in the background section, this application provides a deep reinforcement learning method based on graph vector update and neighborhood search. In this method, a deep reinforcement learning model is trained using training data. The trained model is then used to encode a test problem, generating graph vectors independent of weight vectors. Decoder parameters are dynamically generated based on the weight vectors. Solution strategies for different optimization preferences are constructed based on the decoder parameters. Graph vectors are updated according to the solution strategies. The test problem is divided into multiple sub-problems with different weight vectors. Graph vectors are transferred between sub-problems corresponding to adjacent weight vectors. Initial solution sets for the sub-problems are generated from multiple different starting points using the trained deep reinforcement learning model. Neighborhood search is performed on the initial solution sets to obtain the solution results. This constructs a more general and robust end-to-end framework. During the training phase, based solely on data from a single distribution, it improves the model's solution accuracy and stability across different scales and spatial distributions, thus better suiting complex environments with unknown or frequently changing distributions in practical applications. This solves the problems of insufficient scalability and lack of sensitivity to data distribution in related technologies.

[0027] Specifically, Figure 1 This is a flowchart illustrating a deep reinforcement learning method based on graph vector update and neighborhood search provided in an embodiment of this application.

[0028] like Figure 1 As shown, this deep reinforcement learning method based on graph vector update and neighborhood search includes the following steps:

[0029] In step S101, training data and test questions are obtained, and a deep reinforcement learning model is trained using the training data.

[0030] The training data typically contains a series of representative multi-objective combinatorial optimization problem instances. These instances can be simplified versions of real-world applications such as the Traveling Salesman Problem or supply chain scheduling, without any specific limitations. The test problems, on the other hand, refer to a set of new problem instances used to evaluate the model's performance and generalization ability after the model has been trained.

[0031] It is understood that the embodiments of this application use a series of predefined multi-objective combinatorial optimization problem instances as training data to train a deep reinforcement learning model, enabling it to learn effective solution strategies; subsequently, test problems are used to verify the performance and generalization ability of the model, ensuring that the model can remain efficient and accurate when dealing with new problems of different scales and distribution characteristics.

[0032] In this embodiment of the application, training a deep reinforcement learning model using training data includes: extracting solution strategies and problem features from the training data; and training the deep reinforcement learning model to learn the solution strategies and problem features.

[0033] The solution strategy refers to a series of decision-making processes adopted by the deep reinforcement learning model in order to achieve the optimization goal, such as how to choose the next action based on the current state to approach the optimal solution; the problem features cover various information in the combinatorial optimization problem, such as the distance between nodes and the weights of each objective function, which are the basis for the model to make effective decisions.

[0034] It is understood that the embodiments of this application train a deep reinforcement learning model by extracting solution strategies and problem features of multi-objective combinatorial optimization problems from the training dataset, so that the model can learn to identify the essential features of the problem and formulate a solution strategy that can guide it to gradually approach the optimal solution, thereby enabling the model to automatically apply the learned knowledge to solve similar problems efficiently.

[0035] In step S102, the test problem is encoded using the trained deep reinforcement learning model to generate a graph vector independent of the weight vector. Decoder parameters are dynamically generated based on the weight vector. Solution strategies for different optimization preferences are constructed based on the decoder parameters. The graph vector is updated based on the solution strategies.

[0036] Among them, the weight vector represents the preference or importance between different optimization objectives, and different weight vectors will lead to different solution strategies; the graph vector refers to the vector generated by the encoder after processing the problem instance to represent the problem structure, which can capture information such as nodes and their relationships in the problem instance; the decoder parameters are dynamically generated based on the weight vector to construct the solution strategy and realize the construction of solutions for different preferences.

[0037] It is understood that the embodiments of this application utilize a pre-trained deep reinforcement learning model to analyze the test problem, generate graph vectors independent of specific weights to represent the core structure of the problem, dynamically adjust the decoder parameters based on the specific weight vectors, thereby formulating corresponding solution strategies for different optimization preferences, and finally, by executing the solution strategy and evaluating its effect, update the graph vectors in reverse to further optimize the quality and adaptability of the solution. This realizes a closed-loop process from problem understanding, strategy formulation to solution optimization and adjustment. It is worth mentioning that this scheme does not update the encoder and decoder parameters in the graph vector update part, but only updates the graph vectors related to the problem, thereby effectively reducing computational overhead, improving model adaptability, and ensuring computational efficiency.

[0038] In this embodiment of the application, the decoder parameters are dynamically generated based on the weight vector, including: obtaining the hypernetwork parameters of the trained decoder; and calculating the decoder parameters based on the hypernetwork parameters and the weight vector.

[0039] Wherein, decoder parameter θ decoder (λ i )=MLP(λ|ψ),θ decoder (λ i ) represents the weight vector λ i The decoder parameters are generated using an MLP (Multilayer Perceptron) supernetwork, where ψ is the supernetwork parameter of the MLP.

[0040] It is understood that, according to the above formula, the embodiments of this application first obtain the trained hypernetwork parameters ψ used to generate decoder parameters, and then combine the hypernetwork parameters ψ with the specific weight vector λ. i Calculate the decoder parameters θ suitable for the current optimization preference. decoder (λ i ) = MLP(λ|ψ), this process allows the model to flexibly adjust the decoding strategy according to different weight vectors to adapt to different needs in multi-objective combinatorial optimization problems, thereby achieving efficient solutions for specific preferences.

[0041] In this embodiment of the application, updating the graph vector according to the solution strategy includes: constructing effective solutions in sequence according to the solution strategy; calculating the probability distribution of the effective solutions and calculating the approximate estimated gradient of the probability distribution; and updating the graph vector according to the approximate estimated gradient.

[0042] Specifically, when actively searching for a specific instance, the graph vector calculated by the decoder is... Where s is a problem instance, and the graph vector and decoder parameters can be used to construct the current weight vector λ. iThe approximate gradient of the following solution π can be calculated using the REINFORCE algorithm (a reinforcement learning method based on policy gradients, used to learn which action to take in a given state to maximize cumulative reward):

[0043]

[0044] Where R(π|G) i ,λ i ) indicates that at weight λ i Solve the reinforcement learning reward function for π, b(G) i ,λ i ) represents the baseline function; p Gi (π i |G i ,λ i ) defines the current weight λ i and graph vectors The following solution strategy constructs the probability distribution of an effective solution in sequence as follows:

[0045] It is understood that, according to the established solution strategy, this application constructs effective solutions to the problem sequentially, calculates the probability distribution of these effective solutions, and estimates an approximate gradient. This gradient reflects how adjusting the graph vectors can improve the quality of the solution. Finally, the approximate gradient information is used to update the graph vectors, thereby optimizing the model's solution process and making the generated solution closer to the optimal solution. This series of steps constitutes a complete loop from strategy execution to feedback graph vector updates, aiming to continuously improve the quality of the solution. It should be noted that updating the entire network parameters would consume a lot of parameter update time during the solution process, affecting the actual solution efficiency of the algorithm. Therefore, this application only updates the graph vectors obtained by the encoder when updating the model parameters. It can effectively solve the problem of inconsistent distribution, and because of the graph vector With weight vector λ i Unrelated issues can be shared between different subproblems through transfer learning, thereby accelerating the solution process.

[0046] In step S103, the test problem is divided into multiple subproblems with different weight vectors. The transfer graph vectors are transferred between subproblems corresponding to adjacent weight vectors. The trained deep reinforcement learning model is used to generate initial solution sets for the subproblems from multiple different starting points. The neighborhood search is performed on the initial solution set to obtain the solution result.

[0047] Neighborhood search is a local search technique that explores the solution space near the current solution to find a better solution. In this embodiment, the neighborhood search strategy is used to improve the quality of solutions directly constructed by the deep reinforcement learning model, aiming to obtain more accurate elite solutions. Specifically, for the same problem instance, the model does not start generating solutions from a single starting point, but from multiple different starting points to cover a wider solution space. This is equivalent to solving the same problem from multiple perspectives, which not only deepens the understanding of the problem structure, but also discovers more high-quality solutions. In addition, solutions generated from different starting points can correct each other during the neighborhood search stage, thereby reducing the impact of initialization bias and ultimately improving the overall quality and diversity of solutions.

[0048] It is understood that the embodiments of this application divide the test problem into multiple sub-problems based on the weight vector, and migrate graph vectors between sub-problems corresponding to adjacent weight vectors to accelerate the solution process. The trained deep reinforcement learning model generates an initial solution set for each sub-problem from multiple different starting points, and neighborhood search is applied to these initial solution sets to optimize them, so as to further improve the quality of the solutions and obtain the final solution results. This not only improves the solution efficiency, but also enhances the adaptability and robustness of the algorithm on complex and diverse problems.

[0049] In this embodiment of the application, the transfer of graph vectors between subproblems corresponding to adjacent weight vectors includes: during the solution of the subproblems corresponding to the weight vectors, identifying the update state of the graph vector of the previous subproblem; if the update state of the graph vector of the previous subproblem is updated, then the updated graph vector is used as the initial vector input of the current subproblem; if the update state of the graph vector of the previous subproblem is not updated, then the encoder is used to extract features from the test problem to obtain the initial graph vector, and the initial graph vector is adjusted by using reinforcement learning.

[0050] It is understood that, in solving the subproblems corresponding to adjacent weight vectors, the embodiments of this application first need to check whether the graph vector of the previous subproblem has been updated. If the graph vector of the previous subproblem has been updated, i.e., it is in an updated state, then the optimized graph vector is directly used as the initial input of the current subproblem to accelerate the solution process. If the graph vector of the previous subproblem has not been updated, i.e., it is in an unupdated state, then the encoder is used to extract features from the test problem to generate the initial graph vector, and then it is adjusted by reinforcement learning. Through the optimization graph vector transfer of adjacent weight vectors, the model can be optimized at a better starting point, thereby accelerating the convergence speed. The transfer strategy not only significantly reduces redundant computation, but also improves the overall efficiency and stability of the algorithm when solving multi-objective optimization problems.

[0051] In this embodiment of the application, the solution result is obtained by performing a neighborhood search on the initial solution set, including: optimizing each solution trajectory in the initial solution set using a local search algorithm, wherein the local search algorithm includes swapping the order of some nodes in the path of each solution trajectory to obtain multiple candidate solutions, selecting the target solution from the multiple candidate solutions to form an elite solution set; and determining the solution result based on the elite solution set.

[0052] Local search algorithm is a heuristic method for solving optimization problems. It seeks better solutions by exploring the neighborhood of the current solution. In this embodiment, the local search algorithm is used to optimize each solution trajectory in the initial solution set. This includes generating multiple candidate solutions by swapping the order of some nodes in each solution trajectory path, and selecting the best one to form an elite solution set. This not only helps to discover higher quality solutions, but also reduces the impact of initialization bias by utilizing the symmetry of the solutions and exploring solutions generated from different starting points, thereby improving the overall quality and diversity of the solutions.

[0053] It is understood that the embodiments of this application utilize a local search algorithm to optimize each solution trajectory in the initial solution set. This process includes swapping the order of some nodes in each solution trajectory to generate multiple candidate solutions, selecting the best one to form an elite solution set, and finally determining the final solution result based on this elite solution set. Specifically, this involves fine-grained adjustment and optimization of the initial solution to explore better solutions in its neighborhood space, thereby ensuring a high-quality final solution. This method can effectively improve the quality of the solution, especially important when facing complex and variable optimization problems.

[0054] The deep reinforcement learning method based on graph vector update and neighborhood search proposed in this application involves training a deep reinforcement learning model using training data, encoding the test problem using the trained deep reinforcement learning model to generate graph vectors independent of weight vectors, dynamically generating decoder parameters based on the weight vectors, constructing solution strategies for different optimization preferences based on the decoder parameters, updating the graph vectors based on the solution strategies, dividing the test problem into multiple sub-problems with different weight vectors, transferring graph vectors between sub-problems corresponding to adjacent weight vectors, generating initial solution sets for the sub-problems from multiple different starting points using the trained deep reinforcement learning model, and performing neighborhood search on the initial solution sets to obtain the solution results. This constructs a more general and robust end-to-end framework. Since the training phase is based only on data from a single distribution, it can improve the solution accuracy and stability of the model on different scales and spatial distributions, thus better suiting complex environments with unknown or frequently changing distributions in practical applications.

[0055] The following specific example further describes the deep reinforcement learning method based on graph vector update and neighborhood search.

[0056] This embodiment selects a bi-objective traveling salesman problem with 40 nodes as the dataset. All coordinates in the dataset are distributed in a single-center cluster, with the cluster center at (0.5, 0.5), and all coordinate values ​​are between (0, 1). The test problem instance data is shown in Table 1, where the "node" column represents the node number, Obj1_X represents the X-coordinate of objective 1, Obj1_Y represents the Y-coordinate of objective 1, Obj2_X represents the X-coordinate of objective 2, and Obj2_Y represents the Y-coordinate of objective 2. The weights of the decomposition problem are set to 101 groups with a uniform distribution, specifically λ1 = (0, 1), λ2 = (0.01, 0.99), ..., λ 101 = (1,0), where Table 1 is the coordinate table of the test data for the 40 nodes generated.

[0057] Table 1

[0058]

[0059]

[0060]

[0061] Based on the above data, the specific implementation steps of this embodiment are as follows:

[0062] 1. Train the model on a known, small-scale dataset.

[0063] (1) Training Dataset Generation: In order to test the generalization ability of the model under different data distributions and to accelerate the training process, this embodiment uses a uniformly distributed dataset with 20 nodes. The dataset is uniformly distributed in a specific two-dimensional space, and the distance and distribution features between nodes will be used as inputs for model training.

[0064] The training process employs a reinforcement learning strategy using neural networks to optimize the training efficiency of the model.

[0065] (2) Sample selection and training iteration: During the training process, the model is updated by continuously selecting preferences and instances: Sample selection: In each training iteration, according to the set preference distribution, the model selects a preference λ from the set of possible preferences. k Then select the matching instance s i This process was optimized by comparing the performance of different instances through multiple experiments.

[0066] (3) Multi-round training: The model is trained in multiple rounds (set as T rounds). In each round of training, the model parameters are updated through reinforcement learning algorithm.

[0067] (4) Optimization algorithm: In each training step, based on the selected preferences and instances, the following optimization objective is adopted:

[0068] Objective function optimization: The error at each training step is calculated, and the model parameters are optimized using gradient descent (e.g., the Adam (Adaptive Moment Estimation) optimization algorithm). Gradient calculation involves differentiating the loss function for each instance selection and preference matching, specifically:

[0069]

[0070] (5) Model convergence and output: After multiple training steps (T rounds), the model parameters are gradually optimized and eventually converge to the optimal solution. At this point, the final trained model parameters are output.

[0071] (6) Parameter Introduction: The following describes the model training parameter settings in this embodiment, such as optimizer, learning rate, and training time. Optimizer: The Adam optimizer is used, and the learning rate is set to 10. -3 The weight decay is set to 10. -6 Each training cycle takes approximately 6 minutes on a single GPU (Graphics Processing Unit). Training consisted of 200 training cycles.

[0072] Second, using the trained model, the algorithm proposed in this invention is used to solve the above test cases. Figure 2 This is the flowchart of the algorithm, and the specific implementation steps are as follows:

[0073] (1) Initialize the solution set: First, initialize an empty solution set S to store the optimal solution in each iteration.

[0074] (2) Generate graph vectors for problem instances: For each weight vector and problem instance, generate the corresponding graph vector. Specifically, in the first iteration, the graph vector is generated using the encoder of the trained model, i.e. If it is a subsequent iteration, the results are adjusted and updated based on the previous result to generate a new graph vector.

[0075] (3) Initialize the current elite solutions: After generating the graph vectors of the problem instances, initialize the current elite solutions. These solutions will serve as the target solutions to be continuously optimized during the search process.

[0076] (4) Model Training and Optimization: Entering the iteration phase, based on the number of training steps T, the following operations are performed in each round: based on the current weight vector λ i Graph vectors Using the hyperparameters ψ of the trained model decoder, the solution π for the corresponding problem instance is constructed. i .

[0077] (5) Calculate the gradient of the current solution using the REINFORCE algorithm in reinforcement learning. Based on the calculated gradient, update the model parameters using the gradient descent method to optimize the quality of the solution.

[0078] (6) Neighborhood Search Strategy for Elite Solution Optimization: After each iteration, a neighborhood search strategy based on multiple starting nodes is used to further optimize the current elite solution. Neighborhood search explores different regions of the solution space to find potentially better solutions. For example... Figure 3 The diagram shows the process flow for multi-starting-point neighborhood search in a problem-solving example.

[0079] (7) Update the solution set: After each iteration, based on the updated model and solution vectors, regenerate the solutions for the problem instance and update the current solution set. Each solution is evaluated, and the optimal solution in the solution set is updated based on the optimization results.

[0080] (8) Return the non-dominated solution set: After all iteration steps are completed, extract the non-dominated solutions from the solution set S. These solutions represent the approximate Pareto optimal solution set of the problem.

[0081] (9) Parameter settings: In this embodiment, the Adam optimizer is used to update the network parameters. The learning rate is determined by performing a grid search within the interval [0.01, 0.00001] and is finally set to 0.005. An appropriate decay rate of 10 is also set. -6 Furthermore, the proposed method incorporates two key hyperparameters: the number of active graph vector updates and the number of neighborhood search iterations, both closely related to the problem size. As the problem size increases, the search space also expands, thus requiring more iterations to avoid getting trapped in local optima. However, excessive iterations may lead to increased computational overhead with limited benefits. Considering both efficiency and solution accuracy, GEULS sets the graph vector update ratio to 0.2 and the neighborhood search ratio to 1.0. For example, for a problem instance with 100 nodes, the graph vector updates are performed 20 times, and the neighborhood search is performed 100 times.

[0082] III. Output Experimental Results. Based on the above operations, the Pareto values ​​of the non-dominated solutions of the problem instance under 101 weight vectors in the target space are shown in Table 2, where Table 2 represents the non-dominated solutions under 101 weight vectors.

[0083] Table 2

[0084]

[0085]

[0086]

[0087]

[0088] Next, referring to the accompanying drawings, a deep reinforcement learning apparatus based on graph vector update and neighborhood search is described according to an embodiment of this application.

[0089] Figure 4 This is a block diagram of a deep reinforcement learning device based on graph vector update and neighborhood search according to an embodiment of this application.

[0090] like Figure 4 As shown, the deep reinforcement learning device 10 based on graph vector update and neighborhood search includes: a training module 201, a generation module 202, and a partitioning module 203.

[0091] The training module 201 is used to acquire training data and test problems, and to train a deep reinforcement learning model using the training data. The generation module 202 is used to encode the test problems using the trained deep reinforcement learning model, generate graph vectors independent of the weight vectors, dynamically generate decoder parameters based on the weight vectors, construct solution strategies for different optimization preferences based on the decoder parameters, and update the graph vectors based on the solution strategies. The partitioning module 203 is used to divide the test problems into multiple sub-problems with different weight vectors, transfer graph vectors between sub-problems corresponding to adjacent weight vectors, generate initial solution sets for the sub-problems from multiple different starting points using the trained deep reinforcement learning model, and perform neighborhood search on the initial solution sets to obtain the solution results.

[0092] In this embodiment, the training module 201 is further configured to: extract the solution strategy and problem features from the training data; and train the deep reinforcement learning model to learn the solution strategy and problem features.

[0093] In this embodiment of the application, the generation module 202 is further configured to: obtain the hypernetwork parameters of the trained decoder; and calculate the decoder parameters based on the hypernetwork parameters and the weight vector.

[0094] In this embodiment of the application, the generation module 202 is further configured to: construct effective solutions sequentially according to the solution strategy; calculate the probability distribution of the effective solutions, calculate the approximate estimated gradient of the probability distribution; and update the graph vector according to the approximate estimated gradient.

[0095] In this embodiment of the application, the partitioning module 203 is further configured to: identify the graph vector update state of the previous subproblem during the subproblem solving process corresponding to the weight vector; if the graph vector update state of the previous subproblem is updated, then the updated graph vector is used as the initial vector input of the current subproblem; if the graph vector update state of the previous subproblem is not updated, then the encoder is used to extract features from the test problem to obtain the initial graph vector, and the initial graph vector is adjusted by using reinforcement learning.

[0096] In this embodiment of the application, the partitioning module 203 is further used to: optimize each solution trajectory in the initial solution set using a local search algorithm, wherein the local search algorithm includes swapping the order of some nodes in the path of each solution trajectory to obtain multiple candidate solutions, selecting the target solution from the multiple candidate solutions to form an elite solution set; and determining the solution result based on the elite solution set.

[0097] It should be noted that the foregoing explanation of the deep reinforcement learning method based on graph vector update and neighborhood search also applies to the deep reinforcement learning device based on graph vector update and neighborhood search in this embodiment, and will not be repeated here.

[0098] The deep reinforcement learning device based on graph vector update and neighborhood search proposed in this application trains a deep reinforcement learning model using training data. The trained deep reinforcement learning model encodes the test problem, generating graph vectors independent of weight vectors. Decoder parameters are dynamically generated based on the weight vectors. Solution strategies for different optimization preferences are constructed based on the decoder parameters. Graph vectors are updated according to the solution strategies, dividing the test problem into multiple sub-problems with different weight vectors. Graph vectors are transferred between sub-problems corresponding to adjacent weight vectors. Initial solution sets for sub-problems are generated from multiple different starting points using the trained deep reinforcement learning model. Neighborhood search is performed on the initial solution sets to obtain the solution results. This constructs a more general and robust end-to-end framework. During the training phase, based only on data from a single distribution, it can improve the solution accuracy and stability of the model on different scales and spatial distributions, thus better suiting complex environments with unknown or frequently changing distributions in practical applications.

[0099] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0100] The memory 301, the processor 302, and the computer program stored on the memory 301 and capable of running on the processor 302.

[0101] When the processor 302 executes the program, it implements the deep reinforcement learning method based on graph vector update and neighborhood search provided in the above embodiments.

[0102] Furthermore, electronic devices also include:

[0103] Communication interface 303 is used for communication between memory 301 and processor 302.

[0104] The memory 301 is used to store computer programs that can run on the processor 302.

[0105] The memory 301 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0106] If the memory 301, processor 302, and communication interface 303 are implemented independently, then the communication interface 303, memory 301, and processor 302 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0107] Optionally, in a specific implementation, if the memory 301, processor 302, and communication interface 303 are integrated on a single chip, then the memory 301, processor 302, and communication interface 303 can communicate with each other through an internal interface.

[0108] Processor 302 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of this application.

[0109] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the deep reinforcement learning method based on graph vector update and neighborhood search as described above.

[0110] This application also provides a computer program product, including a computer program or instructions, which, when executed, implement the deep reinforcement learning method based on graph vector update and neighborhood search as described above.

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0113] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0114] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0115] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0116] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A deep reinforcement learning method based on graph vector update and neighborhood search, characterized in that, Includes the following steps: Acquire training data and test questions, and use the training data to train a deep reinforcement learning model; The test problem is encoded using a trained deep reinforcement learning model to generate a graph vector independent of the weight vector. Decoder parameters are dynamically generated based on the weight vector. A solution strategy for different optimization preferences is constructed based on the decoder parameters. The graph vector is updated based on the solution strategy. The test problem is divided into multiple subproblems with different weight vectors. The graph vectors are transferred between subproblems corresponding to adjacent weight vectors. An initial solution set for the subproblems is generated from multiple different starting points using a trained deep reinforcement learning model. The solution result is obtained by performing a neighborhood search on the initial solution set.

2. The deep reinforcement learning method based on graph vector update and neighborhood search according to claim 1, characterized in that, The step of training a deep reinforcement learning model using the training data includes: Extract the solution strategy and problem features from the training data; The deep reinforcement learning model is trained to learn the solution strategy and the problem features.

3. The deep reinforcement learning method based on graph vector update and neighborhood search according to claim 1, characterized in that, The step of dynamically generating decoder parameters based on the weight vector includes: Obtain the hypernetwork parameters of the trained decoder; The decoder parameters are calculated based on the hypernetwork parameters and the weight vector.

4. The deep reinforcement learning method based on graph vector update and neighborhood search according to claim 1, characterized in that, Updating the graph vector according to the solution strategy includes: Construct effective solutions sequentially according to the aforementioned solution strategy; Calculate the probability distribution of the effective solution, and calculate the approximate estimated gradient of the probability distribution; The graph vector is updated based on the approximate estimated gradient.

5. The deep reinforcement learning method based on graph vector update and neighborhood search according to claim 1, characterized in that, The step of transferring the graph vector between corresponding sub-problems of adjacent weight vectors includes: During the process of solving the subproblems corresponding to the weight vector, the update state of the graph vector of the previous subproblem is identified; If the graph vector of the previous subproblem is in the updated state, then the updated graph vector is used as the initial vector input for the current subproblem. If the previous sub-problem graph vector update state is not updated, then the encoder is used to extract features from the test problem to obtain an initial graph vector, and then the initial graph vector is adjusted by using reinforcement learning.

6. The deep reinforcement learning method based on graph vector update and neighborhood search according to claim 1, characterized in that, The process of obtaining the solution result by performing a neighborhood search on the initial solution set includes: The local search algorithm is used to optimize each solution trajectory in the initial solution set. The local search algorithm includes swapping the order of some nodes in the path of each solution trajectory to obtain multiple candidate solutions, and selecting the target solution from the multiple candidate solutions to form an elite solution set. The solution result is determined based on the elite solution set.

7. A deep reinforcement learning device based on graph vector update and neighborhood search, characterized in that, include: The training module is used to acquire training data and test questions, and to train a deep reinforcement learning model using the training data. The generation module is used to encode the test problem using the trained deep reinforcement learning model, generate a graph vector that is independent of the weight vector, dynamically generate decoder parameters based on the weight vector, construct a solution strategy for different optimization preferences based on the decoder parameters, and update the graph vector based on the solution strategy. The partitioning module is used to divide the test problem into multiple subproblems with different weight vectors, transfer the graph vector between subproblems corresponding to adjacent weight vectors, generate an initial solution set for the subproblems from multiple different starting points using a trained deep reinforcement learning model, and perform a neighborhood search on the initial solution set to obtain the solution result.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the deep reinforcement learning method based on graph vector update and neighborhood search as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed, they implement the deep reinforcement learning method based on graph vector update and neighborhood search as described in any one of claims 1-6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed, they implement the deep reinforcement learning method based on graph vector update and neighborhood search as described in any one of claims 1-6.

Images