Method for solving set coverage problem based on deep reinforcement learning

Abstract

The invention provides a method for solving a set coverage problem based on deep reinforcement learning. According to the method, an SCP problem instance serves as input, a problem input through a problem decomposition module is decomposed into a plurality of sub-problems, the problem decomposition module divides all subsets into a plurality of groups, and it is guaranteed that the subsets in the groups are associated and the association of the subsets in different groups is relatively small; the method comprises the following steps of: obtaining a plurality of sub-problems, sending each sub-problem to each solving module for respectively solving, solving the problem by the solving module by constructing a trained deep reinforcement learning model, finally inputting the solutions obtained by each solving module into a global optimization module for a fusion stage, and merging the solutions of the sub-problems to obtain an input solution of an SCP problem instance. The performance of the method is superior to other artificial intelligence models used for solving the combinatorial optimization problem in the current industry when solving the two types of set coverage problems, and the advantages of the deep learning model can be fully utilized. The generalization of the method can be used for solving different problems, so that online real-time solving of the set coverage problem becomes possible.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and more specifically, to a method for solving set covering problems based on deep reinforcement learning. Background Art

[0002] The set cover problem is a classic site selection problem, which mainly addresses how to select the minimum number of elements from some sets to cover a given set of elements. In practical applications, the set cover problem has a wide range of application scenarios, such as the site selection of emergency facilities and public facilities, advertising placement, etc. Solving this problem is an NP-hard problem, and even exhaustive search is very difficult for medium-sized problems. Currently, traditional methods for solving set cover problems mainly include two categories: exact methods and heuristic methods. Exact methods are a type of algorithm that obtains the global optimal solution, mainly including branch and bound and cutting plane algorithms. Compared with exact methods, heuristic methods can obtain local optimal solutions within an acceptable time.

[0003] 1. Research on solving set covering problems based on traditional methods:

[0004] An exact algorithm is one that can obtain an exact solution to a problem. For the set cover problem (SCP), the simplest exact algorithm is brute force search, which views the set cover problem as a permutation of all nodes. The solution involves searching through all possible permutations and combinations to find the solution with the lowest cost. When the problem contains N nodes, this method has a time complexity of O(N!), making it NP-hard and almost infeasible in practical applications. A better approach involves algorithms based on integer linear programming, which transform the SCP problem into an ILP problem and then compress the search space using branch-and-bound methods. This compression process first decomposes the large problem into multiple smaller problems, then uses linear relaxation to calculate lower bounds for different branches, thereby predicting which branches do not require further search. Such algorithms can achieve optimal solutions more quickly in practical examples. A more advanced approach is the branch-and-cut method, which combines branch-and-bound with the cutting plane method. By designing cutting planes for different problems, this method produces a tighter estimate of the lower bound, further narrowing the solution space. Solvers based on this method can now solve SCP problems with tens of thousands of nodes.

[0005] Heuristic algorithms aim to obtain suboptimal solutions to problems in a short period of time, but they generally cannot theoretically guarantee the quality of the solutions. Common heuristic algorithms for SCP problems can be broadly categorized into constructive and improved approaches based on their solution process. Constructive approaches directly construct a feasible solution given a set of rules. Commonly used constructive approaches include the nearest neighbor method, minimum cost insertion method, random insertion method, and minimum spanning tree method.

[0006] Algorithms based on a single solution include simulated annealing, tabu search, and neighborhood search. These algorithms search based on the initial solution and use a series of methods to improve the search efficiency and ultimately obtain a local optimal solution.

[0007] 2. Solving combinatorial optimization problems based on deep learning:

[0008] In recent years, algorithms for solving combinatorial optimization problems based on deep learning and reinforcement learning have gradually emerged. These algorithms can be divided into two categories: end-to-end deep learning algorithms and algorithms that combine deep learning with traditional heuristic methods.

[0009] (1) End-to-end deep learning solution algorithm:

[0010] The overall process of the end-to-end algorithm is similar to a constructive heuristic algorithm, which gradually constructs a feasible solution. Modern deep learning network models are used to solve combinatorial optimization problems and are trained through supervised learning or reinforcement learning. For example, the pointer network combines the sequence-to-sequence learning framework with an improved attention mechanism. The encoder compresses the entire input sequence into a fixed-length vector, which is then used by the decoder to generate the output sequence. The core idea of the pointer network is to use the encoder to encode the input sequence of the combinatorial optimization problem and derive a feature vector. Then, combined with the attention calculation method, this feature vector is used to gradually construct a solution in an autoregressive manner. Autoregression refers to selecting a node at a time and determining the next node based on the previously selected node until a complete solution is formed. Alternatively, the problem can be constructed as a graph structure input model, and the probability of the node is output through a graph neural network.

[0011] (2) Solving algorithm combining deep learning method and heuristic method:

[0012] Unlike end-to-end heuristic methods, these algorithms typically combine deep learning with existing heuristic rules. Local search is a classic method for solving combinatorial optimization problems. Current local search algorithms primarily rely on manually designing search heuristic rules to achieve better optimization results. Given the remarkable learning capabilities of deep reinforcement learning in various fields in recent years, scholars have begun to study how to use deep reinforcement learning methods to automatically learn the heuristic rules of local search algorithms, thereby achieving better search capabilities than manually designed search rules.

[0013] Solving methods based on deep learning models surpass traditional algorithms in terms of speed and scale, even solving problems that are difficult for traditional algorithms. They have become a hot topic of research in recent years. Unconstrained by manual experience, these methods can automatically discover problem-solving strategies. Once trained, the models can be generalized to solve any similar problem, eliminating the drawback of traditional algorithms that are designed specifically for problems with the same structure. In particular, as the scale of problems increases, deep learning methods far surpass traditional algorithms in speed. Supervised learning often suffers from a lack of training sets due to its extremely high computational complexity, but reinforcement learning can address this issue. Summary of the Invention

[0014] The purpose of the embodiments of the present disclosure is to provide a method for solving the set cover problem based on deep reinforcement learning, which is applied to the problem of express delivery warehouse location selection. This problem can be modeled as a set cover problem, but existing deep learning methods are mostly used for simpler single-objective combinatorial optimization problems in practice, lacking deep learning methods for SCP problems. Furthermore, existing models are generally capable of handling relatively small-scale problems, and further experiments have not been conducted on larger-scale problems such as express delivery network layout. Furthermore, there is room for improvement in both accuracy and efficiency.

[0015] In general, a method for solving the set covering problem based on deep reinforcement learning is provided. The method uses express station distribution data modeled as instances of the SCP problem as input, with express stations as nodes. A maximum service distance S is set, and if the distance between two nodes does not exceed S, an edge exists between the two stations. The problem input by the problem decomposition module is decomposed into multiple subproblems, and each subproblem is sent to a separate solution module for solution. The solution module solves the problem by constructing a trained deep reinforcement learning model. Finally, the solutions obtained by each solution module are input into the global optimization module for a fusion phase, which combines the solutions to the subproblems to obtain the solution to the input SCP problem instance.

[0016] The problem decomposition module divides all subsets into multiple groups, ensuring that the subsets between groups are correlated and the correlation between subsets in different groups is relatively small.

[0017] The specific method for ensuring that the subsets between groups are associated and the subsets of different groups are relatively less associated is: i As a node, when S i ∩S j When it is not equal to 0, establish edge e ij , S i ∩S j The value of the node is recorded as the weight between nodes as w ij , calculate S i With S j The expected value of the weight Qij , calculate the difference between the two and record it as

[0018]

[0019] Where δ(s i ,s j ) indicates whether the two sets are in the same group. In the initial stage, the value is 0; if they are in the same group, the value is accumulated, otherwise no calculation is performed; at the time of initialization, each set is treated as an independent group G k =S k , for each subset node S i , calculate its move to group G j When the gain is j Abstracted as a super node, the expected value of the weight is recalculated. After multiple iterations, until Δq does not change much, that is, the grouping is stable, the subset partitioning structure will eventually be obtained, and each group will be regarded as a sub-problem of set coverage.

[0020] The deep reinforcement learning model of the solution module is a set covering problem solution algorithm based on deep reinforcement learning. Its specific implementation method is as follows: first, the solution process of the set covering problem is modeled as a Markov decision process; the set covering problem is modeled as a Markov process by defining appropriate state space, action space, reward function and state transition rules. The reward function is specifically constructed as follows:

[0021]

[0022] The current state is S t , action is a t , the subset is u j , the new state is S t+1 , the node space is U t+1 ;

[0023] Afterwards, the network model is designed following the encoder-decoder pattern, where the encoder's embedding features consist of two parts: static embedding and dynamic embedding. For the static embedding, centrality encoding and side information encoding are introduced. The set cover problem is represented as a graph structure, where nodes represent elements in the set and edges represent the coverage relationship between nodes.

[0024] The centrality encoding method introduces node importance information into the model by measuring the coverage capacity of the nodes;

[0025] The dynamic embedding updates the coverage status of the node in each decoding step, so that the model can better understand the role of the node in the current solution set. The specific implementation method is: define a vector, the vector g of each node j jrepresents the coverage status of the node, and g j Dynamic update is performed in each decoding step; suppose that in decoding step t, for subset i, C(π t ) is defined as the set of nodes covered by i, and the sum of g of the nodes in C(i) is calculated. j value;

[0026] At the beginning of the decoding process, the g of all nodes j The value is initialized to 1 and becomes 0 when overwritten: g j =1,j=1,2,...N; in each decoding step, g j The value is updated as follows according to the following formula:

[0027]

[0028] Where T is the dividing point, a1 and a2 control the growth rate at different stages, and are updated according to the access status of the node. Once node j is not accessed, the g of node j j The value will increase accordingly. Once the node is covered, its g j The value will be 0 if node j is always ignored and its g j The value will accumulate to a relatively large value, and when it is not covered for more than T steps, its growth rate will further increase. Therefore, this vector contains the coverage status of each node: whether it is covered and the degree to which it is ignored during the solution process. This vector can effectively reflect the coverage information of the subset, thereby guiding the selection of the next subset;

[0029] The role of the decoder is to generate a probability distribution for each node being selected, and then select subsets one by one in each decoding process to construct a solution set; at the beginning of the decoding process, the model takes the current partial solution as input, calculates its query value, and calculates the degree of matching with the key values of all nodes. The query value of the partial solution and the key value of the node are generated by the subset embedding after the encoder is processed by multiple layers of Transformer; in order to prevent the subset that has been selected from being selected again during the decoding process, a mask mechanism is introduced. Whenever a node is selected, its selection probability will be set to 0, thereby avoiding the selection of the selected node again in the subsequent decoding steps.

[0030] The global optimization module merges the solution sets of the sub-problems obtained from the lower-level problems, and inputs them as simplified problems in the original problem into the deep reinforcement model solver for solving, thereby obtaining the final solution to the problem.

[0031] The technical effects to be achieved by the embodiments of the present invention are:

[0032] This paper uses deep reinforcement learning to solve two types of set covering problems: the Position Set Covering Problem (LSCP) and the Maximum Set Covering Problem (MCLP). By analyzing the characteristics of set covering problems and modeling them using Markov processes, we address the shortcomings of traditional combinatorial optimization methods and propose a framework for solving set covering problems using deep reinforcement learning. To address the difficulty of training large-scale set covering problems, we design a hierarchical reinforcement learning framework and corresponding sparse and compression algorithms to optimize the framework's inference performance.

[0033] The effect of the present invention is achieved by:

[0034] A framework for solving set covering problems based on deep reinforcement learning: This paper designs a Transformer-based encoder-decoder model that combines the different characteristics of two types of set covering problems and designs a targeted model structure and training method. The proposed deep reinforcement learning-based model for solving these two types of set covering problems outperforms other AI models currently used in the industry for solving combinatorial optimization problems, fully leveraging the advantages of deep learning models. Its generalization can be applied to solve different problems, making it possible to solve set covering problems online and in real time.

[0035] Large-scale set covering problem solving method based on hierarchical reinforcement learning: The set covering problem solving framework based on deep reinforcement learning faces the problems of limited computing power and difficulty in training when training large-scale problems. Therefore, a hierarchical reinforcement learning framework is designed to solve large-scale set covering problems. This allows the advantages of deep learning algorithms to be utilized when solving large-scale set covering problems. A hierarchical set covering problem solving algorithm is proposed. This algorithm not only utilizes the efficient solving ability of deep learning models, but also uses the method of generating sub-problems, giving the algorithm better generalization ability than end-to-end deep learning solving algorithms. It can solve large-scale set covering problems with dynamic characteristics, variable solution sequence length, and overcome the limitation that large-scale set covering problems cannot be trained and solved under limited computing power.

[0036] This paper proposes a low-latency optimization technique for deep learning problem solving frameworks based on a sparse attention mechanism and a bit compression algorithm. When solving set cover problems, the time required for solving them increases significantly with the scale of the problem. This paper addresses this issue by designing a corresponding sparse attention mechanism and bit compression algorithm. This reduces the number of parameters and computations required during the solution, reduces memory usage, increases throughput, and further enhances the speed of the solution.

[0037] Based on the layout problem of the distribution of express delivery warehouses in real life, this algorithm was applied to the location selection problem of express delivery warehouses, and 500 express delivery stations in Chaoyang District, Beijing were selected for solution. The solution consists of two problems. Problem one is to select the minimum number of nodes from the express delivery stations as all the stations that the warehouse can cover within the maximum service distance; Problem two is to select a specified number of nodes from the express delivery stations as the maximum number of stations that the warehouse can cover within the maximum service distance. For both problems, the present invention can obtain site selection solutions that are close to the optimal solution.

[0038] Table 1 Performance of the algorithm in the express warehouse location problem

[0039] BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0041] Figure 1 1 is a schematic diagram illustrating an architecture diagram of a method for solving a set covering problem based on deep reinforcement learning according to an embodiment of the present disclosure;

[0042] Figure 2 is a schematic diagram illustrating a Markov process according to an embodiment of the present disclosure;

[0043] Figure 3 is a schematic diagram showing a model structure based on Tranformer according to an embodiment of the present disclosure;

[0044] Figure 4 FIG. 4 is a flow chart illustrating a fusion phase according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the present application. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and conciseness, descriptions of features known in the art may be omitted.

[0046] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will become clear after understanding the disclosure of this application.

[0047] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0048] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are used solely to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Thus, what is referred to as a first member, first component, first region, first layer, or first portion in the examples described herein may also be referred to as a second member, second component, second region, second layer, or second portion without departing from the teachings of the examples.

[0049] In the specification, when an element (such as a layer, region, or substrate) is described as being “on,” “connected to,” or “coupled to” another element, the element may be directly “on,” “connected to,” or “coupled to” the other element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on,” “directly connected to,” or “directly coupled to” another element, there may be no other elements present therebetween.

[0050] The terms used herein are intended only to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular is intended to include the plural. The terms "comprise," "include," and "have" indicate the presence of the recited features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0051] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure pertains after understanding the present disclosure. Unless expressly defined otherwise herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0052] Furthermore, in describing the examples, when it is deemed that a detailed description of well-known related structures or functions would cause ambiguous interpretation of the present disclosure, such detailed description will be omitted.

[0053] Figure 1 is a schematic diagram illustrating a method for solving a set covering problem based on deep reinforcement learning according to an embodiment of the present disclosure.

[0054] In order to achieve the above-mentioned purpose of the invention, the technical framework adopted by the present invention is as follows Figure 1 shown.

[0055] In response to the above-mentioned purpose of the invention, the present invention designs a deep reinforcement learning algorithm based on deep reinforcement learning to solve the set covering problem, a Transformer encoding and decoding model reinforcement learning algorithm, models the solution to the set covering problem, analyzes the problems that existing research cannot adapt to when solving the set covering problem, constructs a deep reinforcement learning model, and uses Jaccard similarity to improve the calculation of self-attention. Secondly, an algorithm based on the combination of static embedding and dynamic embedding is proposed and designed to adapt to the solution process of the set covering problem and improve the solution accuracy. Based on the characteristics of the problem, the idea of sparse attention and bit compression is used to optimize the reasoning efficiency of the model. Because the training speed of the algorithm is very slow when facing large-scale problems, this seriously limits the practical application value of the algorithm. To solve this problem, a hierarchical algorithm design is further proposed.

[0056] Large-scale set covering solution based on hierarchical reinforcement learning:

[0057] To leverage the advantages of deep learning algorithms in solving large-scale SCP problems, this paper draws inspiration from the traditional heuristic algorithm approach to large-scale problems. Using the divide-and-conquer approach, it simplifies large-scale problems into multiple, less challenging sub-problems, thereby reducing the overall difficulty of solving the problem and leveraging the advantages of deep learning algorithms in solving sub-problems. The layered algorithm structure is well suited to the divide-and-conquer approach. The upper-level algorithm simplifies the original problem and generates sub-problems, while the lower-level algorithm focuses on solving the sub-problems.

[0058] After the SCP problem instance is input, the decomposition stage decomposes it into multiple sub-problems, and the generated sub-problems are handed over to the decoder for solving. Then, in the fusion stage, the solutions of the sub-problems are merged to finally obtain the solution to the input SCP problem instance.

[0059] Our goal is to divide all subsets into multiple groups, and try to ensure that the subsets between groups are related, and the subsets of different groups are relatively less related. The algorithm process is as follows: For a large-scale set covering problem, we divide each subset S i As a node, when S i ∩S j When it is not equal to 0, establish edge e ij , S i ∩S j The value of the node is recorded as the weight between nodes as w ij , calculate S i With S j The expected value of the weight Q ij, calculate the difference between the two and record it as As shown in the following formula. If the two are in the same group, the value is accumulated, otherwise no calculation is performed. If the grouping effect is good, the accumulated value should be as large as possible. If the division is too fine and no subsets are merged, the accumulated value will be 0. The core idea of this algorithm is to maximize the connection within the group while minimizing the connection between groups. At the initialization, each set is treated as an independent group G k =S k , for each subset node S i , calculate its move to group G j Then each group G j Abstracted as a super node, the expected value of the weight is recalculated. After multiple iterations, until Δq does not change much, that is, the grouping is stable, the subset partitioning structure will eventually be obtained, and each group will be regarded as a sub-problem of set coverage.

[0060]

[0061] Where δ(s i ,s j ) indicates whether two sets are in the same group. In the initial stage, the value is 0.

[0062] During the subproblem-solving phase, the present invention proposes a deep reinforcement learning model to solve the subproblems. However, due to the potential overlap between subproblems, simply merging the solution sets of all subproblems may result in a poorly feasible solution. Therefore, during the fusion phase, the solution sets of the subproblems obtained from the underlying problems are merged and then fed into the deep reinforcement model solver as a simplified version of the original problem, yielding the final solution.

[0063] For the maximum set cover problem, a differential election of candidate subsets is required. For example, to select the nodes with the most coverage from p subsets, 2p are preset and distributed according to the size of the subproblems. In the fusion stage, a candidate pool of 2p will be obtained, and then p subsets will be screened out from the candidate pool of 2p.

[0064] Algorithm for solving set covering problem based on deep reinforcement learning:

[0065] In order to use the deep reinforcement learning algorithm to train the model, we first need to model the solution process of the set covering problem as a Markov decision process (MDP). The specific process is as follows: Figure 2 As shown in the figure. From the problem analysis, we can see that the set covering problem can be modeled as a Markov process by defining appropriate state space, action space, reward function and state transition rules. The specific process is as follows:

[0066] (1) State space (S): In the MDP of the set cover problem, the state represents the set of nodes that are currently covered. Assume that the node space of the problem is U = {u1,u2,...,u n} contains n nodes. In the initial state, no node is covered. As the actions are executed, the nodes are gradually covered. Therefore, the state can be represented by a binary vector. Whether each node is covered can be represented by 0 or 1. Each state corresponds to a specific partial solution. The state update process is based on the action selected by the agent at a certain moment, that is, selecting a set, adding the set to the solution set, and updating its newly covered nodes. The state change at each step can be represented by a set addition operation. For the LSCP problem, the state transition process will eventually end in a state where the nodes are completely covered; for the MCLP problem, the final state transition process will end when the specified number of subsets have been selected.

[0067] (2) Action Space (A): In each state, the agent can choose to join a set to the solution set, and the action selected corresponds to covering certain specific nodes. The choice of action may be constrained by the current state. For example, under certain strategies, it may be preferred to cover important nodes that are not currently covered.

[0068] (3) State transition (T) The state transition rule reflects how the system transitions from the current state to the next state after the agent selects an action in a certain state. Specifically, after the agent selects a set to join the solution set, the state of the system will be updated to the new coverage state. Each state transition corresponds to the addition of a new candidate subset. Assume that the current state is S t , and the agent chooses to set the subset u j Add to the current solution set, the new state S t+1 will be the covering subset u j The ultimate goal of LSCP state transition is to achieve full coverage, that is, the solution set covers all nodes, and the ultimate goal of MCLP state transition is to select a specified number of subsets.

[0069] (4) Reward function (R): The reward function is used to guide the learning process of the agent. In the set problem, we can set a stage reward in each step of the selection process. Usually, after the agent selects a set to join the solution set, it is given a reward of 0, because before the final full coverage solution, all the intermediate steps are only partial solutions and are not truly optimized. For the LSCP problem, when the full coverage state is reached, the agent will receive a negative reward, which can be calculated by the size of the solution. For example, assuming that the solution finally obtained by the agent contains k subsets, the reward can be set to -k, where the smaller k is, the better the solution is and the higher the reward is. Specifically, the reward function can be defined as:

[0070]

[0071] The current state is S t , action is a t , the cost is C t+1 , the new state is S t+1 , the node space is U t+1 . That is, a negative reward is given when a complete solution is reached, and the absolute value of the reward is the size of the current solution set; in other cases, the reward is 0. This design can encourage the agent to select as few nodes as possible when reaching a complete solution. For MCLP problems, when a specified number of subsets are selected, the agent will receive a positive reward, which can be calculated based on the number of nodes covered by the solution set. For example, assuming that the solution finally obtained by the agent contains n nodes, the reward can be set to n, where the larger n is, the better the solution is and the higher the reward is.

[0072] After modeling the problem as a Markov process, a reinforcement learning algorithm can be used for training, enabling the model to learn how to solve the set cover problem while interacting with the environment. The overall structure of the model used in this algorithm is similar to the Transformer network, which has been widely used in natural language processing and computer vision in recent years. The main body of the model is composed of a multi-head attention mechanism and a multi-layer perceptron (MLP). The decoding part incorporates some masking mechanisms designed based on the characteristics of the problem being solved.

[0073] The network model design in this invention follows the Encoder-Decoder mode, and the overall structure is as follows: Figure 3As shown in the figure, the encoder encodes the problem instance features and obtains the embedding value of each node in the graph. The decoder then calculates a weight value based on the embedding value of each node and the current partial solution using the attention mechanism. This weight represents the probability distribution of the selected node. The decoding process can use autoregressive methods to select nodes one by one until the selected nodes constitute a complete solution to the problem.

[0074] The encoder structure is a classic Transformer Encoder. Its purpose is to output an encoded feature vector for each node. The embedding feature proposed in this invention consists of two parts: static embedding and dynamic embedding. In solving the set cover problem, node coverage and interactions are key factors influencing the solution. Regarding static embedding, this invention introduces centrality encoding and edge information encoding to construct a more expressive feature vector space, thereby better capturing the coverage relationships and potential connections between nodes. The set cover problem can be naturally represented as a graph structure, where nodes represent elements in a set and edges represent the coverage relationships between nodes. To fully utilize the information in the graph structure, this invention designs two graph structure encoding methods: centrality encoding and edge information encoding. These two encoding methods, respectively, focus on node importance and edge coverage relationships, providing richer feature information for the model. Centrality encoding aims to capture the importance of a node in the graph. In the set cover problem, the greater the coverage of a node, the greater its importance in the solution set. Therefore, centrality encoding introduces information about node importance into the model by measuring the coverage capacity of a node. Degree centrality is the simplest centrality measure, which measures the importance of a node by calculating the degree of the node (that is, the number of edges connected to the node).

[0075] In the process of solving the set cover problem, the state of the node is not always static. Specifically, in the set cover problem, the coverage state of the node will change dynamically as the solution progresses. For example, if a node is not covered in multiple decoding steps, it may become a "neglected" node and need to be emphasized more strongly in subsequent steps to avoid missing key nodes. Therefore, the concept of dynamic embedding is introduced on the basis of the traditional static embedding representation in order to solve the problem of dynamic changes in node state, so that the decoding process can be adaptively adjusted according to the coverage state of the node. Dynamic embedding can update the coverage state of the node in each decoding step, so that the model can better understand the role of the node in the current solution set. In order to effectively deal with the dynamic characteristics of the problem, this chapter defines a vector, the vector g of each node j j represents the coverage status of the node, and g j Dynamic updates are performed at each decoding step.

[0076] Specifically, the calculation process of node dynamic embedding is as follows: Assume that in the decoding step t, for subset i, C(π t ) is defined as the set of nodes covered by i. The following data is calculated: sum the g of the nodes in C(i) j At the beginning of the decoding process, the g of all nodes j The value is initialized to 1 and becomes 0 when overwritten:

[0077] g j =1,j=1,2,...N

[0078] At each decoding step, g j The value is updated as follows according to the following formula:

[0079]

[0080] Where T is the dividing point, a1 and a2 control the growth rate at different stages. Update according to the access status of the node. Once node j is not accessed, the g of node j j The value will increase accordingly. Once the node is covered, its g j The value will be 0 if node j is always ignored and its g j The value will accumulate to a relatively large value, and when it is not covered for more than T steps, its growth rate will further increase. Therefore, the vector contains the coverage status of each node: whether it is covered and the extent to which it is ignored during the solution process. This vector can effectively reflect the coverage information of the subset, thereby guiding the selection of the next subset.

[0081] The decoder's role in this model is to generate a probability distribution for each node being selected, and then select subsets one by one during each decoding step to construct a solution set. At the beginning of the decoding process, the model takes the current partial solution as input, calculates its query value, and measures its match with the key values of all nodes. The query value and node keys of the partial solution are generated by the subset embeddings after the encoder has processed them through multiple layers of Transformers. The core step of the decoder is to calculate the selection probability of each node during each decoding round. This process is implemented through an attention mechanism. The decoder calculates the weight of each node based on the current solution state (i.e., the query value) and the keys of all subsets, and selects the most appropriate subset to visit. To prevent subsets that have already been selected during the decoding process from being selected again, we introduce a masking mechanism. After each node is selected, its selection probability is set to 0, preventing it from being selected again in subsequent decoding steps. This mechanism ensures that the model gradually constructs a valid solution set and avoids falling into local optima.

[0082] The decoder in this paper is based on the attention mechanism. K and V in the decoder are derived from the embedding values of each node obtained by the encoder, while Q is designed based on the problem being solved. The decoder includes an attention calculation and a linear transformation. The attention calculation uses this vector to obtain the corresponding attention weight, which can be used as the output probability distribution.

[0083] In the Transformer model, the core of the attention mechanism is to capture global dependencies by computing the associations (i.e., attention weights) between all pairs of elements in a sequence. However, for combinatorial optimization tasks such as the set cover problem, the score calculation of the traditional attention mechanism has a significant flaw: standard attention learns weights through data-driven learning and does not explicitly exploit the structural characteristics of the set cover problem (such as the overlap and complementarity between subsets). To address this issue, a sparse attention design guided by Jaccard similarity achieves efficient optimization through the following steps. Jaccard similarity, as shown in the formula, is a classic metric for measuring the degree of overlap between two sets, with a value range of [0,1]. In SCP, the Jaccard similarity is calculated between the candidate subset and the covering set of the current state. This calculation has a clear physical meaning: a high Jaccard value (close to 1) indicates that the coverage of the two sets is highly overlapping. Choosing both sets simultaneously would result in redundant coverage, and the model should suppress such redundant decisions. A low Jaccard value (close to 0) indicates that the coverage of the candidate subset is complementary to the current set, and the model tends to choose this set to expand coverage. By introducing Jaccard similarity as a score offset, the coverage relationship between sets can be directly encoded into the attention mechanism, guiding the model to focus on key interactions while filtering out irrelevant noise. The model corresponding to the solver model algorithm is trained using a deep reinforcement learning algorithm, which enables the model to achieve better generalization performance.

[0084] The reinforcement learning algorithm used in the training model of the present invention is the classic REINFORCE algorithm, a policy gradient-based algorithm. Based on the equivalence of node sequence representations, a method for calculating the advantage function based on the mean of a set of solutions is constructed. The equivalence of node sequences in SCP problems means that if the solution to an SCP problem is represented in the form of a node sequence, the same solution can have multiple equivalent representations. For example, for a 5-node SCP problem instance, the sequence {v1, v2, v3, v4, v5} and the sequence {v2, v3, v4, v5, v1} both represent the same solution.

[0085] The present invention has technically implemented the algorithm formulated in the invention purpose section, and used pytorch to implement the model construction and training. The hierarchical deep reinforcement learning algorithm proposed in the present invention can effectively solve the set covering problem, starting from a partial solution represented by an empty sequence, and gradually adding other nodes so that the sequence eventually constitutes a feasible solution to the set covering problem. The algorithm is divided into two layers, in which the upper-layer algorithm uses a modular index to simplify the complete graph of the original problem and decomposes the large-scale set covering problem into multiple sub-problems. The lower-layer algorithm uses a deep reinforcement learning algorithm and a Transformer model, and improves it using dynamic embedding and Jaccard similarity in combination with the characteristics of the set covering problem. In addition, a sparse attention mechanism and a bit compression mechanism are designed to optimize the computational performance of the model and improve the utilization of the GPU. For model training, a corresponding REINFORCE algorithm with a mean benchmark is designed for training, and finally a trained neural network model is obtained to regressively solve the above sub-problems.

[0086] Compared with the Pointerformer algorithm, this method has improved accuracy in the scenario of solving the set cover problem. The Pointerformer algorithm is a state-of-the-art algorithm for solving the TSP problem proposed by Yan et al. in 2023. Here, the model is transferred to the scenario of solving the set cover problem.

[0087] After experiments, the performance data comparison of the model is shown in the table:

[0088] Table 2 Performance of the algorithm on the Random100 set cover problem

[0089]

[0090] Table 3 Performance of the algorithm on the Random1000 set cover problem

[0091]

[0092] The results in the table show that the proposed deep reinforcement learning algorithm surpasses the AM algorithm in terms of solution accuracy. The introduction of the Jaccard similarity attention mechanism and dynamic embedding effectively improves model accuracy. The hierarchical algorithm effectively leverages the advantages of deep learning to solve the set cover problem. In particular, for the LSCP problem, the genetic algorithm cannot solve it within 60 seconds, while the hierarchical deep reinforcement learning algorithm only takes 0.0686 seconds.

[0093] The algorithm for solving the set cover problem based on deep reinforcement learning implemented by the present invention is based on pytorch for model building, training and reasoning. The algorithm designed for the invention content includes a set cover problem solving framework based on deep reinforcement learning, a large-scale set cover solving method based on hierarchical reinforcement learning, and a performance optimization technology based on sparse attention mechanism and compression algorithm. The model is trained using the REINFORCE method. The proposed hierarchical algorithm consists of two parts: an upper-layer decomposition algorithm and a lower-layer solver. The first part is mainly a manually designed algorithm, and the lower layer is a model trained using a deep reinforcement learning algorithm. The graphics card used for training is GeForce GTX 3070Ti (16GB), 8 CPU cores. The hardware configuration of the verification environment is the same as the training environment. The process is divided into the following steps:

[0094] Build a dataset for model training.

[0095] The training and test datasets consist of N nodes randomly generated in a two-dimensional Euclidean space. The coordinates of these nodes follow a two-dimensional uniform distribution.

[0096] Establish spatial associations between nodes and calculate a distance matrix representing the distances between nodes. Select a threshold and calculate the adjacency matrix based on the distance matrix. If the distance between two nodes is less than or equal to the threshold, the corresponding position in the adjacency matrix is assigned a value of 1; otherwise, it is assigned a value of 0.

[0097] Training phase: Build the model according to the model architecture, use NVIDIA GPU to train the model, and save the optimal model weights.

[0098] The model phase uses the optimal model saved during the training phase for inference. Sparse attention and bit compression mechanisms are used to optimize the inference process and reduce inference latency. For large-scale problems, a hierarchical algorithm is required to decompose the problem into subproblems, use the model to solve the subproblems, and then fuse the subproblems as the solution to the original problem.

[0099] While some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. An intelligent scheduling method based on deep reinforcement learning to solve the set coverage problem. The method takes the current distribution data of express delivery stations as input and first constructs the problem task: selecting the minimum number of nodes from the express delivery stations that the warehouse can cover within the maximum service distance. A specified number of nodes are selected from the express delivery stations as the maximum number of stations that the warehouse can cover within the maximum service distance; On this basis, the express station distribution data is modeled as graph data, with express stations as nodes and a maximum service distance S set. If the distance between two nodes does not exceed S, there is an edge between the two stations. The problem input by the problem decomposition module is decomposed into multiple sub-problems. The problem decomposition module divides all subsets into multiple groups to ensure that the subsets between groups are correlated and the correlation between subsets in different groups is relatively small; and sends each sub-problem to each solution module for solution. The solution module solves the problem by constructing a trained deep reinforcement learning model. Finally, the solutions obtained by each solution module are input into the global optimization module for fusion stage, and the solutions of the sub-problems are merged to obtain the optimal layout of the express station distribution.

2. The method for solving the set covering problem based on deep reinforcement learning according to claim 1, characterized in that: The specific method for ensuring that the subsets between groups are associated and the subsets of different groups are relatively less associated is: i As a node, when S i ∩S j When it is not equal to 0, establish edge e ij , S i ∩S j The value of the node is recorded as the weight between the nodes as w ij , calculate S i With S j The expected value of the weight Q ij , calculate the difference between the two and record it as : Where δ(s i ,s j ) indicates whether the two sets are in the same group. In the initial stage, the value is 0; if they are in the same group, the value is accumulated, otherwise no calculation is performed; at the time of initialization, each set is treated as an independent group G k =S k , for each subset node S i , calculate its move to group G j When the gain is j Abstracted as a super node, the expected value of the weight is recalculated. After multiple iterations, until Δq does not change much, that is, the grouping is stable, the subset partitioning structure will eventually be obtained, and each group will be regarded as a sub-problem of set coverage.

3. The method for solving the set covering problem based on deep reinforcement learning according to claim 2, characterized in that: The deep reinforcement learning model of the solution module is a set covering problem solution algorithm based on deep reinforcement learning. Its specific implementation method is as follows: first, the solution process of the set covering problem is modeled as a Markov decision process; the set covering problem is modeled as a Markov process by defining appropriate state space, action space, reward function and state transition rules. The reward function is specifically constructed as follows: The current state is S t , action is a t , the subset is u j , the new state is S t+1 , the node space is U t+1 ; Afterwards, the network model is designed following the encoder-decoder pattern, where the encoder's embedding features consist of two parts: static embedding and dynamic embedding. For the static embedding, centrality encoding and side information encoding are introduced. The set cover problem is represented as a graph structure, where nodes represent elements in the set and edges represent the coverage relationship between nodes. The centrality encoding method introduces node importance information into the model by measuring the coverage capacity of the nodes; The dynamic embedding updates the coverage status of the node in each decoding step, so that the model can better understand the role of the node in the current solution set. The specific implementation method is: define a vector, the vector g of each node j j represents the coverage status of the node, and g j Dynamic update is performed in each decoding step; suppose that in decoding step t, for subset i, C(π t ) is defined as the set of nodes covered by i, and the sum of g of the nodes in C(i) is calculated. j value; At the beginning of the decoding process, the g of all nodes j The value is initialized to 1 and becomes 0 when overwritten: g j =1,j=1,2,...N; in each decoding step, g j The value is updated as follows according to the following formula: Where T is the dividing point, a1 and a2 control the growth rate at different stages, and are updated according to the access status of the node. Once node j is not accessed, the g of node j j The value will increase accordingly. Once the node is covered, its g j The value will be 0 if node j is always ignored and its g j The value will accumulate to a relatively large value, and when it is not covered for more than T steps, its growth rate will further increase. Therefore, this vector contains the coverage status of each node: whether it is covered and the degree to which it is ignored during the solution process. This vector can effectively reflect the coverage information of the subset, thereby guiding the selection of the next subset; The role of the decoder is to generate a probability distribution for each node being selected, and then select subsets one by one in each decoding process to construct a solution set; at the beginning of the decoding process, the model takes the current partial solution as input, calculates its query value, and calculates the degree of matching with the key values of all nodes. The query value of the partial solution and the key value of the node are generated by the subset embedding after the encoder is processed by multiple layers of Transformer; in order to prevent the subset that has been selected from being selected again during the decoding process, a mask mechanism is introduced. Whenever a node is selected, its selection probability will be set to 0, thereby avoiding the selection of the selected node again in the subsequent decoding steps.

4. The method for solving the set covering problem based on deep reinforcement learning according to claim 3, characterized in that: The global optimization module merges the solution sets of the sub-problems obtained from the lower-level problems, and inputs them as simplified problems in the original problem into the deep reinforcement model solver for solving, thereby obtaining the final solution to the problem.

Images