Multi-objective reinforcement learning intelligent storage path optimization method based on preference

Abstract

The invention discloses a preference-based multi-objective reinforcement learning intelligent storage path optimization method, which comprises the following steps of: firstly, establishing a row-column cross storage layout scheme in a partition mode, and secondly, establishing an inter-node distance calculation model; then, establishing a multi-target capacity constraint intelligent storage path optimization mathematical model taking the minimization of the total transportation distance and the minimization of the maximum transportation distance as targets; then solving the established model by a preference-based multi-target reinforcement learning algorithm, taking a directed graph as input, performing node embedding through an attention encoder, performing context embedding of nodes through a preference-based attention decoder, and performing variation through mutation operation based on an evolutionary genetic search algorithm; and finally, learning an optimal preference condition strategy by using a reinforcement algorithm to generate a solution. The method not only can effectively reduce the transportation conflict probability and reduce the vehicle transportation distance, but also can effectively balance the conflict between targets, and generates a dense tradeoff approximate Pareto solution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent warehouse path optimization, in particular to a multi-objective reinforcement learning intelligent warehouse path optimization method based on preferences. BACKGROUND

[0002] Intelligent warehouse refers to a warehouse management system that uses advanced information technology and automation technology to improve warehouse management and logistics operation efficiency. Intelligent warehouse upgrades traditional warehouse systems to intelligent and automated modern warehouse systems with the help of computer science, artificial intelligence, Internet of Things, robots and other technologies, to improve the efficiency, accuracy and flexibility of warehouse operations, and reduce warehouse costs. In the field of intelligent warehouse, the transportation efficiency of the warehouse will directly affect the daily operation and long-term development of the enterprise, and how to improve the intelligent warehouse path optimization algorithm and technology to improve the transportation efficiency and accuracy of goods in the warehouse is a problem to be solved at present. At the same time, with the continuous development and maturity of automation technology, automatic navigation equipment and robots are increasingly widely used in the field of warehousing; big data and artificial intelligence technology are also widely used in the field of intelligent warehouse path planning.

[0003] The prototype of vehicle routing problem (VRP) is how to deliver gasoline to gas stations from the port with the minimum driving distance to meet the needs of these gas stations, which was proposed by Dantzig and Ramse in 1959. VRP refers to the process of transporting goods from a distribution center to demand points with different demand amounts of goods under various constraints, using vehicles, and planning vehicle distribution routes reasonably to optimize the target. It is defined by network, demand, vehicle, cost and target. As a classic combinatorial optimization problem, more VRP variants have been proposed by researchers as the research deepens, such as VRP with time window, multi-distribution center VRP, multi-objective optimization VRP, etc. Since VRP is an NP-hard combinatorial optimization problem, it has led to many solution methods, which can be divided into three categories: exact algorithms, heuristic algorithms and machine learning algorithms.

[0004] Multi-objective combinatorial optimization (MOCO) refers to considering multiple objectives in an optimization problem, and finding an optimal solution by reasonably combining the optimization results of different objectives. In practical problems, multiple objectives often need to be optimized, and these objectives have conflicting or interdependent relationships. The mathematical definition of a minimization multi-objective optimization problem is as follows:

[0005]

[0006] where, is the number of objective functions; is the number of decision variables; is the decision variable vector; is the feasible solution space, represents the entire decision space; is the objective vector. For , if all constraints are satisfied, is a feasible solution, and all feasible solutions constitute the feasible solution set; assume are two feasible solutions in the decision space, for , both hold, and such that holds, then Pareto Dominance is denoted as , and is called a non-dominated solution in Pareto, is called a dominated solution in Pareto. Assume , and such that Pareto Dominance , then is called a Pareto Optimal Solution, and the solution set composed of all Pareto Optimal Solutions is called the Pareto Optimal Solution Set (PS). The Pareto Optimal Solution Set is mapped to the objective space by the function , and the set of objective vectors obtained in the objective space is called the Pareto Front (PF), i.e., the Pareto Front is the image of the Pareto Optimal Solution Set in the objective space, i.e., .

[0007] The existing warehouse path planning does not consider the diversification and complexity of the warehouse transportation process, and the model established is usually single objective, which has certain distance with the actual situation. In addition, the exact method and approximate method used by some researchers to find the entire Pareto frontier is not realistic in calculation, and the existing heuristic-based method and learning-based method are basically to find a small subset of the approximate Pareto frontier, and the decision maker can only select the solution from this small set, and the set does not necessarily contain the preferred solution; in addition, it is difficult for the traditional heuristic algorithm to adapt to the dynamic warehouse logistics system. For the multi-objective path planning established by part of the research, when dealing with multi-objective problems, it is only through simple weighted aggregation to convert the large-scale multi-objective problem into a single objective problem for solving, and this method is usually biased, so it is difficult to achieve the balance between objectives, resulting in a suboptimal solution. Therefore, the establishment of a preferred multi-objective reinforcement learning intelligent warehouse path optimization method can not only theoretically expand the research scope of intelligent warehouse and path planning, provide a selectable theoretical framework, but also reduce the warehouse logistics transportation cost, improve the warehouse operation efficiency, and promote the development of intelligent warehouse. SUMMARY

[0008] In view of the above technical deficiencies, the present application aims to solve the above-mentioned problems in the current research, and proposes a row-column intersection warehouse layout scheme, which is used to establish a single-direction mode intelligent transportation vehicle path planning network directed graph, and by means of multi-objective combination optimization technology, a multi-objective capacity constrained intelligent warehouse path optimization (MOCIWRO) mathematical model is established, and then a preferred multi-objective reinforcement learning method (P-MORL) is proposed to learn an optimal preference condition strategy, and a path scheme with the lowest proportion of targets is constructed for each preference, which effectively approximates the Pareto optimal frontier, so that the decision maker can obtain the preferred preference-based solution.

[0009] The application adopts the following technical scheme: a preferred multi-objective reinforcement learning intelligent warehouse path optimization method, comprising the following steps: S1: constructing a row-column intersection warehouse layout scheme, establishing a node distance calculation model based on partition and a single-direction mode intelligent transportation vehicle path planning network directed graph; S2: establishing two objective functions of minimizing total transportation distance and minimizing maximum transportation distance, considering the constraint conditions, and establishing a multi-objective capacity constrained intelligent warehouse path optimization mathematical model; S3: solving the model in step S2 based on a multi-objective reinforcement learning algorithm based on decision maker preferences, which takes a path network directed graph as input, performs node embedding through a preference-agnostic attention encoder; S4: decoding the data of step S3 based on a preference-based attention decoder, which first performs context embedding of nodes, then calculates the probability of selecting the next node, and finally uses a multi-layer perceptron to generate decoder parameters conditioned on preferences through mutation operations based on evolutionary genetic search algorithms; S5: performing reinforcement learning on the data of step S4 based on a multi-objective reinforcement algorithm under the condition of decision maker preferences, first constructing a target function based on preference and weighted Chebyshev aggregation to realize dimension reduction of multiple objectives, then using a reinforcement algorithm to estimate the gradient of the preference condition cost to generate learnable model parameters, and finally generating a solution that effectively approximates the Pareto optimal frontier.

[0010] As a kind of based on the multi-objective reinforcement learning intelligent warehousing path optimization method of preference of the present application, wherein: the row-column intersection warehousing layout scheme in step S1 is described as two-dimensional coding using grid in warehousing, to represent the position of storage node, and the minimum unit of grid is square with side length of 1;Establish a kind of based on partition mode expression as connecting left top corner and right top corner with bottom midpoint respectively, divide the storage area into left , right , right three storage areas, and the picking point is located directly below the warehouse;The steps to determine the area where the storage node is located are as follows: Step 1: given node position coordinates , , is the total number of horizontal grids, and is specified as an odd number, is the total number of vertical grids; Step 2: if the coordinate position satisfies , then the node is located in the left lower area, denoted as ; Step 3: if the coordinate satisfies and , then the node is located in the upper area, denoted as ; Step 4: if the coordinate position satisfies , then the node is located in the right lower area, denoted as .

[0011] As a kind of based on the multi-objective reinforcement learning intelligent warehousing path optimization method of preference of the present application, wherein: the distance calculation model between nodes in step S1 calculates the distance between storage nodes In the case of a row-column intersection warehouse layout scheme, the calculation steps are as follows: Step 1: Given two nodes and ; Step 2: If the coordinate position satisfies , then , where if , then , otherwise ; Step 3: If the coordinate position satisfies , then , where if , then , otherwise ; Step 4: If the coordinate position satisfies , then , where if , then , otherwise ; Step 5: If the coordinate position satisfies , then ; Step 6: If the coordinate satisfies , then ; Step 7: If the coordinate position satisfies , then ; Step 8: If the coordinate position satisfies , then .

[0012] As a preferred multi-objective reinforcement learning intelligent warehouse path optimization method of the present application, in step S1, the distance calculation model between the picking point and the storage node is described as the distance between the picking point to the first and last storage node . The calculation formula is as follows: .

[0013] As a preferred multi-objective reinforcement learning intelligent warehouse path optimization method of the present application, in step S2, the multi-objective capacity constraint intelligent warehouse path optimization mathematical model objective function includes: The first objective function minimizes the total transportation distance is expressed as: , The second objective function minimizes the maximum transportation distance​ Expressed as: , in, is a 0-1 decision variable, representing the vehicle Whether it is stored by the node Driving towards , is 1, otherwise 0, ; Indicates the number of storage nodes; Indicates the number of intelligent transport vehicles; Representation node the distance between them; 、 They represent the distance from the pickup point to the first and last storage nodes of the path respectively; the second objective function, minimizing the maximum transportation distance, means minimizing the maximum value of the transportation distance of all vehicles while satisfying the storage capacity of all nodes.

[0014] As a preference-based multi-objective reinforcement learning intelligent warehousing path optimization method of the present invention, the preference description in step S3 is a vector of the decision maker's relative preference for multiple optimization objectives, and the preference is a set of scalarized vectors that satisfy and ,in is the target number, It is the Preference of a target.

[0015] As a preference-based multi-objective reinforcement learning intelligent warehousing path optimization method of the present invention, wherein: the multi-objective reinforcement learning algorithm based on decision maker preference in step S3 includes the following steps: Step 1: First, the algorithm takes the path network directed graph as input, which is described as follows: Intelligent transportation vehicle path planning network directed graph uses To indicate that Represents a collection of storage nodes Meet at the pickup point The union of ; Represents a set of road segments, representing the connection between two nodes; Step 2: The algorithm then performs node embedding via a preference-agnostic attention encoder, which is described as follows: The preference-agnostic attention encoder takes a question instance as input and outputs a set of Embedding of dimensional nodes , expressed as ,in and represents the learnable parameters, Represents node attributes, including node location and storage capacity, that is, ; Step 3: Then the algorithm decodes by a preference-based attention decoder, described as: the preference-based attention decoder takes the embeddings of all nodes as input, and selects the next node with probability ; the algorithm defines a preference-conditioned random policy parameterized by , described as , where represents a preference-conditioned random policy; denotes the path that has been selected; Step 4: Then the algorithm performs the context embedding of the node, described as: the context embedding of the node is in time step , the decoder first constructs a context embedding from the first selected node and the last selected node , secondly, the matrix projects the concatenated -dimensional vector onto a -dimensional vector, and then further aggregates the context embedding with the embeddings of all nodes through a multi-head attention model, the expression of the context embedding is described as: where , , are the query, key and value of the multi-head attention model, respectively; , , are the three projection matrices; denotes the parameters of the multi-head attention model; Step 5: Then the algorithm calculates the probability of selecting the next node, described as: the node probability is calculated using the embedding of the node, denoted as where is the embedding dimension; is a hyperparameter; denotes the transpose of the vector ; denotes the nodes that have been visited; denotes the nodes that have been selected and have no influence on the selection of subsequent nodes; is the hyperbolic tangent function; all the nodes that have been visited are set to and will not be selected as the next node; the probability of selecting the node at time step is calculated through the normalized exponential function, denoted as where the numerator is the node of The exponent of the denominator is the total number of nodes The exponential sum of is a natural constant; Step 6: The algorithm then mutates through a mutation operation based on the evolutionary genetic search algorithm, which is described as follows: The mutation operation based on the evolutionary genetic search algorithm includes a set of four random mutation operators and two crossover mutation operations; Step 7: The algorithm then uses a multilayer perceptron to generate decoder parameters conditioned on preferences, as described below: The input to the multilayer perceptron is a dimensional preference vector and model parameters , the multilayer perceptron first generates a hidden embedding , and then by linear projection Map the hidden embeddings into decoder parameters; where is a multi-layer perceptron model with parameters and parameter matrix 、 is a learnable parameter; Step 8: The algorithm then uses multi-objective reinforcement under preference conditions to learn the data, which is described as: Establishing the objective function is described as For a path problem, the objective function is a function containing vectors of different targets, i.e. , based on preference The formula for dimensionality reduction using weighted Chebyshev aggregation is described as ,in It is The ideal value of a goal; For a given instance , the goal is to minimize the expected goal of all preferences, described as ,in represents the objective function, represents the model parameters, is the vector of all possible preferences A collection of Indicates that from the distribution Sampling in , Indicates that the strategy Sampling in , Indicates the expected operation on the random variable; in order to train the model, it is necessary to select from different sets of instances in each iteration Random sampling examples , and define the training loss as ; Step 9: Finally, the algorithm uses reinforcement algorithm to estimate the gradient of the preference condition cost, described as: Estimate the gradient of the preference condition cost is for a given instance and preference , using reinforcement algorithm to estimate, expressed as , where, denotes the gradient operator; denotes the advantage function; is the baseline to reduce the gradient variance; in each update step, randomly select each combination of preference , instance and different routes , give the approximate gradient as: ; where, for each update step, use the same shared baseline estimation function for each combination, then randomly select the starting node of each route to push out different path selection.

[0016] As a preferred multi-objective reinforcement learning intelligent warehouse path optimization method based on the application, the mutation operation based on evolutionary genetic search algorithm in step S4 is described as applying a set of four random mutation operators and two crossover mutation operations. Four random mutation operators include the following operation steps: Step 1: Random node exchange mutation, described as randomly selecting two nodes from the selected routes without violating the capacity constraint, generating a new route, and updating the route capacity, expressed as ; Step 2: Random node transfer mutation, described as transferring a randomly selected node from one route to another route as the last node, expressed as ; Step 3: Random arc exchange mutation, described as following the same process as random node exchange mutation, except that adjacent nodes within the route, i.e. arc nodes, are randomly selected for exchange, expressed as ; Step 4: Random arc transfer mutation, described as following the same process as random node transfer mutation, but instead of randomly selecting a single node, adjacent nodes within the path, i.e. arc nodes, are randomly selected for transfer as the last node, expressed as .

[0017] As a preference-based multi-objective reinforcement learning intelligent warehousing path optimization method of the present invention, the two crossover mutation operations of the mutation operation based on the evolutionary genetic search algorithm in step S4 include heuristic inheritance crossover (HIC) and random inheritance crossover (RIC), and the operation steps are as follows: Step 1: Given two parent nodes , and satisfies ; Step 2: Calculation , and get the two minimum values and Corresponding routes and ; Random selection ; Step 3: Find The route with the most identical nodes and ; Step 4: Perform sub-route inheritance operations: , ; Step 5: Calculate floating nodes: , ; Step 6: 、 Randomly join 、 At the end of the middle route, a new route is formed, and the remaining floating nodes are added as new routes. 、 , thus forming a new sub-route , .

[0018] The beneficial effects of the present application are that: compared with the traditional warehouse layout scheme, the row-column intersection warehouse layout scheme provided by the present application can more effectively reduce the transportation conflict probability and make the vehicle transportation path shorter; the established intelligent warehouse path optimization model considers multiple objectives, conforms to the actual situation, and uses weighted Chebyshev aggregation for multi-objective dimension reduction, effectively balancing the conflicts of each objective; considering the relative preference of the decision maker for the optimization objective, the preference is added to the multi-objective dimension reduction and reinforcement learning solving process, which can handle actual problems such as different transportation tasks and different goods; combining heuristic methods and machine learning methods, on the basis of mainly using reinforcement learning methods, combining heuristic ideas based on mutation operation of evolutionary genetic search algorithm, realizing the diversity of solutions and expanding the search space span, which can accelerate the convergence speed of the model; with the help of multi-objective reinforcement learning method, by establishing a preference condition random strategy model to adapt to all preferences, an optimal preference condition strategy is learned, a path scheme with the lowest proportion of objectives is constructed for each preference, a balance is found between objectives, dense approximate Pareto optimal solutions can be generated, the Pareto optimal frontier can be effectively approximated, more trade-off solutions can be generated, and the decision maker can obtain a preference-based intelligent warehouse path solution. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0020] Figure 1 The overall flowchart of a preference-based multi-objective reinforcement learning intelligent warehouse path optimization method provided for the first embodiment of the present application; Figure 2 The intelligent warehouse row-column intersection layout scheme in the preference-based multi-objective reinforcement learning intelligent warehouse path optimization method provided for the first embodiment of the present application; Figure 3 The multi-objective reinforcement learning algorithm model architecture diagram in the preference-based multi-objective reinforcement learning intelligent warehouse path optimization method provided for the first embodiment of the present application; Figure 4 The four random mutation operators and two cross variation operation schematic diagrams of the evolutionary genetic search algorithm in the preference-based multi-objective reinforcement learning intelligent warehouse path optimization method provided for the first embodiment of the present application; Figure 5A model training flowchart of a preference-based multi-objective reinforcement learning algorithm in a preference-based multi-objective reinforcement learning intelligent warehousing path optimization method provided by the second embodiment of the present invention; Figure 6 A node distribution diagram of a specific test case of intelligent warehousing in a preference-based multi-objective reinforcement learning intelligent warehousing path optimization method provided by the second embodiment of the present invention; Figure 7 A comparison chart of the loss value changes in the training loss function and the test set Pareto frontier chart of the P-MORL algorithm in a preference-based multi-objective reinforcement learning intelligent warehouse path optimization method provided in the second embodiment of the present invention, with and without four random mutation operators and two crossover mutation operations based on an evolutionary genetic search algorithm; Figure 8 A solution path diagram for a specific test example of intelligent warehousing in a preference-based multi-objective reinforcement learning intelligent warehousing path optimization method provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0022] Example 1 Reference Figures 1 to 4 , as one embodiment of the present invention, provides a preference-based multi-objective reinforcement learning intelligent warehousing path optimization method, comprising: S1: Construct a row-column cross-warehousing layout scheme, establish a partition-based node distance calculation model and a one-way mode intelligent transportation vehicle path planning network directed graph; For further information, see Figure 2 The row and column cross storage layout scheme is described as a two-dimensional encoding method for the storage using a grid to represent the location of the storage node. The smallest unit of the grid is a square with a side length of 1. The bottom midpoint is connected to the left top corner and the right top corner, dividing the storage area into the lower left corner and the lower right corner. , directly above , lower right There are three storage areas, and the pickup point is located directly below the warehouse; the steps to determine the area where the storage node is located are as follows: Step 1: Given a node Location coordinates , , is the total number of horizontal grids, and is specified to be an odd number. is the total number of vertical grids; Step 2: If the coordinate position satisfies , then the node is located in the lower left area, recorded as ; Step 3: If the coordinate position satisfies and , then the node is located in the upper region, recorded as ; Step 4: If the coordinate position satisfies , then the node is located in the lower right area, recorded as .

[0023] For further information, see Figure 2 , the distance between storage nodes In the case of a storage layout solution based on rows and columns crossing, the calculation steps are as follows: Step 1: Given two nodes and ; Step 2: If the coordinate position satisfies ,but , among which if ,but ,otherwise ; Step 3: If the coordinates satisfy ,but , among which if ,but ,otherwise ; Step 4: If the coordinates satisfy ,but , among which if ,but ,otherwise ; Step 5: If the coordinate position satisfies ,but ; Step 6: If the coordinate position satisfies ,but ; Step 7: If the coordinate position satisfies ,but ; Step 8: If the coordinates satisfy ,but .

[0024] For further information, see Figure 2 , pickup point To the first and last storage nodes distance and The calculation formula is the same, and the calculation steps are as follows: Step 1: Specify the pickup point and head (tail) storage nodes ; Step 2: if , then ; Step 3: if , then ; Step 4: if , then ; Step 5: if , then .

[0025] S2: Establish two objective functions of minimizing total transportation distance and minimizing maximum transportation distance, consider the constraint conditions, and establish a multi-objective capacity-constrained intelligent warehouse routing optimization (MOCIWRO) mathematical model; Further, the objective functions of the established MOCIWRO mathematical model include: The first objective function minimizes the total transportation distance is expressed as: , The second objective function minimizes the maximum transportation distance is expressed as: , wherein, is a 0-1 decision variable, indicating whether the vehicle is driven from the storage node to , which is 1, otherwise 0, ; represents the number of storage nodes; represents the number of intelligent transportation vehicles; represents the distance between nodes ; , respectively represent the distance from the pickup point to the head and tail storage nodes; the second objective function of minimizing the maximum transportation distance means that under the condition of meeting all node storage amounts, the maximum value in the transportation distance of all vehicles is minimized.

[0026] Further, the decision variables of the established MOCIWRO mathematical model include: Decision variable 1: is expressed as: ; Decision variable 2: is expressed as: .

[0027] Further, the constraint conditions of the established MOCIWRO mathematical model include: Condition 1: satisfying the maximum load and the storage amount of the storage node, and being expressed as: , ; Condition 2: each storage node must be served by a vehicle, and being expressed as: ; Condition 3: there is only one vehicle arriving at a storage node, and being expressed as: ; Condition 4: there is only one vehicle leaving a storage node, and being expressed as: .

[0028] Wherein represents a set of storage nodes; represents a set of intelligent transportation vehicles; represents the number of each vehicle; represents the number of each node; represents the storage amount of the storage node ; represents the maximum load of the intelligent transportation vehicle.

[0029] It should be noted that the problem description of the MOCIWRO mathematical model established by the present application is as follows: in a certain intelligent warehouse, there is a pickup point D providing transportation services for multiple storage nodes. The pickup point makes decisions according to the storage information and intelligent transportation vehicle information of the storage nodes, etc., so that the vehicle transports goods from the pickup point to the storage nodes, and the storage information must be satisfied, the vehicle load must not exceed the maximum load, and the vehicle must return to the pickup point after completing transportation and not perform transportation again. Limitations such as restrictions. It is required to reasonably arrange the driving route to achieve the goals of minimizing the total transportation distance and minimizing the maximum transportation distance under the condition of satisfying all constraints. In addition, it should be noted that the model established by the present application belongs to the single vehicle depot path problem without time window, with capacity constraint and vehicle closed. The reason for considering the second objective function, i.e., minimizing the maximum transportation distance, is that if the transportation path is too long, it will easily lead to warehouse congestion and unable to complete the transportation task on time.

[0030] S3: proposing a multi-objective reinforcement learning algorithm based on decision maker's preference to solve the model in step S2, which takes the path network directed graph as input, and performs node embedding through a preference-agnostic attention encoder; Further, refer to Figure 3, the preference in the proposed multi-objective reinforcement learning algorithm based on decision maker's preference (P-MORL) refers to: a vector of the relative preference degree of the decision maker to multiple optimization objectives, the preference is a set of scalarized vectors, satisfying and , wherein is the number of objectives, is the preference for the th objective.

[0031] Further, referring to Figure 3 , the directed graph of the intelligent transportation vehicle path planning network in the proposed P-MORL algorithm refers to: the directed graph uses to represent, wherein represents the union of the node set and the pickup point set , that is ; represents the road segment set, each road segment represents the connection between two vertices in the node set , with distance, time, etc.

[0032] It should be noted that each pickup point has a corresponding number of intelligent transportation vehicle sets , each vehicle has its own maximum load, maximum transportation distance, etc.

[0033] Further, referring to Figure 3 , the node embedding in the proposed P-MORL algorithm refers to: the preference-agnostic attention encoder takes a problem instance as input, and outputs a set of dimensional node embeddings for each node, represented as , wherein and represent learnable parameters, represent node attributes, including node location and storage capacity, that is .

[0034] S4: propose a preference-based attention decoder to decode the data of step S3, first perform node context embedding, then calculate the probability of selecting the next node, then use a mutation operation based on evolutionary genetic search algorithm, and finally use a multilayer perceptron to generate decoder parameters conditioned on preference; Further, referring to Figure 3 , the preference-based attention decoder in the P-MORL algorithm refers to: the parameters of the decoder are conditioned on the preference To condition, an attention model is used as the base encoder-decoder model, the main model structure is a multi-head attention layer, and the input is the embedding of all nodes, and the next node is selected in turn with a probability of . The P-MORL algorithm defines a preference-conditioned random strategy parameterized by : , where represents a preference-conditioned random strategy for generating an effective transport sequence according to the given preference parameters ; is the probability of selecting as the next node given and the previous selection .

[0035] Further, referring to Figure 3 , the context embedding of the node in the proposed P-MORL algorithm refers to: in the time step , the decoder first constructs a context embedding from the first selected node and the last selected node , then the matrix projects the connected dimensional vector to a dimensional vector, and then through the multi-head attention model, further aggregates the context embedding with the embedding of all nodes , the expression of the context embedding is: where , , are the query, key and value of the multi-head attention model respectively; , , are the three projection matrices; denotes the parameters of the multi-head attention model.

[0036] Further, referring to Figure 3 , the calculation of the probability of selecting the next node in the proposed P-MORL algorithm refers to: using the embedding of each node to calculate the probability of selecting the next node, the expression is: , where is the embedding dimension; is a hyperparameter; denotes the transpose of the vector . Indicates the nodes that have been visited; Indicates that the selected node has no effect on the subsequent node selection; is the hyperbolic tangent function; all nodes that have been visited are set to , will not be selected as the next node. When selecting The probability of a node is calculated by the normalized exponential function: , The numerator is Nodes The exponent of the denominator is the total number of nodes The exponential sum of is a natural constant; For further information, see Figure 4 ,The mutation operation based on the evolutionary genetic search algorithm in the proposed P-MORL ,algorithm refers to : applying a set of four random mutation operators and two ,crossover mutation operations to achieve this function.

[0037] Among them, the four random mutation operators include the following operation steps: Step 1: Random node exchange mutation, described as randomly selecting two nodes from the selected route to exchange without violating the capacity constraint, generating a new route, and updating the routing capacity, expressed as ; Step 2: Random node transfer mutation, described as transferring a randomly selected node from one route to another as the end node, represented as ; Step 3: Random arc exchange mutation, described as following the same process as random node exchange mutation, except that adjacent nodes within the route, i.e. arc nodes, are randomly selected for exchange, expressed as ; Step 4: Random arc transfer mutation, which is described as following the same process as random node transfer mutation, but instead of randomly selecting a single node, it randomly selects adjacent nodes in the path, i.e., arc nodes, to transfer as the end nodes, which is expressed as .

[0038] Among them, the two crossover mutation operations include the following steps: (1) Heuristic inheritance crossover: It is a deterministic crossover operator that first selects the routing set from the Select two groups of routes 、 As the parent routing group, then select the two best routes from the parent routing and inherit them to the child routing , The best route is defined as the route with the minimum value of the maximum load weight minus the total storage of the route. Then, in the parent route, the route with the most number of path overlaps is selected, the inheritance to the child route is exchanged respectively, and then the node set of the parent route is used to subtract the nodes of the child route after the inheritance nodes to obtain the nodes that do not exist in the child route, which are called floating nodes. Then, the floating nodes are randomly added to the end of each route of the child route after the inheritance nodes to form new routes, and the remaining floating nodes are added to the child route as new routes, thereby forming new child routes; (2) random inheritance crossover: the same process as the heuristic inheritance crossover is followed, and the difference is that the two routes inherited from the parents are randomly selected, rather than selecting the two best routes.

[0039] It should be noted that the two crossover mutation operations include the following operation steps: Step 1: Given two parent nodes , and satisfy ; Step 2: Calculate , get the two minimum values and corresponding to the routes and ; randomly select ; Step 3: Find the routes and with the most same nodes in ; Step 4: Perform child route inheritance operation: , ; Step 5: Calculate the floating nodes: , ; Step 6: Randomly add , to the end of the route in , to form new routes, and the remaining floating nodes are added to , as new routes, thereby forming new child routes , .

[0040] Further, referring to Figure 3 , the use of a multi-layer perceptron to generate a preference-conditioned decoder in the P-MORL algorithm refers to: the algorithm uses a simple multi-layer perceptron to generate preference-conditioned decoder parameters, and the input is a dimensional preference vector and model parameters , the multilayer perceptron first generates a hidden embedding , and then maps the hidden embedding to the decoder parameters through a linear projection . Here is the multilayer perceptron model, and the parameters and the parameter matrix , are learnable parameters.

[0041] S5: The multi-objective reinforcement algorithm under the preference condition is proposed to perform reinforcement learning on the data of step S4. First, a target function based on the preference and weighted Tchebycheff aggregation is constructed to realize the dimension reduction of the multi-objective, and then a reinforcement algorithm is used to estimate the gradient of the preference condition cost to generate learnable model parameters. Finally, a solution that effectively approximates the Pareto optimal frontier is generated.

[0042] Further, referring to Figure 3 , the weighted Tchebycheff aggregation in the P-MORL algorithm refers to: Weighted-Tchebycheff (Weighted-TCH) is a multi-objective combination optimization strategy, which is a weighted distance-based target function transformation method, aiming to find the best compromise solution among multiple conflicting objectives. It is an extension of the traditional Tchebycheff aggregation method, which allows the objectives to be weighted according to their relative importance. In the sub-problem associated with the scalarization preference considered by the present application, the Weighted-TCH aggregation function to be minimized is expressed as follows: , where is the ideal value of , that is, for all solutions , , there is , which is the ideal value of the function , that is, among all feasible solutions, , the minimum value is obtained. is the feasible solution space. is the value of the th objective function. In this function, first, the absolute value of the deviation on each objective function is calculated, that is, , and then multiplied by the scalar value of the preference.

[0043] It should be noted that the purpose of the Weighted-TCH aggregation function is to measure the gap between the solution and the ideal value of the objective function to evaluate the quality of the solution. The smaller the Weighted-TCH aggregation function value of the solution , the better the solution The smaller the distance between the solution and the ideal value of the objective function, the better the solution Therefore, if we want to ensure that all objective functions are close to the optimal solution, we need to minimize the solution that is farthest from the ideal value of each objective. So we first solve the solution that is farthest from the ideal value of each objective, that is, , and then minimize these solutions, that is, Minimizing the weighted distance can make the Pareto optimal solution found in the solution process more balanced, rather than biased towards a specific objective function.

[0044] For further information, see Figure 3 , the establishment of the objective function in the P-MORL algorithm means: For the MOCIWRO problem, the objective function is a vector containing two different objectives, namely , based on preference The formula for dimensionality reduction using Weighted-TCH is: .

[0045] For a given instance , the goal is to minimize the expected target of all preferences, which is expressed as:

[0046] in, represents the objective function, represents the model parameters, is the set of all possible preference vectors uniformly distributed over all valid preferences A collection of Represents a distribution Sampling in , From a parameter and given preferences Strategy Sampling in , Indicates the expected operation on the random variable. In order to train the model, it is necessary to select from different sets of instances in each iteration. Random sampling examples , and define the training loss as .

[0047] Furthermore, the multi-objective reinforcement in the P-MORL algorithm means: for a given instance and specific preferences , the model uses a reinforcement algorithm to estimate the gradient of the preference conditional scalar cost, which is expressed as: ; in, represents the gradient operator, For fixed and , for the objective function with respect to the parameter To solve the gradient of ; is the advantage function in the reinforcement algorithm; is a baseline for the expected cost of reducing the gradient variance. At each update step, we randomly draw Combined Preferences 、 Instances and Different routes , the approximate gradient expression is given as: ; For each update step, Combined use of the same shared baseline estimation function , and then randomly select each route The starting node of the network is used to introduce different path choices.

[0048] Example 2 Reference Figures 5 to 8 , is an embodiment of the present invention, which provides a preference-based multi-objective reinforcement learning intelligent warehousing path optimization method. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through model training and testing and comparative experiments. Running environment: On Windows 10 system, use PyTorch 2.0.0+cpu and Python 3.9.6, and perform model training and testing in PyCharm 2022.1.

[0049] Parameter setting: Fixed the size of the row and column cross layout of the intelligent storage .sampling preferences, one of which is The horizontal coordinate of the storage node position is from the interval Random sampling, the vertical axis is from the interval Random sampling; the coordinates of the pickup point are ; Then normalize the coordinate position, that is, divide the horizontal and vertical coordinates by and ; Storage node storage capacity from the collection Randomly extract from the , and divide the result by the scaling factor 30 to get the discrete storage amount ; Maximum load capacity of intelligent transport vehicles .

[0050] Training example: When using the P-MORL algorithm proposed in this invention to train the MOCIWRO mathematical model, CPU is used for training, and a total of 300 epochs are set. In each epoch, 100,000 MOCIWRO problem instances are randomly generated; sampling Problem examples and nodes to calculate the policy gradient and set ,Right now , , so that each time the training is performed with 64 instances of data, and Other parameters are set as follows: node embedding dimension is 128, the number of encoder layers is 6, the dimensions of query, key and value vectors of the multi-head attention model are 16, the number of attention mechanism heads is 8, Hyperparameters The number of hidden layers is 10, the hidden layer dimension is 512, the softmax prediction method is selected during model training, and the Adam optimizer is used. The base learning rate is 1e-4, the weight decay is 1e-5, the learning rate scheduler is StepLR, the episode parameter of a single epoch is 100*100, and the batch processing parameter is 64.

[0051] For further information, see Figure 5 The process for training the MOCIWRO mathematical model using the proposed P-MORL algorithm is described as follows: First, the model parameters are set. The corresponding parameters are created through the training class and the state is initialized. Then, instances are encoded using an attention-based encoder and transferred to a set of embedded data. A multi-layer perceptron-based decoder then performs multi-head attention calculations on the preferences and embedded data to arrive at a solution. Each iteration consists of multiple batches, each of which includes multiple forward steps for selecting nodes. Node distances are calculated using a partition-based node distance calculation model. The model is trained through continuous iterations. Once nodes are selected, mutations are performed using an evolutionary genetic search algorithm. For node selection, the model records the vehicle's current capacity and the remaining storage capacity of all nodes. If a node has already been accessed or its storage capacity exceeds the vehicle's current capacity, it is blocked as inaccessible to the vehicle. If no nodes are accessible, the vehicle returns to the depot. Once all node storage is full, node selection is complete, and a valid solution is obtained. Furthermore, the model uses a weighted Chebyshev aggregation method to calculate rewards, which minimizes the distance between the values ​​of each objective function and a reference point.

[0052] Test example: the P-MORL algorithm model trained above is used to test the MOCIWRO problem, and the parameters are set as follows: the number of test set instances is 100, the test batch processing parameter batch is 100, the number of biases is 501, and the other parameter settings are the same as the model training parameter settings.

[0053] Further, the performance of the model is judged using the following indicators: Indicator 1: Hypervolume (HV) indicator, which measures the convergence and distribution of the solution set to evaluate the performance of the multi-objective optimization algorithm. Specifically, it is realized by calculating the space hypervolume formed by the non-dominated solution set and the reference point, and can be used to measure the quality of the approximate set related to Pareto optimality. If the hypervolume of a solution set is larger, its indicator value will also be larger, indicating that the performance of the solution set is better.

[0054] Indicator 2: Inverted Generational Distance (IGD) is an indicator for evaluating the convergence and diversity of multi-objective optimization algorithms, and its value is obtained by calculating the distance between the true Pareto front set and the approximate Pareto front set. The smaller the IGD, the closer the approximate solution set to the true Pareto front, indicating that the performance of the algorithm is better.

[0055] Indicator 3: Test set solving time (Time).

[0056] Specific test example: refer to Figure 6 In a certain intelligent warehouse, there is a pickup point, and now the number of intelligent transport vehicles and the number of storage nodes are needed for transportation services to make the storage nodes full of goods. The relevant parameters are set as follows: the test batch processing parameter batch is 1; the number of biases is 11; the other parameter settings are the same as the parameter settings of the model training. The relevant information of the storage nodes and the pickup point is shown in Table 1. The coordinates of the pickup point are .

[0057] Table 1 Position and storage capacity information of storage nodes and pickup points

[0058] Refer to Figure 7, the loss value of the loss function when the MOCIWRO mathematical model is trained using the P-MORL algorithm proposed in the application changes normally, the loss value is slightly oscillated, the loss value curve starts to be flat after falling, and is more in line with the convergence condition, which shows that the training has good effect. In addition, the descending rate of the training loss function loss value after the algorithm is added with the four random mutation operators and two crossover mutation operations based on the evolutionary genetic search algorithm is faster than the descending rate of the training loss function loss value without using the four random mutation operators and two crossover mutation operations based on the evolutionary genetic search algorithm, which shows that the four random mutation operators and two crossover mutation operations based on the evolutionary genetic search algorithm can accelerate the model convergence.

[0059] Referring to Figure 7 , the Pareto front result when the MOCIWRO mathematical model is tested using the P-MORL algorithm, for a given number of preferences, the P-MORL algorithm can generate a corresponding solution, and the more the preferences, the closer to the Pareto optimal front.

[0060] The average performance of each index when the MOCIWRO mathematical model is tested using the P-MORL algorithm proposed in the application is compared with the performance index of other algorithms, and the result is shown in Table 2. Table 2 Comparison result of average performance of each index tested by P-MORL algorithm and performance of other algorithms

[0061] Among them, the MOEA / D algorithm is a decomposition-based multi-objective evolutionary algorithm, the MOGLS algorithm is a multi-objective genetic local search algorithm, and the NSGAII algorithm is a multi-objective genetic algorithm based on Pareto dominance. These algorithm frameworks are algorithms based on evolutionary algorithm framework.

[0062] The larger the HV value, the better the performance of the solution set; the smaller the IGD, the closer the approximate solution set to the true Pareto front, which means that the performance of the algorithm is better; the smaller the solution time of the test set, the faster the model solution speed. From the comparison result, it can be seen that in the comparison of the above algorithms, the P-MORL proposed in the application has the largest HV index, the smallest IGD index, and the shortest solution time, which shows that the performance of the P-MORL algorithm is obviously better than the above three algorithms.

[0063] Referring to Figure 8 , the result of solving the specific test example using the P-MORL algorithm proposed in the application is shown in Table 3. Table 3 Result of solving specific test example by P-MORL algorithm

[0064] Among them, the path The pickup point represents the cumulative load of the intelligent transport vehicle, the storage amount represents the storage amount of the storage node, and the distance represents the transport distance from the previous node to the current node. As can be seen from Table 3, each vehicle has its own transport path, and the storage amount of each storage node has been served by the vehicle, and there is no unserved storage node, and each storage node is only served by one vehicle; at the same time, the load of the vehicle does not exceed its maximum load, each vehicle starts from the pickup point, returns to the pickup point after serving the node, and the vehicle only performs one transport. In this result, target 1, i.e., the total transport distance, is the sum of the distances of all vehicles, which is 8.1042; target 2, i.e., the maximum transport distance of the vehicle, is the distance of vehicle 3, which is 2.3775. The above result is a one-time experimental result with a preference of . Each preference will produce 5 experimental results, and the preference is added by the Weighted-TCH method, the reward is calculated, and finally the average reward result of 5 times is obtained, i.e., target 1 is 8.1042, and target 2 is 2.3775.

[0065] In summary, the multi-objective reinforcement learning intelligent warehouse path optimization method based on preference provided by the present application is a superior method for solving the intelligent warehouse path optimization problem, which can find a balance between the targets, can generate a dense approximation of the Pareto optimal solution based on the preference, and effectively approximate the Pareto optimal frontier. The method is obtained under the innovative ideas of the proposed row-column cross warehouse layout scheme, the partition-based node distance calculation model, the weighted Chebyshev aggregation, the consideration of the relative preference of the decision maker for the optimization target, the combination of the heuristic method and the machine learning method, and the multi-objective reinforcement learning algorithm.

[0066] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced by equivalents without departing from the scope of the technical solutions of the present application. Any form of equivalent replacement or modification of the technical solutions and technical contents disclosed by the present application without departing from the spirit and scope of the technical solutions of the present application should be covered in the scope of the claims of the present application, and still belongs to the protection scope of the present application.

Claims

1. A preference-based multi-objective reinforcement learning intelligent warehousing path optimization method, characterized by: The steps include: S1: Construct a row-column cross-warehousing layout plan, establish a partition-based node distance calculation model and a directed graph of the intelligent transport vehicle path planning network in a one-way mode; S2: Establish a multi-objective capacity-constrained intelligent warehousing path optimization mathematical model, and establish two objective functions: minimizing the total transportation distance and minimizing the maximum transportation distance; S3: A multi-objective reinforcement learning algorithm based on decision maker preferences solves the model in step S2. The algorithm takes a directed graph of a path network as input and performs node embedding via a preference-agnostic attention encoder. S4: The preference-based attention decoder decodes the data from step S3 by first embedding the node context, then calculating the probability of selecting the next node, then performing a mutation operation based on an evolutionary genetic search algorithm, and finally using a multi-layer perceptron to generate decoder parameters conditioned on the preference; S5: Based on the multi-objective reinforcement algorithm under the decision maker's preference conditions, reinforcement learning is performed on the data of step S4. First, an objective function based on preference and weighted Chebyshev aggregation is constructed to achieve multi-objective dimensionality reduction. Then, a reinforcement algorithm is used to estimate the gradient of the preference condition cost, generate learnable model parameters, and finally generate a solution that effectively approaches the Pareto optimal frontier.

2. The preference-based multi-objective reinforcement learning intelligent warehousing path optimization method according to claim 1, characterized in that: The row-column cross storage layout scheme constructed in step S1 is to use a grid method to perform two-dimensional encoding for the storage to represent the location of the storage node, and the minimum unit of the grid is a square with a side length of 1; Establishing a partition-based node distance calculation model is to connect the bottom midpoint to the left top corner and the right top corner, and divide the storage area into the lower left corner and the lower right corner. , directly above , lower right There are three storage areas, and the pickup point is located directly below the warehouse; the steps to determine the area where the storage node is located are as follows: Step 1: Given a node Location coordinates , , is the total number of horizontal grids, and is specified to be an odd number. is the total number of vertical grids; Step 2: If the coordinate position satisfies , then the node Located in the lower left area, denoted as ; Step 3: If the coordinates satisfy and , then the node Located in the upper area, recorded as ; Step 4: If the coordinate position satisfies , then the node Located in the lower right area, denoted as .

3. The preference-based multi-objective reinforcement learning intelligent warehousing path optimization method according to claim 1 or 2, characterized in that: The distance between the storage nodes of the node distance calculation model in step S1 In the case of a storage layout solution based on rows and columns crossing, the calculation steps are as follows: Step 1: Given two nodes and ; Step 2: If the coordinate position satisfies ,but , among which if ,but ,otherwise ; Step 3: If the coordinate position satisfies ,but , among which if ,but ,otherwise ; Step 4: If the coordinate position satisfies ,but , among which if ,but ,otherwise ; Step 5: If the coordinate position satisfies ,but ; Step 6: If the coordinates satisfy ,but ; Step 7: If the coordinate position satisfies ,but ; Step 8: If the coordinate position satisfies ,but .

4. The preference-based multi-objective reinforcement learning intelligent warehousing path optimization method according to claim 1 or 2, characterized in that: The distance between the pickup point and the storage node in the node distance calculation model in step S1 is described as the pickup point To the first and last storage nodes distance and The calculation formula is the same as that of , which is described as: 。 5. The preference-based multi-objective reinforcement learning intelligent warehousing path optimization method according to claim 1, characterized in that: The objective functions of the mathematical model for establishing multi-objective capacity-constrained intelligent warehousing path optimization in step S2 include: Minimize total transport distance Expressed as: ， Minimizing the maximum transport distance means minimizing the maximum transport distance of all vehicles while satisfying the storage capacity of all nodes. Expressed as: ， in, is a 0-1 decision variable, representing the vehicle Whether it is stored by the node Driving towards , if yes, take 1, otherwise take 0, ; Indicates the number of storage nodes; Indicates the number of intelligent transport vehicles; Representation node the distance between them; 、 Respectively represent the distance from the pickup point to the first and last storage nodes of the path.

6. The preference-based multi-objective reinforcement learning intelligent warehousing path optimization method according to claim 1, characterized in that: The decision maker preference description in step S3 is a vector of the decision maker's relative preference for multiple optimization objectives. The preference is a set of scalar vectors that satisfy and ,in is the target number, It is the Preference of a target.

7. The preference-based multi-objective reinforcement learning intelligent warehousing path optimization method according to claim 1, characterized in that: The multi-objective reinforcement learning algorithm based on decision maker preference in step S3 includes the following steps: Step 1: Take the path network directed graph as input, described as: Intelligent transportation vehicle path planning network directed graph uses To indicate that Represents a collection of storage nodes Meet at the pickup point The union of ; Represents a set of road segments, representing the connection between two nodes; Step 2: Node embedding via a preference-agnostic attention encoder, described as: The preference-agnostic attention encoder takes a question instance as input and outputs a set of Embedding of dimensional nodes , expressed as ,in and represents the learnable parameters, Represents node attributes, including node location and storage capacity, that is, ; Step 3: Decode by preference-based attention decoder, which is described as follows: The preference-based attention decoder takes the embeddings of all nodes as input and sequentially decodes them with size The probability of selecting the next node ; The algorithm defines a Parameterized preference conditional random strategy , described as ,in represents a preference-conditioned stochastic strategy; Indicates the path that has been selected; Step 4: Perform context embedding of nodes, which is described as follows: The context embedding of nodes is at the time step The decoder first selects the node and the last selected node To construct a contextual embedding , and then the matrix Connect dimensional vector projected onto a dimensional vector, and then further embed the context into the embedding of all nodes through the multi-head attention model To perform aggregation, the context embedding expression is described as: ,in, 、 、 are the query, key, and value of the multi-head attention model respectively; 、 、 are three projection matrices; Represents the parameters of the multi-head attention model; Step 5: Calculate the probability of selecting the next node, as follows: The node probability is calculated using the node embedding, expressed as ,in is the embedding dimension; is a hyperparameter; Represents a vector The transpose of Indicates the nodes that have been visited; Indicates that the selected node has no effect on the subsequent node selection; is the hyperbolic tangent function; all nodes that have been visited are set to , will not be the next node to be selected; at the time step Select a node The probability of is calculated by the normalized exponential function, expressed as , where the numerator is the node of The exponent of the denominator is the total number of nodes The exponential sum of is a natural constant; Step 6: Mutation is performed through a mutation operation based on an evolutionary genetic search algorithm, which is described as follows: The mutation operation based on the evolutionary genetic search algorithm includes a set of four random mutation operators and two crossover mutation operations; Step 7: Use a multilayer perceptron to generate the decoder parameters conditioned on the preference, as described below: The input of the multilayer perceptron is a dimensional preference vector and model parameters , the multilayer perceptron first generates a hidden embedding , and then by linear projection Map the hidden embeddings into decoder parameters; where is a multi-layer perceptron model with parameters and parameter matrix 、 is a learnable parameter; Step 8: Use multi-objective reinforcement under preference conditions to learn the data, which is described as: Establishing the objective function is described as For a path problem, the objective function is a function containing vectors of different targets, i.e. , based on preference The formula for dimensionality reduction using weighted Chebyshev aggregation is described as ,in It is The ideal value of a goal; For a given instance , the goal is to minimize the expected goal of all preferences, described as ,in represents the objective function, represents the model parameters, is the vector of all possible preferences A collection of Indicates that from the distribution Sampling in , Indicates that the strategy Sampling in , Indicates the expected operation on the random variable; in order to train the model, it is necessary to select from different sets of instances in each iteration Random sampling examples , and define the training loss as ; Step 9: Use the reinforcement algorithm to estimate the gradient of the preference conditional cost, which is described as: the gradient of the estimated preference conditional cost is for a given instance and preferences , estimated using the reinforcement algorithm, expressed as ,in, represents the gradient operator; represents the advantage function; is a baseline for reducing the gradient variance; in each update step, each Combined Preferences 、 Instances and Different routes , giving the approximate gradient as: ； For each update step, Combined using the same shared baseline estimation function , and then randomly select each route The starting node of the network is used to introduce different path choices.

8. The preference-based multi-objective reinforcement learning intelligent warehousing path optimization method according to claim 1, characterized in that: The mutation operation based on the evolutionary genetic search algorithm in step S4 is implemented by applying a set of four random mutation operators and two crossover mutation operations; the four random mutation operators include the following operation steps: Step 1: Random node exchange mutation, described as randomly selecting two nodes from the selected route to exchange without violating the capacity constraint, generating a new route, and updating the routing capacity, expressed as ; Step 2: Random node transfer mutation, described as transferring a randomly selected node from one route to another as the end node, represented as ; Step 3: Random arc exchange mutation, described as following the same process as random node exchange mutation, except that adjacent nodes within the route, i.e. arc nodes, are randomly selected for exchange, expressed as ; Step 4: Random arc transfer mutation, described as following the same process as random node transfer mutation, but instead of randomly selecting a single node, it randomly selects adjacent nodes in the path, i.e., arc nodes, to transfer as the end nodes, expressed as .

9. The preference-based multi-objective reinforcement learning intelligent warehousing path optimization method according to claim 8, characterized in that: The two crossover mutation operations of the mutation operation based on the evolutionary genetic search algorithm in step S4 include heuristic inheritance crossover and random inheritance crossover, and the operation steps are as follows: Step 1: Given two parent nodes , and satisfies ; Step 2: Calculation , and get the two minimum values and Corresponding routes and ; Random selection ; Step 3: Find The route with the most identical nodes and ; Step 4: Perform sub-route inheritance operations: , ; Step 5: Calculate floating nodes: , ; Step 6: 、 Randomly join 、 At the end of the middle route, a new route is formed, and the remaining floating nodes are added as new routes. 、 , thus forming a new sub-route , .

Images