Multimodal multi-target storage yard site selection optimization method based on reinforcement learning

Abstract

The invention provides a multi-modal multi-target storage yard site selection optimization method based on reinforcement learning. The method comprises the following steps: S1, obtaining storage yard site selection coordinates to form a first population; s2, acquiring a niche set and a first data set population; s3, judging whether the number of the training samples is smaller than a preset threshold value or not; if yes, turning to step S4; if not, turning to step S5; s4, evolving and updating the first population, and synchronously updating to obtain a second data set population; s5, establishing a reinforcement learning model and performing training by using the training sample; s6, screening according to a training result to obtain an optimal niche strategy, and evolving and updating the first population to obtain a second population; step S7, judging whether an evolutionary iteration stop condition is met or not: if yes, outputting a second population as a multi-modal multi-target storage yard site selection optimization result; if not, returning to the step S3. The beneficial effects of the invention are that the method can achieve the optimization of the site selection of the storage yard according to the multi-mode site selection scene of the storage yard, and improves the optimization efficiency of site selection of the storage yard.

Description

Technical Field

[0001] The present invention relates to the technical field of storage yard site selection optimization methods, and in particular to a multi-modal and multi-objective storage yard site selection optimization method based on reinforcement learning. Background Art

[0002] The yard site selection optimization problem is widely present in practical life and engineering application problems, such as logistics warehousing yard site selection, container yard site selection, etc. In these yard site selection optimization problems, it is usually necessary to consider multiple optimization objectives. Optimizing one of the objectives may cause another objective to deteriorate. Therefore, it is necessary to find a yard solution that meets multiple objectives. The current existing technology is to provide a good solution by establishing a mathematical model and using a multi-objective combined optimization method, and find a suitable Pareto frontier, that is, a suitable yard site selection optimization solution.

[0003] However, the current existing technical methods can only find a single yard method that meets the target requirements, and cannot find a multi-yard optimization solution that meets the target requirements, that is, a multi-modal solution. In other words, there is a multi-modal yard site selection solution that meets multi-objective requirements, that is, a multi-modal multi-objective yard site selection optimization solution. Since the current existing technology only considers optimizing multi-objective yard selection under a single modal scenario, it cannot find a multi-modal yard site selection solution. Therefore, the difficulty in optimizing multi-modal multi-objective yard site selection lies in how to search for multi-modal solutions; secondly, when searching for multi-modal solutions, how to maintain the convergence and feasibility of the multi-modal yard optimization solution is crucial and has great challenges. Summary of the invention

[0004] The technical problem to be solved by the present invention is to realize the optimization of storage yard site selection according to the multi-modal storage yard site selection scenario and improve the efficiency of storage yard site selection optimization. In order to overcome the defects of the above-mentioned prior art (or related technology), the present invention provides a multi-modal multi-objective storage yard site selection optimization method based on reinforcement learning.

[0005] The present invention provides a multi-modal and multi-objective storage yard location optimization method based on reinforcement learning, comprising the following steps: Step S1, for different storage yard scenarios, obtaining the storage yard site selection coordinates in each storage yard scenario, and taking each storage yard site selection coordinate as a separate individual to form a first population; Step S2, obtaining a plurality of different niche strategies to form a niche set and a first data set population for training a reinforcement learning model; Step S3, determining whether the number of training samples of the first data set population is less than a preset threshold: If yes, go to step S4; If not, go to step S5; Step S4, based on a genetic algorithm, the first population is updated through evolution of each of the microhabitat strategies in the microhabitat set, and the first data set population is synchronously updated when the first population is evolved and updated to obtain a second data set population, and then the process goes to step S5; Step S5, establishing the reinforcement learning model and using each of the training samples in the first data set population or the second data set population to train the reinforcement learning model; Step S6, screening the optimal niche strategy from the niche set according to the training result of the reinforcement learning model, and using the optimal niche strategy to evolve and update the first population to obtain a second population; Step S7, determine whether the preset evolution iteration stop condition is met: If yes, output the second population as a multi-modal multi-objective storage yard location optimization result to assist in storage yard location selection; If not, return to step S3.

[0006] Compared with the prior art, the multi-modal multi-objective storage yard location optimization method based on reinforcement learning has the following advantages: In the present invention, the first population is obtained and generated through step S1, the microhabitat set and the first data set population are obtained through step S2, whether the training sample is sufficient is determined through step S3, the first data set population is evolved and updated through step S4, the reinforcement learning model is trained through step S5, the microhabitat strategy is screened and the first population is evolved and updated through step S6, and the evolution iteration stop condition is determined through step S7. The multimodal yard site selection scenario in the yard site selection optimization problem is considered in the whole process, and the yard site selection coordinates with different selectivities are provided through the multimodal yard site selection scenario, which can provide users with yard site selection schemes of different modes for selection, adopts the reinforcement learning model self-training to select the appropriate microhabitat strategy to search the multimodal yard site selection scheme, regards multiple microhabitat strategies as behaviors in reinforcement learning, and continuously trains the reinforcement learning model to select the microhabitat strategy with the highest probability as the optimal microhabitat strategy to improve the yard site selection optimization efficiency, and uses the optimal microhabitat strategy to evolve and update the first population to obtain the multimodal multi-objective yard site selection optimization result.

[0007] In a possible implementation, step S4 includes: Step S41, randomly selecting one of the niche strategies in the niche set, and using the niche strategy to evolve and update the first population to generate an offspring population based on a genetic algorithm; Step S42, integrating the first population and the offspring population into a set, and selecting and generating a new second population in the set using an environmental selection strategy; Step S43, obtaining and obtaining a state value according to the pre-evolution convergence quality, pre-evolution diversity quality, and pre-evolution feasibility quality of the second population, obtaining and obtaining a reward value according to the weight parameter of the pre-evolution convergence quality, the weight parameter of the pre-evolution diversity quality, the weight parameter of the pre-evolution feasibility quality, the post-evolution convergence quality, the post-evolution diversity quality, and the post-evolution feasibility quality, obtaining and obtaining a record sample according to the pre-evolution convergence quality, the pre-evolution diversity quality, the pre-evolution feasibility quality, the microhabitat strategy, and the reward value, and forming the second data set population according to the state value, the reward value, and the record sample.

[0008] Compared with the existing technology, the above technical solution can improve the diversity, convergence and feasibility of the reinforcement learning model by introducing convergence quality, diversity quality and feasibility quality, while considering the optimization of storage yard site selection under multiple conflicting objectives and multiple constraints to improve the rationality of storage yard site selection.

[0009] In a possible implementation, in step S43, the state value is obtained by the following calculation formula: : ; in, represents the state value; represents the convergence quality before the evolution; represents the quality of diversity before said evolution; represents the feasibility quality before evolution.

[0010] In a possible implementation, in step S43, the reward value is obtained by the following calculation formula: : ; in, represents the reward value; A weight parameter representing the convergence quality before the evolution; represents the convergence quality before the evolution; a weight parameter representing the quality of the diversity before evolution; represents the quality of diversity before said evolution; A weight parameter representing the feasibility quality before evolution; represents the feasibility quality before said evolution; represents the convergence quality after the evolution; represents the quality of diversity after said evolution; Represents the feasibility quality after evolution.

[0011] In a possible implementation manner, in step S43, the recorded sample is obtained by the following calculation formula: : ; in, represents said record sample; represents the convergence quality before the evolution; represents the quality of diversity before said evolution; represents the feasibility quality before said evolution; represents the niche strategy; represents the reward value; represents the convergence quality after the evolution; represents the quality of diversity after said evolution; Represents the feasibility quality after evolution.

[0012] In a possible implementation, step S5 includes: Step S51, determining whether the reinforcement learning model is established: If yes, go to step S52; If not, go to step S53; Step S52, randomly selecting a plurality of the training samples from the first data set population as training sets to train the reinforcement learning model, and using a genetic algorithm to evolve and update each of the training samples in the first data set population every 10 trainings; Step S53, establishing the reinforcement learning model, and randomly selecting a plurality of the training samples from the second data set population as training sets to train the reinforcement learning model.

[0013] In a possible implementation, in step S2, the niche set includes the niche strategy of the clustering algorithm and the niche strategy of the ring topology structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of the steps of the present invention; Figure 2 It is a schematic diagram of the implementation process of the present invention; Figure 3 is a specific flow chart of step S4 of the present invention; Figure 4 It is a specific flow chart of step S5 of the present invention. DETAILED DESCRIPTION

[0015] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present invention, and are not intended to limit the protection scope of the embodiments of the present invention. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0016] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] See also Figure 1 The embodiment of the present invention discloses a multi-modal and multi-objective storage yard location optimization method based on reinforcement learning, comprising the following steps: Step S1, for different storage yard scenarios, obtaining the storage yard site selection coordinates in each storage yard scenario, and taking each storage yard site selection coordinate as a separate individual to form a first population; Step S2, obtaining a plurality of different niche strategies to form a niche set and a first data set population for training a reinforcement learning model; Step S3, determining whether the number of training samples of the first data set population is less than a preset threshold: If yes, go to step S4; If not, go to step S5; Step S4, based on a genetic algorithm, the first population is updated through evolution of each of the microhabitat strategies in the microhabitat set, and the first data set population is synchronously updated when the first population is evolved and updated to obtain a second data set population, and then the process goes to step S5; Step S5, establishing the reinforcement learning model and using each of the training samples in the first data set population or the second data set population to train the reinforcement learning model; Step S6, screening the optimal niche strategy from the niche set according to the training result of the reinforcement learning model, and using the optimal niche strategy to evolve and update the first population to obtain a second population; Step S7, determine whether the preset evolution iteration stop condition is met: If yes, output the second population as a multi-modal multi-objective storage yard location optimization result to assist in storage yard location selection; If not, return to step S3.

[0018] Continue to see Figure 1 In view of the problems and difficulties existing in the current multi-modal and multi-objective storage yard site selection optimization problem, the present invention proposes a multi-modal and multi-objective storage yard site selection optimization method based on reinforcement learning. The method adopts multiple microhabitat strategies to collaboratively search for multi-modal storage yard solutions. In the process of co-evolution of multiple microhabitat strategies, a reinforcement learning model is used to adaptively select appropriate microhabitat strategies to search for multi-modal storage yard site selection solutions, so as to provide users with different solution options.

[0019] In the specific implementation scheme of the present application, the yard scene is a multimodal scene of different types, which can be a container yard scene, a logistics warehousing yard scene, or other concrete actual scenes, and the yard site selection coordinates obtained from the yard scene can be the world longitude and latitude coordinates of a certain location in the yard scene, or the x-axis horizontal coordinate and y-axis vertical coordinate converted from the world longitude and latitude coordinates of a certain point after converting the yard scene into a virtual two-dimensional plane scene, that is, the yard scene in the present application is a real scene, the yard site selection coordinates are physical quantities of the concrete physical world, and the first group includes the yard site selection coordinates, and the second group obtained by step S7 includes the optimized yard site selection coordinates, and the optimized yard site selection coordinates can be provided to the operator as a multimodal multi-objective yard site selection optimization result to assist in specific site selection.

[0020] See also Figure 3 , the step S4 comprises: Step S41, randomly selecting one of the niche strategies in the niche set, and using the niche strategy to evolve and update the first population to generate an offspring population based on a genetic algorithm; Step S42: Integrate the first population and the offspring population into a set , and in the collection Using an environmental selection strategy to select and generate a new second population; Step S43, obtaining and obtaining a state value according to the pre-evolution convergence quality, pre-evolution diversity quality, and pre-evolution feasibility quality of the second population , obtain and obtain the reward value according to the weight parameter of the convergence quality before evolution, the weight parameter of the diversity quality before evolution, the weight parameter of the feasibility quality before evolution, the convergence quality after evolution, the diversity quality after evolution, and the feasibility quality after evolution , obtain and according to the pre-evolution convergence quality, the pre-evolution diversity quality, the pre-evolution feasibility quality, the niche strategy, the reward value Get a sample record , and the state value , the reward value and the recorded sample Included in the dataset population Evolutionary updates within.

[0021] See also Figure 4 , the step S5 comprises: Step S51, determining whether the reinforcement learning model is established: If yes, go to step S52; If not, go to step S53; Step S52, randomly selecting a plurality of the training samples from the first data set population as training sets to train the reinforcement learning model, and using a genetic algorithm to evolve and update each of the training samples in the first data set population every 10 trainings; Step S53, establishing the reinforcement learning model, and randomly selecting a plurality of the training samples from the second data set population as training sets to train the reinforcement learning model.

[0022] See also Figure 2 , when implementing the plan, the steps are as follows: Step A1: Randomly initialize a population of N individuals , every individual The coordinates of are the selectable yard location coordinates in the site selection optimization; Step A2: Initialize the microhabitat set , microhabitat collection It contains clustering algorithms, ring topology structures and other niche strategies, and initializes the data set population of the reinforcement learning model is empty; Step A3: Perform the following steps to enter the iterative evolution stage: Step A3-1, if the dataset population The number of training samples in is less than the preset threshold Then perform the following steps: Step A3-1-1, from the microhabitat strategy set A microhabitat strategy is randomly selected in the set, and a genetic algorithm is used to evolve the population with the assistance of the selected microhabitat strategy. ; Step A3-1-2, Population Genetic algorithm is used to evolve and update and generate offspring population , and in the collection Use environmental selection strategies to select and generate new populations ; Step A3-1-3, calculate the population using the following formula State values ​​in reinforcement learning models : ; in, The populations The convergence quality, diversity quality and feasibility quality of Step A3-1-4, use the following formula to calculate the population Reward value in reinforcement learning model : ; in, They are The weight parameter of The populations The convergence quality, diversity quality and feasibility quality after evolution; Step A3-1-5, calculate the population using the following formula Recording examples in reinforcement learning models : ; in, the niche strategy used by the current population; In this way, as the population continues to evolve and update, the initial dataset population in the reinforcement learning model can be collected Perform synchronous evolution updates; Step A3-2, if the dataset population The number of training samples in is greater than or equal to the preset threshold Then perform the following steps: Step A3-2-1: If the reinforcement learning model has not been established, establish a reinforcement learning model , and from the dataset population Random Selection training samples as the training set to train the reinforcement learning model ; Step A3-2-2: If the reinforcement learning model has been established, then select the population from the dataset Random Selection training samples as the training set to train the reinforcement learning model , and use genetic algorithm to evolve and update the training samples every 10 times; Step A3-2-3, based on the training results of the reinforcement learning model, Select the microhabitat strategy with the highest probability value As the current population The optimal niche strategy used during evolutionary renewal; Step A3-3, using the optimal niche strategy selected during the genetic algorithm evolution process Evolutionary renewal population ; Step A3-4: If the evolutionary iteration termination condition is met, output the final population That is the final multi-modal multi-objective yard site selection coordinates. If it is not satisfied, repeat step A3.

[0023] Continue to see Figure 2 , in step A1, initialize the population , which contains N individuals, each of which The vector value represents the position coordinates, such as individual They represent the horizontal and vertical coordinates of the yard coordinates respectively.

[0024] Continue to see Figure 1 Compared with the traditional yard site selection optimization method, the present invention takes into account the multi-modal yard site selection scenario in the yard site selection optimization, and can provide users with yard site selection plans of different modes for users to choose; secondly, compared with the traditional yard site selection optimization method, this method simultaneously considers the yard site selection optimization under multiple conflicting objectives and multiple constraints; and, a reinforcement learning model is used for self-training to select appropriate microhabitat strategies to search for multi-modal yard site selection plans. In this method, multiple microhabitat strategies are regarded as behaviors in reinforcement learning, and a reasonable reward function is designed. The reinforcement learning model is continuously trained to select the microhabitat strategy with the highest probability, and the microhabitat strategy is used to search for multi-modal yard site selection plans.

[0025] Continue to see Figure 1, the following is an expansion of the reinforcement learning model. The reinforcement learning model has the following four main components: policy; reward; value function; environment model; among them, policy refers to the strategy of the agent (intelligent body) to choose actions, and the agent chooses actions according to this strategy. The strategy here does not refer to how to choose actions under a specific state, but from a global perspective, the policy can be regarded as a function f about the state s. The input of this function is the state s (the state needs to be described in a form that can be input) and the output is an action. In the reinforcement learning model, the goal is to learn a policy and use this policy to choose actions so that the final reward is maximized; reward defines the purpose of the reinforcement learning model. It tells the agent whether your action is good or bad. The agent can get a reward after performing an action at time t. , which shows how well the agent performs at this step. From the series of actions performed at 0, 1, …, t-1, t, we can get , the purpose of the agent is to maximize the accumulation of these R, which is to maximize the reward; value function represents the good or bad in the long run, that is, the value of a state s refers to the average reward in state s. Starting from state s, there are many actions to choose from, and then reaching the next state, there are many actions to choose from. This is like a tree, constantly expanding, different actions will get different rewards, and value function is to represent the average reward of a state s; environment model means that if everything about the environment is known, it is said that the environment is known, that is, modelbased, that is, in this case, the agent knows what its state transition probability is after selecting an action, and what the reward is. If all these are known, the dynamic programming method (DP) can be used to solve the problem, but in real life, it is difficult to know the transition probability between states. This situation is called model ree. Therefore, it is impossible to directly use the dynamic programming method to solve the problem.

[0026] See also Figure 1 In this scheme, the specific implementation method of using the microhabitat strategy to update the population evolution is not expanded, and this process is the existing technology.

[0027] In the description of the present invention, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0028] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A multi-modal and multi-objective yard location optimization method based on reinforcement learning, characterized in that: The following steps are involved: Step S1, for different storage yard scenarios, obtaining the storage yard site selection coordinates in each storage yard scenario, and taking each storage yard site selection coordinate as a separate individual to form a first population; Step S2, obtaining a plurality of different niche strategies to form a niche set and a first data set population for training a reinforcement learning model; Step S3, determining whether the number of training samples of the first data set population is less than a preset threshold: If yes, go to step S4; If not, go to step S5; Step S4, based on a genetic algorithm, the first population is updated through evolution of each of the microhabitat strategies in the microhabitat set, and the first data set population is synchronously updated when the first population is evolved and updated to obtain a second data set population, and then the process goes to step S5; Step S5, establishing the reinforcement learning model and using each of the training samples in the first data set population or the second data set population to train the reinforcement learning model; Step S6, screening the optimal niche strategy from the niche set according to the training result of the reinforcement learning model, and using the optimal niche strategy to evolve and update the first population to obtain a second population; Step S7, determine whether the preset evolution iteration stop condition is met: If yes, output the second population as a multi-modal multi-objective storage yard location optimization result to assist in storage yard location selection; If not, return to step S3.

2. The multi-modal multi-objective storage yard location optimization method based on reinforcement learning according to claim 1 is characterized in that: The step S4 comprises: Step S41, randomly selecting one of the niche strategies in the niche set, and using the niche strategy to evolve and update the first population to generate an offspring population based on a genetic algorithm; Step S42, integrating the first population and the offspring population into a set, and selecting and generating a new second population in the set using an environmental selection strategy; Step S43, obtaining and obtaining a state value according to the pre-evolution convergence quality, pre-evolution diversity quality, and pre-evolution feasibility quality of the second population; obtaining and obtaining a reward value according to the weight parameter of the pre-evolution convergence quality, the weight parameter of the pre-evolution diversity quality, the weight parameter of the pre-evolution feasibility quality, the post-evolution convergence quality, the post-evolution diversity quality, and the post-evolution feasibility quality; obtaining and obtaining a record sample according to the pre-evolution convergence quality, the pre-evolution diversity quality, the pre-evolution feasibility quality, the microhabitat strategy, and the reward value; and forming the second data set population according to the state value, the reward value, and the record sample.

3. The multi-modal multi-objective storage yard location optimization method based on reinforcement learning according to claim 2 is characterized in that: In step S43, the state value is obtained by the following calculation formula: : ； in, represents the state value; represents the convergence quality before the evolution; represents the quality of diversity before said evolution; represents the feasibility quality before evolution.

4. The multi-modal multi-objective storage yard location optimization method based on reinforcement learning according to claim 2 is characterized in that: In step S43, the reward value is obtained by the following calculation formula: : ； in, represents the reward value; A weight parameter representing the convergence quality before the evolution; represents the convergence quality before the evolution; a weight parameter representing the quality of the diversity before evolution; represents the quality of diversity before said evolution; A weight parameter representing the feasibility quality before evolution; represents the feasibility quality before the evolution; represents the convergence quality after the evolution; represents the quality of diversity after said evolution; Represents the feasibility quality after evolution.

5. The multi-modal multi-objective storage yard location optimization method based on reinforcement learning according to claim 2 is characterized in that: In step S43, the recorded sample is obtained by the following calculation formula: : ； in, represents said record sample; represents the convergence quality before the evolution; represents the quality of diversity before said evolution; represents the feasibility quality before the evolution; represents the niche strategy; represents the reward value; represents the convergence quality after the evolution; represents the quality of diversity after said evolution; Represents the feasibility quality after evolution.

6. The multi-modal multi-objective storage yard location optimization method based on reinforcement learning according to claim 1 is characterized in that: The step S5 comprises: Step S51, determining whether the reinforcement learning model is established: If yes, go to step S52; If not, go to step S53; Step S52, randomly selecting a plurality of the training samples from the first data set population as training sets to train the reinforcement learning model, and using a genetic algorithm to evolve and update each of the training samples in the first data set population every 10 trainings; Step S53, establishing the reinforcement learning model, and randomly selecting a plurality of the training samples from the second data set population as training sets to train the reinforcement learning model.

7. The multi-modal multi-objective storage yard location optimization method based on reinforcement learning according to claim 1 is characterized in that: In the step S2, the niche set includes the niche strategy of the clustering algorithm and the niche strategy of the ring topology structure.

Images