A Method and System for Evaluating and Optimizing Multi-AGV Scheduling Strategies Based on Approximate Synchronization Estimation

Abstract

This invention discloses a method and system for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronous estimation. The method includes: constructing a shared strategy model and a joint strategy evaluation model for multiple AGVs; generating training samples by having AGVs interact with the environment based on the shared strategy model and the joint strategy evaluation model, and calculating the marginal dominance function; calculating the update magnitude cutoff based on the uncertainty of the AGV strategy; and, under any environmental state, replacing the joint dominance function with the marginal dominance function, and combining it with a near-end strategy optimization algorithm, using the gradient cutoff as the strategy ratio pruning interval, and optimizing the strategy of each AGV individually until the multi-AGV scheduling strategy is optimized. This method can achieve stable and accurate strategy evaluation in AGV scheduling tasks, improving training stability and scheduling efficiency.

Description

Technical Field

[0001] This invention belongs to the field of AGV scheduling, specifically relating to a method and system for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronization estimation. Background Technology

[0002] Many real-world decision-making tasks, such as traffic scheduling, multi-sensor collaboration, and robot swarm cooperation, can be modeled as multi-agent collaborative decision-making problems. In such problems, to ensure the independence between agents and the completeness of information during training, a centralized evaluation-decentralized execution training framework is typically adopted. This framework requires differentiated evaluation of agent policies based on a shared reward function.

[0003] In policy-based deep reinforcement learning methods, a centralized critic network is typically used to evaluate each agent's policy. The critic network takes the environment state, past counterfactual experiences, and each agent's policy as input, and its evaluation of the current agent's policy depends on the specific past behaviors of other agents. However, for the current agent, other agents are considered part of the environment, and their behaviors are uncertain. Due to this uncertainty, the same behavior of the current agent may receive different evaluations depending on different counterfactual experiences, thus affecting the stability of the training process. Furthermore, since other agents' policies are constantly being updated, there is a discrepancy between the distribution of new policies and past experiences; this asynchrony between experience and policy leads to certain errors in the evaluation results.

[0004] Reducing the variance caused by uncertainty in the strategy evaluation of multiple Automated Guided Vehicles (AGVs) is of great significance for the stability of the training process. At the same time, synchronizing experience and strategy in strategy evaluation is crucial for reducing evaluation error. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned shortcomings and provide a method and system for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronous estimation. This method can achieve stable and accurate strategy evaluation in AGV scheduling tasks, thereby improving the stability of training and scheduling efficiency.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The first aspect of this invention is to provide a method for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronization estimation, comprising:

[0008] Construct a strategy model for multi-AGV sharing and a joint strategy evaluation model;

[0009] Based on a multi-AGV shared strategy model and a joint strategy evaluation model, AGVs interact with the environment to generate training samples and calculate the marginal advantage function.

[0010] Calculate the gradient cutoff amount based on the uncertainty of the AGV strategy;

[0011] Under any environmental condition, the joint advantage function is replaced by the edge advantage function, and the near-end policy optimization algorithm is used to optimize the policy ratio interval by using the gradient cutoff amount as the policy cutoff interval, until the multi-AGV scheduling policy is optimized.

[0012] As a further improvement of the present invention, the construction of a multi-AGV shared strategy model and a joint strategy evaluation model includes:

[0013] Using the historical trajectory information, current observation information, and AGV identifier of each AGV as inputs, and the probability of AGV actions as inputs, a decision model shared by multiple AGVs is constructed through a recurrent neural network;

[0014] Using the actions of all AGVs and the environmental state as inputs, a joint action-state value function is input, and a joint policy evaluation model is constructed based on the critic network.

[0015] As a further improvement of the present invention, the strategy model and joint strategy evaluation model based on multi-AGV sharing, which allows AGVs to interact with the environment to generate training samples and calculate the marginal advantage function, includes:

[0016] Based on the strategy model shared by multiple AGVs and the joint strategy evaluation model, the actions of the AGV to be evaluated are fixed, and multiple sets of actions are sampled according to the joint strategy of all other AGVs except the AGV itself.

[0017] By combining multiple sets of sampled actions with the actions of the AGV to be evaluated, a set of state transition data is generated by simulating the interaction process.

[0018] The state transition data is input into the joint policy evaluation model, and the joint action-state value function corresponding to each transition data is calculated according to the method of calculating the median function in reinforcement learning.

[0019] The marginal advantage function of the AGV to be evaluated is obtained by averaging the combined motion-state value function.

[0020] For each AGV, repeat the above steps to obtain the edge advantage function for each AGV.

[0021] As a further improvement of the present invention, the simulated interaction process requires the construction of a state transition model and a reward function model, which take the global state and joint actions as inputs.

[0022] As a further improvement of the present invention, the step of calculating the updated gradient cutoff amount based on the uncertainty of the AGV strategy includes:

[0023] The action distribution of each AGV is obtained through the policy network, the variance of the action distribution is calculated, and the calculation result is used as the update gradient cutoff value of the corresponding AGV.

[0024] As a further improvement of the present invention, obtaining the action distribution of each AGV through the policy network includes:

[0025] Generate corresponding action masks based on the list of executable actions of the AGV, and multiply the output of each AGV policy network by the corresponding action mask to obtain the value function estimate of the legal action;

[0026] Then, a differentiable probability distribution is obtained through the Günbel activation function, and a variable parameter t is introduced to dynamically adjust the probability distribution of different legal action value functions, thus obtaining the action distribution of each AGV.

[0027] As a further improvement of the present invention, the individual optimization of the strategy for each AGV includes:

[0028] Using the edge advantage function as the update baseline, combined with the near-end policy optimization algorithm, an update gradient cutoff is added as an update constraint. The AGV policy network is updated by pruning the policy ratio, and the pruning interval controls the AGV update magnitude. The policy of each AGV is optimized individually.

[0029] A second aspect of the present invention is to provide a multi-AGV scheduling strategy evaluation and optimization system based on approximate synchronization estimation, comprising:

[0030] The building module is used to construct a shared strategy model and a joint strategy evaluation model for multiple AGVs;

[0031] The interaction module is used to generate training samples by allowing AGVs to interact with the environment based on a strategy model shared by multiple AGVs and a joint strategy evaluation model, and to calculate the marginal advantage function.

[0032] The uncertainty module is used to calculate the updated gradient cutoff based on the uncertainty of the AGV strategy.

[0033] The optimization module is used to optimize the strategy of each AGV individually under any environmental condition by replacing the joint advantage function with the edge advantage function, combining the near-end policy optimization algorithm, using the gradient cutoff amount as the policy ratio pruning interval, until the multi-AGV scheduling strategy is optimized.

[0034] A third aspect of the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation.

[0035] A fourth aspect of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation.

[0036] Compared with the prior art, the present invention has the following technical effects:

[0037] This invention presents an AGV scheduling strategy evaluation and optimization method based on approximate synchronous estimation. By increasing the sample size, the variance of strategy estimation is reduced, thereby improving the stability of the training process. Simultaneously, by estimating the uncertainty of agent policies, the policy update magnitude of each agent is dynamically constrained, achieving approximate synchronous policy estimation and reducing policy evaluation errors. In multi-AGV scheduling tasks with a shared reward function, the marginal dominance function is obtained by integrating the joint value function according to the policies of different AGVs. This enables differentiated evaluation of different AGV policies under the shared reward function, thereby achieving centralized scheduling of the multi-AGV system. Attached Figure Description

[0038] Figure 1 Here is a flowchart of a multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation, as provided in this invention.

[0039] Figure 2 A method framework diagram is provided for an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the edge advantage estimation method of the present invention;

[0041] Figure 4 This is a schematic diagram of the approximate synchronization estimation method of the present invention.

[0042] Figure 5 This invention provides a multi-AGV scheduling strategy evaluation and optimization system based on approximate synchronization estimation;

[0043] Figure 6 This is a schematic diagram of an electronic device provided by the present invention. Detailed Implementation

[0044] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0045] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0046] It should be noted that the terminals involved in the embodiments of this application may include, but are not limited to, mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers, personal computers (PCs), MP3 players, MP4 players, wearable devices (e.g., smart glasses, smartwatches, smart bracelets), smart home devices, and other smart devices.

[0047] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0048] like Figure 1 As shown, the first objective of this invention is to provide a method for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronization estimation, comprising the following steps:

[0049] S1, Construct a shared strategy model and a joint strategy evaluation model for multiple AGVs;

[0050] S2, based on the strategy model shared by multiple AGVs and the joint strategy evaluation model, enables AGVs to interact with the environment to generate training samples and calculate the marginal advantage function;

[0051] S3, Calculate the updated gradient cutoff based on the uncertainty of the AGV strategy;

[0052] S4. Under any environmental condition, the joint advantage function is replaced by the edge advantage function. Combined with the near-end policy optimization algorithm, the gradient cutoff amount is used as the policy ratio pruning interval. The policy of each AGV is optimized separately until the multi-AGV scheduling policy is optimized.

[0053] The proposed method for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronous estimation is grounded in marginal advantage estimation. By increasing the number of samples, it reduces the variance of strategy estimation, thereby improving the stability of the training process. Simultaneously, by estimating the uncertainty of agent policies, it dynamically constrains the policy update magnitude of each agent, achieving approximate synchronous strategy estimation and reducing policy evaluation errors. The algorithm's performance has been significantly improved on a general multi-agent reinforcement learning testing platform. This method can achieve stable and accurate strategy evaluation in AGV scheduling tasks, improving training stability and scheduling efficiency.

[0054] As an optional approach, in the above steps, the gradient cutoff amount is calculated based on the uncertainty of the AGV strategy, including:

[0055] The action distribution of each AGV is obtained through the policy network, the variance of the action distribution is calculated, and the calculation result is used as the update gradient cutoff value of the corresponding AGV.

[0056] The step of obtaining the action distribution of each AGV through the policy network includes:

[0057] Generate corresponding action masks based on the list of executable actions of the AGV, and multiply the output of each AGV policy network by the corresponding action mask to obtain the value function estimate of the legal action;

[0058] Then, a differentiable probability distribution is obtained through the Günbel activation function, and a variable parameter t is introduced to dynamically adjust the probability distribution of different legal action value functions, thus obtaining the action distribution of each AGV.

[0059] In order to achieve synchronous estimation of different AGV strategies under the multi-agent training framework of centralized evaluation and decentralized execution, this step uses the variance of each AGV strategy as the standard for the uncertainty of agent actions. Based on the magnitude of uncertainty, the strategy change is restricted to a certain range by truncating the policy update amplitude, thereby achieving approximate synchronous estimation.

[0060] As an optional approach, the strategy for each AGV can be optimized individually in the above steps, including:

[0061] Using the edge advantage function as the update baseline, combined with the near-end policy optimization algorithm, an update gradient cutoff is added as an update constraint. The AGV policy network is updated by pruning the policy ratio, and the pruning interval controls the AGV update magnitude. The policy of each AGV is optimized individually.

[0062] This invention, based on a deep reinforcement learning algorithm, reduces the variance of AGV policy evaluation through multi-sample estimation; it also approximates and simplifies synchronous estimation by using update magnitude truncation, ensuring both policy evaluation accuracy and training efficiency. This invention can be applied to AGV system scheduling tasks based on online policy multi-agent reinforcement learning to improve the accuracy of AGV scheduling policy evaluation and the stability of the training process.

[0063] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments:

[0064] Combination Figure 2 The intelligent agent is an AGV. This embodiment provides a method for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronization estimation, including the following steps:

[0065] Step 1: Construct a shared strategy model and a joint strategy evaluation model for multiple AGVs. The specific process is as follows:

[0066] A shared strategy model for multiple AGVs is constructed using a recurrent neural network. The inputs include the historical trajectory information of each AGV (including historical actions and historical observation information), the current observation information, and the AGV identifier, as well as the probability of the AGV actions. A joint strategy evaluation model is constructed based on a commentator network, which takes the actions of all AGVs and the environmental state as inputs and inputs a joint action-state value function.

[0067] Step Two: Based on the shared strategy model and joint strategy evaluation model of multiple AGVs, allow the AGVs to interact with the environment to generate training samples, and calculate the marginal advantage function. (Refer to...) Figure 3 As shown, the specific process is as follows:

[0068] 1) Based on the strategy model shared by multiple AGVs, fix the action of the AGV to be evaluated, and sample multiple sets of actions according to the joint strategy of all other AGVs except the AGV; multiple sets of interactive action samples can be collected.

[0069] 2) Combine the sampling action with the action of the AGV to be evaluated, and simulate the interaction process one step ahead to generate a set of state transition data; keep the samples in the sample buffer pool.

[0070] 3) Calculate the joint action-state value function corresponding to each transition data point, based on the method for calculating the median function in reinforcement learning;

[0071] 4) Average the set of values ​​to obtain the marginal advantage function of the AGV to be evaluated;

[0072] 5) For each AGV, repeat steps 1) to 4) above.

[0073] In step two, in order to simulate the interaction between the AGV and the environment, it is necessary to construct a state transition model and a reward function model. Both models take the global state and joint actions as inputs.

[0074] The sample buffer pool is used to randomly sample training samples, the commentator network is trained, TD-residual estimation is performed to obtain counterfactual advantage estimation, and then the Monte Carlo method is used to obtain marginal advantage estimation.

[0075] Step 3: Calculate the gradient cutoff value based on the uncertainty of the AGV strategy. The specific process is as follows:

[0076] The action distribution of each AGV is obtained through the policy network, the variance of the action distribution is calculated, and the calculation result is used as the update gradient cutoff value of the corresponding AGV.

[0077] As an optional solution, in step three, such as Figure 4 As shown, since AGVs can only execute certain specific actions from their action set, to improve action sampling efficiency, corresponding action masks are generated based on the AGV's list of executable actions. The output of each AGV policy network is multiplied by the corresponding action mask to obtain the value function estimate of the legal action. Then, a differentiable probability distribution is obtained through the Gumbel activation function. This work introduces a variable parameter t to dynamically adjust the probability distribution of different legal action value functions, making the distribution gradually flatten during the training process and making the training process more stable.

[0078] Step 4: Under any environmental condition, replace the joint advantage function with the edge advantage function obtained in Step 2, and combine it with the near-end policy optimization algorithm. Use the gradient cutoff value obtained in Step 3 as the policy ratio pruning interval to optimize the policy of each AGV separately.

[0079] As an optional approach, in step four, the algorithm uses the marginal advantage function estimate obtained in step two as the updated baseline and adds the variance estimate from step three as the update constraint. Directly solving this optimization problem requires calculating the Hessian matrix corresponding to the KL divergence, which is computationally expensive and slow in training efficiency.

[0080] This embodiment references the near-end policy optimization method, updating the AGV policy network through policy ratio pruning, with the pruning interval controlling the AGV update magnitude. When the dominance function estimate is positive, the AGV increases the probability of the corresponding action, but the increase does not exceed a set threshold.

[0081] More specifically, such as Figure 4 As shown, the sample buffer pool randomly samples training samples to train the policy network, obtains the trajectory action probabilities from 1 to n, calculates the gradient of the policy for trajectory n by performing gradient constraint calculation, and then obtains the policy network optimization result.

[0082] The proposed method for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronous estimation is grounded in edge advantage estimation. By increasing the number of samples, it reduces the variance of strategy estimation, thereby improving the stability of the training process. Simultaneously, by estimating the uncertainty of AGV strategies, it dynamically constrains the strategy update magnitude for each AGV, achieving approximate synchronous strategy estimation and reducing strategy evaluation errors. The algorithm's performance has been significantly improved on a general multi-AGV reinforcement learning testing platform.

[0083] For example, a shared multi-agent policy network can be implemented using a multilayer perceptron and a recurrent neural network, where the input layer is a recurrent neural network and the output layer is a multilayer perceptron. The critic network is implemented using a multilayer perceptron. The output layer size of the policy network and the critic network can be adjusted according to the requirements of the task to achieve different levels of policy fitting and evaluation capabilities. The experience replayer is implemented using readable storage units.

[0084] The algorithm involved in this invention can be deployed on any machine with storage units and floating-point operation capabilities. For simulation tasks, a corresponding simulation environment needs to be deployed on the machine; for tasks in real-world scenarios, corresponding sensors and actuators also need to be deployed to enable physical interaction with the environment.

[0085] As a specific embodiment, such as Figure 5 As shown, the second objective of this invention is to provide a multi-AGV scheduling strategy evaluation and optimization system based on approximate synchronization estimation, comprising:

[0086] The building module is used to construct a shared strategy model and a joint strategy evaluation model for multiple AGVs;

[0087] The interaction module is used to generate training samples by allowing AGVs to interact with the environment based on a strategy model shared by multiple AGVs and a joint strategy evaluation model, and to calculate the marginal advantage function.

[0088] The uncertainty module is used to calculate the updated gradient cutoff based on the uncertainty of the AGV strategy.

[0089] The optimization module is used to optimize the strategy of each AGV individually under any environmental condition by replacing the joint advantage function with the edge advantage function, combining the near-end policy optimization algorithm, using the gradient cutoff amount as the policy ratio pruning interval, until the multi-AGV scheduling strategy is optimized.

[0090] According to embodiments of this application, this application also provides an electronic device and a non-transitory computer-readable storage medium storing computer instructions.

[0091] Figure 6This is a schematic diagram of an electronic device used to implement the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation according to embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0092] like Figure 6 As shown, the electronic device includes one or more processors 501, a memory 502, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI (Graphical User Interface) on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take a processor 501 as an example.

[0093] The memory 502 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause the at least one processor to execute the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to execute the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation provided in this application.

[0094] The memory 502, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer-executable programs, and units, such as the program instructions / units corresponding to the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation in the embodiments of this application. The processor 501 executes various server functions and data processing by running the non-transient software programs, instructions, and units stored in the memory 502, thereby implementing the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation in the above method embodiments.

[0095] The memory 502 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function. The data storage area may store data created by the use of the electronic device implementing the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation provided in the embodiments of this application. Furthermore, the memory 502 may include high-speed random access memory and may also include non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 502 may optionally include memory remotely located relative to the processor 501. These remote memories can be connected via a network to the electronic device implementing the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation provided in the embodiments of this application. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0096] The electronic device based on the evaluation and optimization method for multi-AGV scheduling strategies using approximate synchronization estimation may further include: an input device 503 and an output device 504. The processor 501, memory 502, input device 503, and output device 504 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0097] Input device 503 can receive input digital or character information, as well as key signal inputs related to user settings and function control of the electronic device implementing the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation provided in the embodiments of this application. Examples of input devices include touchscreens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, and joysticks. Output device 504 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, LCD (liquid crystal display), LED (light-emitting diode) displays, and plasma displays. In some embodiments, the display device may be a touchscreen.

[0098] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, PLD (programmable logic device)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0100] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0101] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0102] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0103] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronization estimation, characterized in that, include: Construct a strategy model for multi-AGV sharing and a joint strategy evaluation model; Based on a multi-AGV shared strategy model and a joint strategy evaluation model, AGVs interact with the environment to generate training samples and calculate the marginal advantage function. Calculate the gradient cutoff amount based on the uncertainty of the AGV strategy; In any environment, the joint advantage function is replaced by the edge advantage function, and the near-end policy optimization algorithm is used to optimize the policy ratio interval by using the gradient cutoff amount as the policy cutoff interval, and the policy of each AGV is optimized separately until the multi-AGV scheduling policy is optimized. The strategy model and joint strategy evaluation model based on multi-AGV sharing generate training samples through AGV interaction with the environment, and calculate the marginal advantage function, including: Based on the strategy model shared by multiple AGVs and the joint strategy evaluation model, the actions of the AGV to be evaluated are fixed, and multiple sets of actions are sampled according to the joint strategy of all other AGVs except the AGV itself. By combining multiple sets of sampled actions with the actions of the AGV to be evaluated, a set of state transition data is generated by simulating the interaction process. The state transition data is input into the joint policy evaluation model, and the joint action-state value function corresponding to each transition data is calculated according to the method of calculating the median function in reinforcement learning. The marginal advantage function of the AGV to be evaluated is obtained by averaging all the joint motion-state value functions. For each AGV, repeat the above steps to obtain the edge advantage function for each AGV; The step of calculating the updated gradient cutoff based on the uncertainty of the AGV strategy includes: The action distribution of each AGV is obtained through the policy network, the variance of the action distribution is calculated, and the calculation result is used as the update gradient cutoff value of the corresponding AGV.

2. The method for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronization estimation according to claim 1, characterized in that, The construction of a multi-AGV shared strategy model and a joint strategy evaluation model includes: Using the historical trajectory information, current observation information, and AGV identifier of each AGV as inputs, and the probability of AGV actions as inputs, a decision model shared by multiple AGVs is constructed through a recurrent neural network; Using the actions of all AGVs and the environmental state as inputs, a joint action-state value function is input, and a joint policy evaluation model is constructed based on the critic network.

3. The method for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronization estimation according to claim 1, characterized in that, The simulated interaction process requires the construction of a state transition model and a reward function model, which take the global state and joint actions as inputs.

4. The method for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronization estimation according to claim 1, characterized in that, The process of obtaining the action distribution of each AGV through the policy network includes: Generate corresponding action masks based on the list of executable actions of the AGV, and multiply the output of each AGV policy network by the corresponding action mask to obtain the value function estimate of the legal action; Then, a differentiable probability distribution is obtained through the Günbel activation function, and a variable parameter t is introduced to dynamically adjust the probability distribution of different legal action value functions, thus obtaining the action distribution of each AGV.

5. The method for evaluating and optimizing multi-AGV scheduling strategies based on approximate synchronization estimation according to claim 1, characterized in that, The individual optimization of the strategy for each AGV includes: Using the edge advantage function as the update baseline, combined with the near-end policy optimization algorithm, an update gradient cutoff is added as an update constraint. The AGV policy network is updated by pruning the policy ratio, and the pruning interval controls the AGV update magnitude. The policy of each AGV is optimized individually.

6. A multi-AGV scheduling strategy evaluation and optimization system based on approximate synchronization estimation, characterized in that, include: The building module is used to construct a shared strategy model and a joint strategy evaluation model for multiple AGVs; The interaction module is used to generate training samples by allowing AGVs to interact with the environment based on a strategy model shared by multiple AGVs and a joint strategy evaluation model, and to calculate the marginal advantage function. The uncertainty module is used to calculate the updated gradient cutoff based on the uncertainty of the AGV strategy. The optimization module is used to optimize the strategy of each AGV individually under any environmental state by replacing the joint advantage function with the edge advantage function, combining the near-end policy optimization algorithm, using the gradient cutoff amount as the policy ratio pruning interval, until the multi-AGV scheduling strategy is optimized. The strategy model and joint strategy evaluation model based on multi-AGV sharing generate training samples through AGV interaction with the environment, and calculate the marginal advantage function, including: Based on the strategy model shared by multiple AGVs and the joint strategy evaluation model, the actions of the AGV to be evaluated are fixed, and multiple sets of actions are sampled according to the joint strategy of all other AGVs except the AGV itself. By combining multiple sets of sampled actions with the actions of the AGV to be evaluated, a set of state transition data is generated by simulating the interaction process. The state transition data is input into the joint policy evaluation model, and the joint action-state value function corresponding to each transition data is calculated according to the method of calculating the median function in reinforcement learning. The marginal advantage function of the AGV to be evaluated is obtained by averaging all the joint motion-state value functions. For each AGV, repeat the above steps to obtain the edge advantage function for each AGV; The step of calculating the updated gradient cutoff based on the uncertainty of the AGV strategy includes: The action distribution of each AGV is obtained through the policy network, the variance of the action distribution is calculated, and the calculation result is used as the update gradient cutoff value of the corresponding AGV.

7. An electronic device, characterized in that: The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the multi-AGV scheduling strategy evaluation and optimization method based on approximate synchronization estimation as described in any one of claims 1-5.

Images