A multi-amr intelligent scheduling optimization method based on variational resolution reinforcement learning

Abstract

The application discloses a multi-AMR intelligent scheduling optimization method based on variational resolution reinforcement learning, relates to the technical field of intelligent scheduling, establishes a map for a scheduling environment, gives different pixel values to different objects in the map, carries out map segmentation based on a quadtree algorithm, and obtains a variational resolution map; a decision model for multi-AMR path planning is constructed, the decision model comprises a state, an action, a reward and a state transition probability; the state refers to a position, and a map block is used to represent the position; path planning is carried out according to a Q-learning algorithm and the decision model, and the multi-AMR is controlled to reach a target position on the variational resolution map. The application solves the dimension disaster problem caused by excessive space information of reinforcement learning when the map of the scheduling environment is large, and improves the training efficiency of traditional reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of intelligent scheduling technology, and in particular to a multi-AMR intelligent scheduling optimization method based on variable resolution reinforcement learning. Background Technology

[0002] AMR stands for Autonomous Mobile Robot, which is a comprehensive robot system based on AI intelligent algorithms that integrates multiple functions such as environmental perception, dynamic decision-making, behavior control, and autonomous execution.

[0003] With rapid economic and social development, the intelligent automation transformation of industries such as manufacturing and commerce has created a broad market prospect for multi-AMR intelligent scheduling. For complex scheduling scenarios involving a large number of tasks and a large map area, multi-AMR intelligent scheduling algorithms are of great significance in improving the overall system's operational efficiency, reducing labor costs, and increasing company profits.

[0004] Reinforcement learning algorithms are capable of controlling multiple agents and are suitable for scheduling multiple AMRs. However, traditional reinforcement learning algorithms suffer from the curse of dimensionality as spatial information increases. To reduce spatial information and avoid the curse of dimensionality, some scholars have proposed combining neural networks with reinforcement learning algorithms. However, this method of function approximation inevitably introduces errors to some extent. Therefore, how to reasonably model the scheduling environment map, reduce unnecessary identical state information, and combine it with traditional reinforcement learning algorithms to avoid the curse of dimensionality caused by excessive spatial information, improve the training efficiency of traditional reinforcement learning, and achieve intelligent scheduling of multiple AMRs remains a technical challenge. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a multi-AMR intelligent scheduling optimization method based on variable resolution reinforcement learning, which solves the problem of dimensionality curse caused by excessive spatial information in reinforcement learning when the scheduling environment map is large, and improves the training efficiency of traditional reinforcement learning.

[0006] To achieve the above objectives, the present invention adopts the following technical solution, including:

[0007] A multi-AMR intelligent scheduling optimization method based on variable resolution reinforcement learning includes the following steps:

[0008] S1. A map is built for the scheduling environment, different pixel values ​​are assigned to different items in the map, and the map is segmented based on the quadtree algorithm to obtain a variable resolution map; the variable resolution map includes map tiles of different sizes, and each map tile contains only one item or is blank; the AMR moves on the blank map tile.

[0009] S2, Construct a decision model for multi-AMR path planning. The decision model includes state, action, reward, and state transition probability; the state refers to the location, which is represented by map tiles.

[0010] S3 performs path planning based on the Q-learning algorithm and decision model, controlling multiple AMRs to reach the target location on the variable resolution map.

[0011] In step S1, the specific steps for obtaining the variable resolution map are as follows:

[0012] S11, determine the size of the scheduling environment and the items within the scheduling environment, clarify the size and spatial relationship of each item; use CAD software to create a planar map model of the scheduling environment, and assign different RGB pixel values ​​to different items using CAD software to obtain an RGB map of the scheduling environment;

[0013] S12 converts the RGB map to a grayscale map;

[0014] S13, set the segmentation criteria for the map / map tile, including: grayscale range standard and size standard;

[0015] The grayscale range standard is: the grayscale range of the map / map tile is greater than 0; where, the grayscale range of the map / map tile is the difference between the maximum grayscale value and the minimum grayscale value.

[0016] Size standard: The length of the shorter side in the map / map tile (i.e., the shorter side length) is greater than or equal to twice the length of the longer side in the AMR (i.e., the longer side length);

[0017] S14. Determine if the map meets the segmentation criteria. If the grayscale range of the map is 0, or the length of the short side of the map is less than twice the length of the long side of the AMR, the map does not meet the segmentation criteria and is not segmented. If the grayscale range of the map is greater than 0, and the length of the short side of the map is greater than or equal to twice the length of the long side of the AMR, the map meets the segmentation criteria. Then, the map is segmented using the quadtree algorithm. Specifically, the map is segmented into four equal-sized map tiles by dividing it into four tiles by the midpoint of the long side and the midpoint of the short side.

[0018] S15. Following the method in step S14, determine whether the segmented map patch meets the segmentation criteria. If the grayscale range of the map patch is greater than 0 and the length of the short side of the map patch is greater than or equal to twice the length of the long side of the AMR, it means that the map patch meets the segmentation criteria. Then, the map patch is segmented into four equal-sized map patches based on the quadtree algorithm. The specific segmentation method is as follows: the map patch is further divided into four equal-sized map patches by dividing it into four equal-sized map patches by dividing it into four equal-sized map patches by dividing it into four equal-sized map patches by dividing it into four equal-sized map patches.

[0019] The map segmentation process continues until all map tiles no longer meet the segmentation criteria, at which point the map segmentation ends, resulting in a variable resolution map.

[0020] After map segmentation, if the short side length of a map patch in the variable resolution map is less than twice the long side length of the AMR, and the grayscale range of the map patch is not 0, then the map patch is reassigned a grayscale value. The specific method is as follows: check the remaining pixels in the map patch except for the blank areas, calculate the average grayscale value of the remaining pixels, and assign all the grayscale values ​​of the map patch to the average grayscale value of the remaining pixels.

[0021] In step S12, the RGB map is converted to a grayscale map using the following formula:

[0022] Gray=(R*30+G*59+B*11) / 100;

[0023] Where Gray represents the grayscale value of a pixel, R represents the red value of a pixel, G represents the green value of a pixel, and B represents the blue value of a pixel.

[0024] In step S2, the construction of the decision model for multi-AMR path planning specifically includes the following steps:

[0025] S21, Establish the state space S = {s1, s2, s3, ..., s...} n ...}, where s n This represents the set of AMR states at time n, where each AMR state includes its current position and target position.

[0026] S22, Establish the action space A = {a1, a2, a3, ..., a...} n ...}, where a n This represents the set of all AMR actions at time n. The AMR actions include five types: up, down, left, right, and stationary.

[0027] S23, Establish the observation space O = {o1, o2, o3....o...} n ...}, where o n This represents the set of all AMR observation positions at time n. The observation position of an AMR refers to the position reached by the AMR after it takes action from its current position.

[0028] S24, Establish the reward space R = {r1, r2, r3, ..., r} n ...}, where r n This represents the set of rewards obtained by each AMR at time n. The reward obtained by an AMR refers to the reward value obtained by the AMR after taking an action from the current position to reach the observation position.

[0029] S25, establish the state transition probability function P under the variable resolution map. The state transition probability refers to the probability that the AMR will move from its current position to a neighboring map tile along the action direction. If there are m neighboring map tiles from the current position along the action direction, then the probability Pi of the AMR moving from its current position to the i-th neighboring map tile along the action direction is:

[0030]

[0031] Where m represents the number of neighboring map tiles that exist from the current position along the action direction; i represents the i-th neighboring map tile from the current position along the action direction, i = 1, ..., m; Pi represents the probability that the AMR moves from the current position to the i-th neighboring map tile along the action direction, and T is a constant; Qi represents the value generated by the AMR moving from the current position to the i-th neighboring map tile along the action direction, i.e., the Q value.

[0032] S26 yields the decision model for multi-AMR path planning as (S,A,O,R,P).

[0033] In step S24, the reward function, i.e., the formula for calculating the reward value, is as follows:

[0034]

[0035] Where r is the reward value obtained by the AMR after taking an action from the current position to reach the observation position; if the observation position reached by the AMR after taking an action from the current position is not the target position, then r = 0; if the observation position reached by the AMR after taking an action from the current position is the target position, it means that the AMR has completed the task, that is, the AMR has reached the target position from the initial position, and t is the time taken for the AMR to complete the task.

[0036] In step S24, the reward function, i.e., the formula for calculating the reward value, is as follows:

[0037]

[0038] Where r is the reward value obtained by the AMR after taking an action from the current position to reach the observation position; if the observation position reached by the AMR after taking an action from the current position is not the target position, then r = 0; if the observation position reached by the AMR after taking an action from the current position is the target position, it means that the AMR has completed the task, that is, the AMR has reached the target position from the initial position, t is the time spent by the AMR to complete the task, and c is the threshold of the reward function.

[0039] In step S3, path planning is performed based on the Q-learning algorithm and decision model. The specific process is as follows:

[0040] S31, initialize the state of each AMR, i.e., the initial position and the target position, and initialize the Q-value table and Q-learning algorithm parameters; wherein, the Q-value table is initialized to 0;

[0041] S32, begin this training session. Each AMR selects an action according to the ε-greedy strategy, calculated using the following formula:

[0042]

[0043] Where, argmax a Q(s n a) indicates that each AMR starts from the current state s n The action that moves the current position to a neighboring map tile and generates the maximum Q value, s n This represents the current state of each AMR, i.e., the state at time n; 'a' represents the action; 'a' represents the action. n This represents the action selected by each AMR at time n; 0 < ε < 1;

[0044] By combining the variable resolution map and the state transition probability function, the state s of each AMR at the next time step, i.e., the (n+1)th time step, is obtained. n+1 The location within is calculated using the following formula:

[0045]

[0046] Where Pi is the AMR from the current state s n The probability that the current position in the map will move along the direction of the action to the nearest neighboring map tile with the maximum Q value, i.e., the i-th neighboring map tile;

[0047] S33, Update the Q-value table using the Q-learning algorithm, the calculation formula is:

[0048] Q'(s n ,a n )=Q(s n ,a n )+α[r n +γmaxQ(s n+1 ,a)-Q(s n ,a n )];

[0049] Where Q'(s) n ,a n ) represents the current state s of each AMR after the update. n and current action a n The corresponding Q value, Q(s) n ,a n ) represents the current state s of each AMR. nand current action a n The corresponding Q value, maxQ(s) n+1 a) represents the next time-step state s of each AMR. n+1 The maximum Q-value generated by the action; α is the learning rate, γ is the discount factor; r n The reward obtained by each AMR at time n;

[0050] S34, determine the state of each AMR in the next time step. n+1 If the current position is the target position, i.e., whether the task has been completed, the training ends, the AMR state is reinitialized, but the updated Q value table is retained, and the process jumps to step S31 to proceed to the next training session. If the task has not been completed, the AMR continues to select an action to enter the next state in the manner of steps S31-S33, and the Q value table is updated until the task is completed and the training ends.

[0051] S35. If the updated Q-value table obtained after training does not change, then the path planning ends and the final Q-value table is obtained.

[0052] S36, According to the final Q-value table, AMR always moves from the current position to the neighboring map tile with the largest Q value, eventually reaching the target position and generating the optimal path.

[0053] In step S32, each AMR is in state s at the next time step. n+1 The positions of the AMRs must not conflict; that is, they cannot be the same position. If a conflict exists, each AMR must reselect an action and obtain its state s for the next time step. n+1 The location within it, until no conflict occurs.

[0054] The advantages of this invention are:

[0055] (1) The multi-AMR intelligent scheduling optimization method based on variable resolution reinforcement learning of the present invention can divide a large map with complex information into multiple small maps of different sizes, and realize the combination of variable resolution map and reinforcement learning. Since AMR needs to make an action judgment at each position and then select an action to enter the next position, the number of map tiles is reduced by using variable resolution map, thus reducing the number of action judgments. This effectively reduces the size of the Q-value table and the number of AMR action decisions, and can greatly improve the training efficiency of traditional reinforcement learning.

[0056] (2) The multi-AMR intelligent scheduling optimization method based on variable resolution reinforcement learning of the present invention addresses the curse of dimensionality caused by excessive spatial information in reinforcement learning when the scheduling environment map is large. By introducing the idea of ​​reducing unnecessary identical information, different types of items in the scheduling environment map are assigned different pixel values, and the map is converted into a grayscale image. At this time, the grayscale values ​​of items with different pixel values ​​are also different. The grayscale range of the map is calculated, and grayscale range standards and size standards are set. If the grayscale range standards and size standards are met, the map is segmented based on the quadtree algorithm, thereby avoiding meaningless segmentation. Each map block after segmentation has the same grayscale value, ensuring that the information is the same and contains only one type of item. Compared with each pixel having one piece of information, each map block of the present invention contains one piece of information, which greatly reduces the amount of map information. It is combined with reinforcement learning for intelligent scheduling of multi-AMR, which effectively avoids the curse of dimensionality caused by excessive spatial information in traditional reinforcement learning algorithms, and can improve the convergence speed and learning efficiency of the algorithm. Attached Figure Description

[0057] Figure 1 This is a flowchart of the multi-AMR intelligent scheduling optimization method based on variable resolution reinforcement learning of the present invention.

[0058] Figure 2 This is a schematic diagram of the variable resolution map based on the quadtree algorithm involved in this invention.

[0059] Figure 3 This is a schematic diagram of the AMR state transition involved in the present invention.

[0060] Figure 4 This is a schematic diagram illustrating the results of the variable resolution reinforcement learning multi-AMR intelligent scheduling optimization method of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Depend on Figure 1 As shown, the specific implementation steps of the multi-AMR intelligent scheduling optimization method based on variable resolution reinforcement learning of the present invention are as follows:

[0063] S1. Obtain the scheduling environment map information, perform planar map modeling using CAD software and known scheduling environment map information, assign different RGB pixel values ​​to different types of items in the map, then convert the map into a grayscale image, perform map segmentation based on the quadtree algorithm and considering the grayscale range of the map, segment into map blocks of different sizes, and obtain a variable resolution map.

[0064] S2, a decision model for multi-AMR path planning is constructed based on Markov decision process and variable resolution map. The decision model includes state, action, observation position, reward, and state transition function under variable resolution. The state includes the current position and target position of the AMR, where the current position refers to the current map tile and the target position refers to the target map tile.

[0065] S3 uses the Q-learning algorithm and variable resolution map in reinforcement learning to perform path planning, controls multiple AMRs to reach the target location under the variable resolution map, obtains rewards to update the Q value table, and realizes the scheduling process.

[0066] In step S1, the specific steps for obtaining the variable resolution map are as follows:

[0067] S11, determine the specific size of the scheduling environment and the items within it, including workbenches and various obstacles, and clarify the size and spatial relationship of each item; use CAD software to model a planar map of the known scheduling environment information, and assign different RGB pixel values ​​to different types of items using CAD software to obtain an RGB map of the scheduling environment.

[0068] S12 converts the RGB map to a grayscale map using the following formula:

[0069] Gray=(R*30+G*59+B*11) / 100;

[0070] Where Gray represents the grayscale value of a pixel, R represents the red value of a pixel, G represents the green value of a pixel, and B represents the blue value of a pixel; different types of items have different grayscale values.

[0071] S13, set the segmentation criteria for the map / map tile, including: grayscale range standard and size standard;

[0072] The grayscale range standard is: the grayscale range of the map / map tile is greater than 0; the grayscale range of the map / map tile is calculated as follows: the maximum grayscale value minus the minimum grayscale value in the map / map tile is the grayscale range of the map / map tile.

[0073] Size standard: The length of the shorter side of the map / map tile (i.e., the short side length of the map) is greater than or equal to twice the length of the longer side of the AMR (i.e., the long side length of the AMR).

[0074] S14. Determine if the map meets the segmentation criteria, including whether the grayscale range value and map size meet the grayscale range value standard and size standard respectively. If the grayscale range value of the map is 0, or the length of the short side of the map is less than twice the length of the long side of the AMR, it means that the map does not meet the segmentation criteria, and the map is not segmented. If the grayscale range value of the map is greater than 0, and the length of the short side of the map is greater than or equal to twice the length of the long side of the AMR, it means that the map meets the segmentation criteria, and the map is segmented based on the quadtree algorithm. The specific segmentation method is: segment the map by the midpoint of the long side and the midpoint of the short side respectively, and divide the map into four map tiles of equal size.

[0075] S15. Following the method in step S14, determine whether the segmented map patch meets the segmentation criteria. If the grayscale range of the map patch is greater than 0 and the length of the short side of the map patch is greater than or equal to twice the length of the long side of the AMR, it means that the map patch meets the segmentation criteria. Then, the map patch is segmented into four equal-sized map patches based on the quadtree algorithm. The specific segmentation method is as follows: the map patch is further divided into four equal-sized map patches by dividing it into four equal-sized map patches by dividing it into four equal-sized map patches by dividing it into four equal-sized map patches by dividing it into four equal-sized map patches.

[0076] Map segmentation ends when all map tiles no longer meet the segmentation criteria, i.e., the grayscale range of the map tiles is 0 or the length of the short side of the map tile is less than twice the length of the long side of the AMR, resulting in a variable resolution map.

[0077] S16. After map segmentation, if the short side length of a map patch is less than twice the long side length of the AMR, but the grayscale range of the map patch is not 0, then the map patch is reassigned a grayscale value, the remaining pixels in the map patch (excluding the blank area) are checked, the average grayscale value of the remaining pixels is calculated, and the grayscale value of the map patch is set to the average grayscale value of the remaining pixels.

[0078] Ultimately, this ensures that individual map tiles in a variable resolution map have the same grayscale value, meaning each tile contains only a single type of item, and the shorter side of each tile is greater than or equal to the longer side of the AMR (Advanced Mapping Scale). Figure 2 As shown. AMR can only move on map tiles in empty areas.

[0079] In step S2, a decision model for multi-AMR path planning is constructed, which specifically includes the following steps:

[0080] S21, Establish the state space S = {s1, s2, s3, ..., s...} n ...}, where sn This represents the set of AMR states at time n. The AMR state includes the current position and the target position of the AMR. The current position refers to the current location of the map tile, and the target position refers to the target map tile.

[0081] S22, Establish the action space A = {a1, a2, a3, ..., a...} n ...}, where a n This represents the set of all AMR actions at time n. The AMR actions include five types: up, down, left, right, and stationary.

[0082] S23, Establish the observation space O = {o1, o2, o3....o...} n ...}, where o n This represents the set of all AMR observation positions at time n. The observation position of an AMR refers to the position reached by the AMR after it takes action from its current position, i.e., the map tile it reaches.

[0083] S24, Establish the reward space R = {r1, r2, r3, ..., r} n ...}, where r n This represents the set of rewards obtained by each AMR at time n. The reward obtained by an AMR refers to the reward value obtained by the AMR after taking an action from the current position to reach the observation position.

[0084] The reward function, or the formula for calculating the reward value, is as follows:

[0085]

[0086] Alternatively, the reward function, i.e., the formula for calculating the reward value, is:

[0087]

[0088] Where r is the reward value obtained by the AMR after taking an action from the current position to reach the observation position. If the observation position reached by the AMR after taking an action from the current position is not the target position, then r = 0. If the observation position reached by the AMR after taking an action from the current position is the target position, it means that the AMR has completed the task, that is, the AMR has reached the target position from the initial position. t is the time spent by the AMR to complete the task. The reward value r is inversely proportional to the time spent t. The less time and the more efficient the path, the more reward is obtained. c is the threshold of the reward function. The value of c is the average time spent by the AMR to complete the same task. It can be further limited. If t is greater than the threshold c, a negative reward is obtained. If t is less than the threshold c, a positive reward is obtained.

[0089] S25, establish the state transition probability function P under the variable resolution map. The state transition probability refers to the probability that the AMR will transition from its current position to each neighboring map tile along the action direction; for example... Figure 3 As shown, if there are m neighboring map tiles along the action direction from the current position of the AMR, then the probability Pi of the AMR moving from the current position to the i-th neighboring map tile along the action direction is:

[0090]

[0091] Where m represents the number of neighboring map tiles that exist for the AMR from its current position along the direction of movement; i represents the i-th neighboring map tile of the AMR from its current position along the direction of movement, i = 1, ..., m;

[0092] Pi represents the probability that the AMR moves from its current position to the i-th neighboring map tile along the direction of its movement. If m = 1, that is, there is only one neighboring map tile where the AMR moves from its current position along the direction of its movement, then the probability that the AMR moves from its current position to that neighboring map tile along the direction of its movement is 1.

[0093] T is a constant, and the larger the value of T, the closer the probability Pi of AMR moving from the current position to the i-th neighboring map tile along the action direction is to random probability.

[0094] Qi represents the value generated by the AMR moving from its current position to the i-th neighboring map tile along the action direction, and is called the Q value. Initially, the state transition probability is a random probability, that is, the neighboring map tile along the action direction is randomly selected. As the number of learning times increases, the value generated by moving to the neighboring map tile that is close to the AMR's target position, i.e., the Q value, will increase. At this time, according to the calculation formula of the state transition probability function, the probability of selecting a neighboring map tile with a large Q value increases, and the probability of selecting other map tiles decreases.

[0095] S26, the decision model for multi-AMR path planning is established as (S,A,O,R,P).

[0096] In step S3, path planning is performed based on the Q-learning algorithm and the variable resolution map to control multiple AMRs to reach the target location under the variable resolution map. The specific steps are as follows:

[0097] S31, initialize the state of each AMR, i.e., the initial position and the target position, and initialize the Q-value table and Q-learning algorithm parameters; wherein, the Q-value table is initialized to 0;

[0098] S32, begin this training session. Each AMR selects an action according to the ε-greedy strategy, calculated using the following formula:

[0099]

[0100] Where, argmax a Q(s n a) indicates that each AMR starts from the current state s n The action that moves the current position to a neighboring map tile and generates the maximum Q value, s n This represents the current state of each AMR, i.e., the state at time n, where 'a' represents the action; a n This represents the action selected at the nth time step of each AMR;

[0101] 0 < ε < 1. ε is usually set to a very small value. If 1 - ε is 0.9, there is a 0.9 probability of selecting an action based on the Q value, and a 0.1 probability of selecting an action randomly. Initially, a random number between 0 and 1 is generated. If the random number is less than ε, an action is selected randomly; otherwise, the action with the largest Q value in the current state is selected.

[0102] By combining the variable resolution map and the state transition probability function, the state s of each AMR at the next time step, i.e., the (n+1)th time step, is obtained. n+1 The location within is calculated using the following formula:

[0103]

[0104] Where Pi is the AMR from the current state s n The probability that the current position in the map will be moved along the direction of action a to the nearest neighboring map tile with the maximum Q value, i.e., the i-th neighboring map tile;

[0105] Initially, a generated random number between 0 and 1 is used. If the random number is less than ε, a map is randomly selected; if the random number is greater than ε and less than ε + (1 - ε) * p1, then action a is selected. n The first map tile in the direction; if the random number is greater than ε+(1-ε)*p1 and less than ε+(1-ε)*P1+ε+(1-ε)*p2, then select the direction along action a. n The second map tile in each direction, and so on. The map tiles are sorted from top to bottom and from left to right.

[0106] To avoid path conflicts when multiple AMRs operate simultaneously, if the state s of each AMR in the next time step... n+1 If the AMRs are located in the same map tile, then the state s of each AMR in the next time step is reselected. n+1 The location within, until there is no conflict.

[0107] S33, use the Q-learning algorithm to update the Q-value table, using the maximum Q-value at the next time step and the Q-value at the current time step to update the current state s of each AMR. n and current action a nThe resulting Q value is calculated using the following formula:

[0108] Q'(s n ,a n )=Q(s n ,a n )+α[r n +γmaxQ(s n+1 ,a)-Q(s n ,a n )];

[0109] Among them, s n Let a be the state of each AMR at time n, i.e., the current state of each AMR; n Let r be the action of each AMR at time n, i.e., the current action of each AMR; n The value is the reward obtained by each AMR at time n; α is the learning rate; γ is the discount factor; maxQ(s) n+1 a) represents the (n+1)th time step, and the s values ​​of each AMR at the (n+1)th time step are... n+1 The maximum Q value produced by the action; Q(s) n ,a n ) represents the current state s of each AMR. n and current action a n The corresponding Q value, Q'(s) n ,a n ) represents the current state s of each AMR after the update. n and current action a n The corresponding Q value is...

[0110] S34, determine the state of each AMR in the next time step. n+1 If the current position is the target position, i.e., whether the task has been completed, the training ends, the AMR state is reinitialized, but the updated Q value table is retained, and the process jumps to step S31 to proceed to the next training session. If the task has not been completed, the AMR continues to select an action to enter the next state in the manner of steps S31-S33, and the Q value table is updated until the task is completed and the training ends.

[0111] Since the Q-value table is initialized to zero, during the first training session, each AMR will take a random action. That is, each AMR randomly selects an action from its initial position. If the action reaches the task point, it receives a reward, which updates the Q-value table. Then, the initial and target positions of the AMRs are reinitialized, but the previous Q-value table is retained, and training begins again. If the task point is not reached, the observation position of the AMR is obtained through observation, and then the action selection continues until the task point is reached. The reward is then received, the Q-value table is updated, and the initial and target positions of the AMRs are reinitialized, but the previous Q-value table is retained, and training begins again.

[0112] S35. If the updated Q-value table obtained after training does not change, then the path planning ends and the final Q-value table is obtained.

[0113] As the Q-value table is updated, due to the greedy strategy, each AMR (Autonomous Mobile Controller) has the highest probability of choosing to move to the nearest map tile that generates the maximum Q-value in each state. Since moving to a map tile closer to the target point yields a larger Q-value, the AMR will move towards the target point. Each time the target point is reached, a reward is obtained, the Q-value table is updated, and then retraining begins. Because the Q-value is related to the reward value, and the reward value is inversely proportional to the time (i.e., the shorter the AMR's time, the larger the reward value, resulting in a larger Q-value), the AMR will follow the path with the shortest time. When the AMR's time can no longer be shortened, path planning is complete.

[0114] S36. Based on the final Q-value table and canceling the greedy strategy, set the AMR to only move towards the neighboring map tile that generates the largest Q-value, record the position of each AMR at each moment, and thus form the optimal path.

[0115] Each AMR travels along its optimal path, determines whether it has reached the target location and completed the task, and checks the task list if the task is completed. If the task list is not empty, it selects the next task; otherwise, it returns to the initial position, thus realizing intelligent scheduling of multiple AMRs on a variable resolution map.

[0116] like Figure 4 As shown, Figure 4 The diagram illustrates the movement routes of two AMRs simultaneously performing scheduling tasks. Based on variable resolution reinforcement learning, the map tiles traversed by the AMRs are obtained. Each AMR uses the midpoint of the smallest tile it passes through as a reference point to perform straight-line movements (up, down, left, right) to accurately enter other map tiles. The AMRs' movements result in the route shown in the diagram, thus completing the intelligent scheduling task for multiple AMRs.

[0117] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-AMR intelligent scheduling optimization method based on variable resolution reinforcement learning, characterized in that, The method comprises the following steps: S1, a map of a scheduling environment is established, different pixel values are given to different objects in the map, the map is segmented based on a quadtree algorithm, and a variable resolution map is obtained; the variable resolution map comprises map blocks of different sizes, and each map block contains only one object or is blank; AMRs move on the blank map blocks; S2, a decision model for multi-AMR path planning is constructed, the decision model comprises states, actions, rewards, and state transition probabilities; the state refers to a position, and the position is represented by a map block; S3, path planning is performed according to a Q-learning algorithm and the decision model, and the multi-AMRs are controlled to reach target positions on the variable resolution map; In step S1, the specific steps of obtaining the variable resolution map are as follows: S11, the size of the scheduling environment and the objects in the scheduling environment are determined, and the size and spatial relationship of the objects are clarified; a CAD software is used to model a planar map of the scheduling environment, and different RGB pixel values are given to different objects by the CAD software to obtain an RGB map of the scheduling environment; S12, the RGB map is converted into a grayscale map; S13, the segmentation criteria of the map / map block are set, including a grayscale range value criterion and a size criterion; The grayscale range value criterion is that the grayscale range value of the map / map block is greater than 0; wherein the grayscale range value of the map / map block is the maximum grayscale value minus the minimum grayscale value in the map / map block; The size criterion is that the length of the shorter side (short side length) in the map / map block is greater than or equal to twice the length of the longer side (long side length) in the AMR; S14, it is judged whether the map meets the segmentation criteria; if the grayscale range value of the map is 0, or the short side length of the map is less than twice the long side length of the AMR, it is indicated that the map does not meet the segmentation criteria, and the map is not segmented; if the grayscale range value of the map is greater than 0, and the short side length of the map is greater than or equal to twice the long side length of the AMR, it is indicated that the map meets the segmentation criteria, and the map is segmented based on the quadtree algorithm; specifically, the map is segmented into four map blocks of equal size by segmenting the long side midpoint and the short side midpoint of the map; S15, it is judged whether the segmented map block meets the segmentation criteria in the manner of step S14; if the grayscale range value of the map block is greater than 0, and the short side length of the map block is greater than or equal to twice the long side length of the AMR, it is indicated that the map block meets the segmentation criteria, and the map block is segmented based on the quadtree algorithm; specifically, the map block is further segmented into four map blocks of equal size by segmenting the long side midpoint and the short side midpoint of the map block; Until all the map blocks do not meet the segmentation criteria, the segmentation of the map is ended, and the variable resolution map is obtained; After the map segmentation, if the short side length of a certain map block in the variable resolution map is less than 2 times the long side length of the AMR, and the gray scale range value of the map block is not 0, the gray scale value of the map block is re-assigned, and the specific assignment method is as follows: the gray scale average value of the remaining pixel points in the map block except the blank area is calculated, and the gray scale value of the map block is assigned as the gray scale average value of the remaining pixel points.

2. The multi-AMR intelligent scheduling optimization method based on variational resolution reinforcement learning according to claim 1, characterized in that, In step S12, the RGB map is converted into a gray scale map, and the calculation formula is: Gray=(R*30+G*59+B*11) / 100; wherein Gray represents the gray scale value of the pixel point, R represents the red value of the pixel point, G represents the green value of the pixel point, and B represents the blue value of the pixel point.

3. The multi-AMR intelligent scheduling optimization method based on variational resolution reinforcement learning according to claim 1, characterized in that, In step S2, the construction method of the decision model of the multi-AMR path planning specifically includes the following steps: S21, the state space S = {s1, s2, s3...s n ...}, wherein s n represents the set of each AMR state at the nth moment, the state of the AMR includes the current position and target position of the AMR; S22, the action space A = {a1, a2, a3...a n ...} is established, wherein a n represents the set of each AMR action at the nth moment, and the actions of the AMR include up, down, left, right, and static five; S23, establishing an observation space O = {o1, o2, o3... o n ...}, wherein o n represents a set of observation positions of each AMR at the nth moment, and the observation position of the AMR refers to the position reached by the AMR after performing an action from the current position; S24, establish reward space R = {r1, r2, r3... r n ...}, wherein r n represents the set of rewards obtained by each AMR at the nth moment, the reward obtained by the AMR refers to the reward value obtained by the AMR after moving from the current position to the observation position; S25, a state transition probability function P under the variable resolution map is established, and the state transition probability is the probability that the AMR moves from the current position to the adjacent map block along the action direction; if there are m adjacent map blocks along the action direction from the current position of the AMR, the probability Pi that the AMR moves from the current position to the i-th adjacent map block along the action direction is: wherein m represents the number of adjacent map blocks along the action direction from the current position of the AMR; i represents the i-th adjacent map block along the action direction from the current position of the AMR, i=1,..m; Pi represents the probability that the AMR moves from the current position to the i-th adjacent map block along the action direction, T is a constant; and Qi represents the value, i.e. Q value, generated when the AMR moves from the current position to the i-th adjacent map block along the action direction; S26, the decision model of the multi-AMR path planning is obtained as (S, A, O, R, P).

4. The multi-AMR intelligent scheduling optimization method based on variational resolution reinforcement learning according to claim 3, characterized in that, In step S24, the reward function, i.e. the reward value, is calculated according to the following formula: wherein r is the reward value obtained when the AMR reaches the observation position after performing the action from the current position; if the observation position reached by the AMR after performing the action from the current position is not the target position, r=0; if the observation position reached by the AMR after performing the action from the current position is the target position, it means that the AMR completes the task, i.e. the AMR reaches the target position from the initial position, and t is the time spent by the AMR to complete the task.

5. The multi-AMR intelligent scheduling optimization method based on variational resolution reinforcement learning according to claim 3, characterized in that, In step S24, the reward function, i.e. the reward value, is calculated according to the following formula: wherein r is the reward value obtained when the AMR reaches the observation position after performing the action from the current position; if the observation position reached by the AMR after performing the action from the current position is not the target position, r=0; if the observation position reached by the AMR after performing the action from the current position is the target position, it means that the AMR completes the task, i.e. the AMR reaches the target position from the initial position, t is the time spent by the AMR to complete the task, and c is the threshold value of the reward function.

6. The multi-AMR intelligent scheduling optimization method based on variational resolution reinforcement learning according to claim 3, characterized in that, In step S3, the path planning is performed according to the Q-learning algorithm and the decision model, and the specific process is as follows: S31, the state, i.e. the initial position and the target position, of each AMR is initialized, and the Q value table and the Q-learning algorithm parameters are initialized; wherein the Q value table is initialized as 0; S32, start the training, each AMR selects an action according to the ε-greedy strategy, and the calculation formula is: where argmax a Q(s n , a) represents the maximum Q value of each AMR moving from the current state s n to the adjacent map block and generating the action, s n represents the current state of each AMR, i.e., the state at the nth time; a represents an action; a n represents the action selected by each AMR at the nth time; 0 < ε < 1; By combining the variable resolution map and the state transition probability function, the state s of each AMR at the next time step, i.e., the (n+1)th time step, is obtained. n+1 The location within is calculated using the following formula: wherein Pi is the probability of the AMR moving from the current state s n to the neighboring tile with the maximum Q-value, i.e., the i-th neighboring tile, along the action direction from the current location in the tile s S33, update the Q value table using the Q-learning algorithm, and the calculation formula is: Q'(s n ,a n ) = Q(s n ,a n ) + a[r n + γ max Q(s n+1 ,a) - Q(s n ,a n )]; wherein Q'(s n ,a n ) is the updated Q value corresponding to the current state s n and the current action a n of each AMR, Q(s n ,a n ) is the Q value corresponding to the current state s n and the current action a n of each AMR, maxQ(s n+1 ,a) is the maximum Q value generated by the next time state s n+1 and the action of each AMR; a is the learning rate, g is the discount factor; and r n is the reward obtained by each AMR at the nth time. S34, judging whether the position of each AMR in the next time state s n+1 is the target position, i.e. whether the task is completed. If the task is completed, the current training is ended, the state of the AMR is reinitialized, but the updated Q value table is kept, and the next training is performed by jumping to step S31. If the task is not completed, the AMR continues to select actions to enter the next time state according to the mode of steps S31-S33, the Q value table is updated, until the task is completed, and the current training is ended. S35, if the updated Q value table obtained after training no longer changes, end the path planning, and obtain the final Q value table; S36, according to the final Q value table, the AMR always moves from the current position to the adjacent map block with the maximum Q value, and finally reaches the target position to generate an optimal path.

7. The multi-AMR intelligent scheduling optimization method based on variational resolution reinforcement learning according to claim 6, characterized in that, In step S32, the positions of the AMRs in the next time state s n+1 cannot conflict, i.e. cannot be the same, and if there is a conflict, each AMR reselects the action and obtains the position of each AMR in the next time state s n+1 again until there is no conflict.

Images