Area coverage method for agent multi-scale frontier perception reinforcement learning in unknown environment

Abstract

The invention provides an intelligent agent multi-scale frontier perception reinforcement learning area coverage method in an unknown environment, which comprises the following steps: constructing a plurality of intelligent agent local neighborhoods which take the position of an intelligent agent as a center and are increased in size and reduced in resolution to form a multi-scale map of an observation space, and each observation scale comprises obstacle information and coverage area information; and long-term planning and local planning of the intelligent agent are facilitated. A self-attention feature fusion mechanism is adopted, and map feature information contained in a multi-scale map and a sensor is fully extracted and utilized, so that an intelligent agent more pays attention to the leading edge of an uncovered area and holes in the covering process. An unknown leading edge reward is constructed, and the hole problem in the coverage process is further solved. A reinforcement learning method is used, other various rewards are combined, the intelligent agent is trained through course learning to complete area coverage of an unknown environment, and generalization of various environments is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of agent area coverage and robot technology, and particularly relates to an agent multi-scale front perception reinforcement learning area coverage method in an unknown environment. BACKGROUND

[0002] An agent is an entity with autonomous learning and exploration capabilities, which can perform complex tasks autonomously through planning, perception and control. In the research of agent-related technologies, area coverage is a very important and challenging problem, and is one of the important technologies for realizing automated services, intelligent manufacturing and smart society, and is a basic requirement for many practical tasks such as industry, agriculture, service industry, search and rescue, and security.

[0003] Although there are many methods to solve the agent area coverage problem at present, they mostly rely on known global map information, and it is difficult to complete the area coverage task in an unknown environment. Deep reinforcement learning methods can learn general intelligent behavior patterns, thereby realizing adaptability in various unstructured obstacles and unknown environments at the edge.

[0004] However, most of the existing area coverage algorithms based on reinforcement learning methods do not fully exploit their advantages in unknown environments, and mostly limit the action space to a finite discrete action grid, which greatly limits the diversity of exploration paths. On the other hand, most existing deep reinforcement learning methods use single-scale features, which do not fully extract map features, and the attention is relatively evenly distributed in the receptive field, which cannot focus on key areas that have not been covered, resulting in the agent easily appearing gaps in the coverage, i.e. the hole phenomenon. The coverage to make up for the holes often forms repeated paths, which will greatly waste energy and time costs, and reduce the efficiency of agent area coverage.

[0005] Therefore, in order to fully exploit the potential of the agent, it is imperative to explore efficient area coverage technology, which will improve the efficiency and generalization of the agent in the area coverage task. SUMMARY

[0006] To solve the problem of agent area coverage, the application provides an agent multi-scale front perception reinforcement learning area coverage method in an unknown environment, which uses reinforcement learning method to train the agent through curriculum learning to complete the area coverage of unknown environment and improve the generalization to various environments.

[0007] An agent multi-scale front perception reinforcement learning area coverage method in an unknown environment, comprising the following steps:

[0008] S1: Obtain a multi-scale map of a target region to be covered in which the agent is currently located, and a distance between the agent and an obstacle, wherein each scale map contains obstacle information and covered region information;

[0009] S2: Input the multi-scale map and the distance between the agent and the obstacle into a feature extraction module to obtain a map information feature f' f ;

[0010] S3: Input the map information feature f' f into a policy network to obtain a control signal f c ;

[0011] S4: The agent executes the control signal f c corresponding angular velocity and linear velocity, and then re-executes steps S1-S4 after performing an action, until the coverage rate of the target region reaches a set threshold.

[0012] Further, the training method of the policy network is as follows:

[0013] Step 1: Place the agent in a training map, and initialize network parameters of a policy network and a value network; wherein the network structures of the policy network and the value network are the same, and the network parameters are different;

[0014] Step 2: Obtain a multi-scale map of a target region to be covered in which the agent is currently located, and input the multi-scale map and a distance between the agent and an obstacle into a feature extraction module to obtain a map information feature f' f ;

[0015] Step 3: Input the map information feature f' f into a policy network to obtain a control signal f c ;

[0016] Step 4: The agent executes the control signal f c corresponding angular velocity and linear velocity, updates the environment state and the agent state, and obtains a reward function for feeding back the progress of the agent in covering the target region; meanwhile, a six-tuple data composed of the environment state and the agent state before updating, the environment state and the agent state after updating, the control signal f c , and the reward function is stored in an experience replay buffer; repeat steps 2-4 until the experience replay buffer is filled.

[0017] Step 5: sample a batch of six-tuple data from the experience replay buffer, wherein each six-tuple data corresponds to a value target value; meanwhile, for each six-tuple data, input the pre-updated and post-updated agent state into the value network to obtain a value prediction value; construct a loss function for representing the difference between the two values according to the value target values and the value prediction values of all six-tuple data, and then use the back propagation algorithm to iteratively update the network parameters of the value network according to the loss function;

[0018] Step 6: obtain the gradient weight in the gradient optimization algorithm adopted by the network parameters of the updated strategy network according to all the value prediction values obtained in step 5, and then update the network parameters of the strategy network through the gradient optimization algorithm;

[0019] Step 7: re-execute steps 2-6 using the strategy network and the value network whose network parameters have been updated until the proportion of the coverage area of the current agent on the training map to the coverable area of the entire training map is greater than the set coverage rate of the current training map;

[0020] Step 8: using the curriculum learning method, place the agent on the next training map with higher complexity, repeat steps 1-7 until the cumulative time step of the agent reaches the set maximum time step, and end the training.

[0021] Further, the reward function R for feeding back the progress of the agent covering the target area is obtained as follows:

[0022] R = λ new R new + λ energy R energy + λ collision R collision + λ variation R variation + λ frontier R frontier

[0023] wherein R new is the reward for covering a new area, λ new is the weight corresponding to R new , R energy is the movement cost penalty, λ energy is the weight corresponding to R energy , R collision is the obstacle collision penalty, λ collision is the weight corresponding to R collision , R variation is the variational reward, λ variation is the weight corresponding to R variation , R frontier is the unknown frontier reward, and λ frontier is the weight corresponding to Rfrontier The corresponding weight.

[0024] Furthermore, the reward for covering new areas is calculated as follows:

[0025] R new =A new / A max

[0026] Among them, A new The control signal f for executing the previous time step c The covered area of ​​the global coverage map obtained after the corresponding angular velocity and linear velocity, A max The control signal f for executing the previous time step c The theoretical maximum coverage area obtained after the corresponding angular velocity and linear velocity.

[0027] Furthermore, the movement cost penalty R energy Collision penalty R collision The method for determining is as follows:

[0028] Each time the agent takes an action, it will receive a fixed movement cost penalty R energy ;

[0029] Each time the agent collides with an obstacle, it will receive an obstacle collision penalty R collision .

[0030] Furthermore, the variational reward R variation The method for determining is:

[0031]

[0032] Where V(x) is the total variation of x, and the value of x is C t or C t-1 , C t The control signal f for executing the current time step c The global coverage map obtained after the corresponding angular velocity and linear velocity, C t-1 is the global coverage map obtained after executing the angular velocity and linear velocity corresponding to the control signal fc in the previous time step, v max is the maximum moving speed of the agent, Δt is the time step between two time steps; x i,j Indicates C t or C t-1 The pixel value at the upper position coordinate (i, j); x i+1.j Indicates C t or C t-1 The pixel value at the upper position coordinate (i+1, j); x i,j+1 Indicates C t or Ct-1 pixel value at the upper position coordinate (i, j+1).

[0033] Further, the unknown front reward R frontier is determined by:

[0034]

[0035] where d old is the distance of the agent to the nearest unknown front at the last time step, d new is the distance of the agent to the nearest unknown front at the current time step, S is an adaptive coefficient, S=1 when the proportion of the covered area is greater than a given threshold β, otherwise S=0, R fmax is a set maximum penalty reward.

[0036] Further, the feature extraction module comprises a feature extraction unit, a full connection layer, a convolution unit, an activation unit, and a multi-head attention mechanism feature fusion unit, wherein the feature extraction unit comprises a plurality of feature extraction subunits, each feature extraction subunit comprises a convolution subunit and an activation subunit, and the number of feature extraction subunits is the same as the number of maps in the multi-scale map.

[0037] The method for obtaining the map information feature f f is as follows:

[0038] Each map in the multi-scale map is input into a feature extraction subunit for feature extraction operation as follows:

[0039] f m =ReLU(Conv(r))

[0040] where r is any scale of map information, Conv(·) is the convolution operation of the convolution subunit, ReLU(·) is the activation operation of the activation subunit, and f m is the single-scale map feature obtained by performing feature extraction operation on any scale of map information.

[0041] The distance between the agent and the obstacle is sequentially operated by the full connection layer and the convolution unit to obtain the distance feature f s .

[0042] The single-scale map features f m corresponding to each map in the multi-scale map and the distance feature f s are combined into an overall feature f f .

[0043] The multi-head attention mechanism feature fusion unit is used to perform feature extraction and fusion on the overall feature f f to obtain the final map information feature f′f .

[0044] Further, the distance between the agent and the obstacle is measured by the laser radar carried on the agent.

[0045] Advantages:

[0046] 1. The application provides an agent multi-scale front perception reinforcement learning area coverage method in an unknown environment, a multi-scale map of an observation space formed by a plurality of local neighborhoods of the agent is constructed, each observation scale contains obstacle information and coverage area information, which facilitates long-term planning and local planning of the agent. A self-attention feature fusion mechanism is used to fully extract and utilize the multi-scale map and the map feature information contained by the sensor, so that the agent pays more attention to the front of the uncovered area and the holes in the coverage process.

[0047] 2. The application provides an agent multi-scale front perception reinforcement learning area coverage method in an unknown environment, a Soft Actor-critic (SAC) reinforcement learning method is used to train a policy network, and the two share a network structure but not weights; that is, the application constructs an end-to-end reinforcement learning area coverage algorithm, combines a curriculum learning method, constructs maps of different difficulties, and gradually increases the coverage completion rate requirement with the deepening of curriculum learning, improves the generality to different environments, realizes continuous action area coverage, has more available path selection, meets the real demand, and can adapt to various environmental characteristics.

[0048] 3. The application provides an agent multi-scale front perception reinforcement learning area coverage method in an unknown environment, an unknown front reward is constructed, so that the agent moves to the unknown front during coverage, fully considers the reward of covering a new area, and if the previous coverage area is overlapped, the reward will be reduced, resulting in a gap when the agent is close to the previously covered part, so that the coverage range has a hole problem, and the low efficiency problem of re-searching the hole to complete coverage after subsequent large-scale coverage is solved.

[0049] 4. The application provides an agent multi-scale front perception reinforcement learning area coverage method in an unknown environment, an experience replay area is set, the utilization rate of data is improved, the sampling frequency of the environment is reduced, and the learning speed is accelerated. DETAILED DESCRIPTION

[0050] Figure 1 The structure diagram of the method provided by the application is shown in the figure;

[0051] Figure 2 The multi-scale map constructed by the method provided by the application is shown in the figure;

[0052] Figure 3 A feature extraction and fusion schematic diagram using the method provided by the present application;

[0053] Figure 4 A course learning environment map schematic diagram constructed using the method provided by the present application;

[0054] Figure 5 A reinforcement learning training flowchart of the agent trained using the method provided by the present application. DETAILED DESCRIPTION

[0055] In order for those skilled in the art to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0056] The present application provides a multi-scale frontier perception deep reinforcement learning regional coverage method for an agent in an unknown environment, which extracts and fuses features of different scale maps and sensor information in the environment by constructing a feature fusion neural network architecture based on an attention mechanism, is more conducive to the agent to focus on uncovered areas and holes, and further solves the problem that the agent produces coverage holes and needs a very high time cost to complete complete coverage in the regional coverage process, Figure 1 The structure schematic diagram of the regional coverage method proposed by the present application is shown.

[0057] As Figure 1 shown, a multi-scale frontier perception reinforcement learning regional coverage method for an agent in an unknown environment includes the following steps:

[0058] S1: Obtain a multi-scale map of a target region to be covered in which the agent is located at a current time step, and a distance between the agent and an obstacle, wherein each scale map contains obstacle information and covered region information; the distance between the agent and the obstacle is measured by a laser radar carried on the agent;

[0059] It should be noted that the performance parameters of the sensor of the agent used by the present application include an observation angle α, a detection range l, and an agent coverage radius r; as Figure 3 shown, the present application constructs a multi-scale map of a local neighborhood observation space of the agent with multiple sizes increasing and resolutions decreasing, each map containing obstacle information and covered region information; for the multi-scale map, n scales are used, and the map spans a square with a side length of m meters; during the movement of the agent, the obstacle information and the covered region information are updated by the sensor.

[0060] S2: input the multi-scale map and the distance between the agent and the obstacle into a feature extraction module to obtain map information features f′f ;

[0061] S3: obtaining the map information feature f' f inputting the policy network to obtain the control signal f c ;

[0062] S4: the agent executes the control signal f c corresponding angular velocity and linear velocity, and after performing an action, steps S1-S4 are re-executed until the coverage rate of the target area reaches the set threshold.

[0063] It should be noted that the Soft Actor-critic (SAC) reinforcement learning method is used to train the policy network. Specifically, the SAC algorithm trains the Actor (policy network) and Critic (value network). The goal of the Actor network is to learn how to take actions that maximize long-term rewards given a state. The Critic network is two independent Q networks, and the goal is to estimate the value of actions in a given state, which guides the agent to effectively explore and cover the target area in combination with the reward mechanism, and enables the Q function to more accurately predict the value of actions. The policy network and the Q network share the network structure but not the weights.

[0064] During the movement of the agent, the overall feature is obtained by extracting and fusing the map and sensor information. The policy network in the SAC algorithm is processed through a linear layer, and then the control signal f c :

[0065] f c = ReLU(Linear(f' f )

[0066] where f' f is the overall feature obtained by extraction, which fuses multi-scale map information and sensor information.

[0067] The reward mechanism guides the agent to effectively explore and cover the target area, the policy network is responsible for generating actions, the agent executes the actions, explores the environment and obtains reward information; the reward function serves as an evaluation standard to provide a guidance signal for the learning of the policy network, so that the policy network gradually converges to the optimal policy that can obtain the maximum reward. The value network evaluates the value of actions and works cooperatively with the policy network to optimize the decision-making process of the agent.

[0068] Based on this, as shown in FIG. 1, the steps of training the policy network are as follows: Figure 5

[0069] ​Step 1: placing the agent in the training map, and initializing the network parameters of the policy network and the value network; wherein the network structures of the policy network and the value network are the same, and the network parameters are different;

[0070] Step 2: obtaining a multi-scale map of a target region to be covered in which the agent is currently located in the time step, inputting the multi-scale map and the distance between the agent and the obstacle into the feature extraction module to obtain a map information feature f' f ;

[0071] Step 3: inputting the map information feature f' f into the policy network to obtain a control signal f c ;

[0072] Step 4: after the agent executes the corresponding angular velocity and linear velocity of the control signal f c of the next time step, updating the environment state and the agent state, and obtaining a reward function for feeding back the progress of the agent in covering the target region; meanwhile, storing a six-tuple data composed of the environment state and the agent state before updating, the environment state and the agent state after updating, the control signal f c , and the reward function into an experience replay buffer; repeating steps 2-4 until the experience replay buffer is filled; the experience replay buffer ensures that the agent can learn from previous experiences, so that the policy network and the value network can be updated more stably and effectively.

[0073] It should be noted that the environment state includes a map environment, a position of the agent, covered region information, and uncovered region information, wherein the map environment includes environment boundaries, obstacle distribution and the like; the agent state includes self-position information, environment information that can be observed within a self-observation angle range, and self-angular velocity and linear velocity;

[0074] Step 5: sampling a batch of six-tuple data from the experience replay buffer, wherein each six-tuple data corresponds to a value target value; meanwhile, for each six-tuple data, inputting the agent state before updating and the agent state after updating into the value network to obtain a value prediction value; constructing a loss function for representing the difference between the two values according to the value target value and the value prediction value corresponding to all six-tuple data, and then iteratively updating the network parameters of the value network according to the loss function by using a back propagation algorithm;

[0075] It should be noted that obtaining the corresponding value target value according to the six-tuple data belongs to the prior art, and the present application will not be described here.

[0076] Step 6: Based on all the value prediction values ​​obtained in step 5, the gradient weights in the gradient optimization algorithm used to update the network parameters of the policy network are obtained, and then the network parameters of the policy network are updated using the gradient optimization algorithm;

[0077] It should be noted that updating the value network makes the value prediction value corresponding to the current six-tuple data closer to the value target value, and using the policy gradient algorithm to update the policy network so that the action output by the policy network can maximize the reward including the entropy term.

[0078] Step 7: Use the updated policy network and value network to re-execute steps 2 to 6 until the ratio of the current agent's coverage area on the training map to the total coverage area of ​​the training map is greater than the coverage ratio set for the current training map.

[0079] Step 8: Use the curriculum learning method to place the agent on the next more complex training map and repeat steps 1 to 7 until the accumulated time steps of the agent reach the set maximum time steps, and then end the training.

[0080] For example, Figure 2 As shown, a training map with gradually complex obstacles, gradually larger maps, and increasing difficulty is constructed as a training environment for reinforcement learning. In other words, the present invention uses a course learning method to gradually increase the difficulty of the map for the agent to cover and the requirements for the agent. As the course learning deepens, the map difficulty and the requirements for coverage completion rate increase step by step. The initial map level is 1. When the proportion of the area covered by the agent reaches a, it is considered that the current map learning is completed. Then the level is increased, the map difficulty is increased, the map size is increased, and the obstacle difficulty is increased. When the level is greater than or equal to n, the proportion of the area covered by the agent is required to be b, so that the agent can gradually learn to cover maps with higher difficulty during the training process, thereby improving the generalization of area coverage.

[0081] It should be noted that to improve coverage efficiency, during training, the present invention can periodically monitor the agent's performance in the environment to observe the effectiveness of area coverage. Based on the monitoring results, hyperparameters can be adjusted and the environment configuration refined to improve training results. A log is maintained throughout the training process, along with information on successful coverage.

[0082] After training is complete, the model is applied to the environment and run for a period of time to evaluate the agent's actual performance. Training stops when the agent successfully completes the area coverage task. In this way, the SAC algorithm teaches the agent to consistently and effectively cover space within a given area while optimizing its strategy through a balance between exploration and learning. This approach not only optimizes coverage efficiency but also maintains a certain degree of flexibility and robustness in the action selection process.

[0083] Further, the method for obtaining the reward function R for the feedback agent to cover the target area is:

[0084] R = λ new R new + λ energy R energy + λ collision R collision + λ variation R variation + λ frontier R frontier

[0085] wherein R new is the reward for covering a new area, λ new is the corresponding weight of R new , R energy is the moving cost penalty, i.e., the agent will be punished for each action, λ energy is the corresponding weight of R energy , R collision is the obstacle collision penalty, i.e., the agent will be punished when colliding with an obstacle, λ collision is the corresponding weight of R collision , R variation is the variational reward, λ variation is the corresponding weight of R variation , R frontier is the unknown frontier reward, and λ frontier is the corresponding weight of R frontier .

[0086] The method for calculating the reward for covering a new area is:

[0087] R new = A new / A max

[0088] wherein A new is the area of the covered region of the global coverage map obtained after the corresponding angular velocity and linear velocity of the control signal f c of the previous time step are executed, and A max is the maximum coverage area theoretically obtained after the corresponding angular velocity and linear velocity of the control signal f c of the previous time step are executed.

[0089] The determination method of the moving cost penalty R energy and the obstacle collision penalty R collision is as follows:

[0090] The agent will obtain a fixed moving cost penalty R energy; the agent receives a collision penalty R every time it collides with an obstacle collision . It is noted that the values of the movement cost penalty R energy and the obstacle collision penalty R collision are set by the specific situation of the actual training process, and optionally, the movement cost penalty R energy is -0.1 and the obstacle collision penalty R collision is -10.

[0091] To solve the problem that the agent is reluctant to overlap the previous area, leaving a small gap when approaching the previously covered part, resulting in holes or stripes in the coverage range, resulting in the need for more cost to search and cover the holes; at the same time, at time step t, there is a global coverage map C t , and the incremental reward is based on the difference between the total variation at the current time step and the previous time step. If the total variation decreases, a positive reward is given, otherwise a penalty is given. The incremental reward is measured by the maximum possible increase in total variation within a time step, which is twice the maximum driving distance within a time step. Therefore, to solve the problem of existing holes, the constructed variation reward R variation is as follows:

[0092]

[0093] where V(x) is the total variation of x, x is C t or C t-1 , C t is the global coverage map obtained after executing the angular velocity and linear velocity corresponding to the control signal f c at the current time step, and C t-1 is the global coverage map obtained after executing the angular velocity and linear velocity corresponding to the control signal f c at the previous time step, v max is the maximum movement speed of the agent, and Δt is the time step length between two time steps; x i,j represents the pixel value at the position coordinate (i, j) of C t or C t-1 ; x i+1.j represents the pixel value at the position coordinate (i+1, j) of C t or C t-1 ; and x i,j+1 represents the pixel value at the position coordinate (i, j+1) of C t or C t-1 . That is, the total variation of C t or C t-1 is given by calculating the difference between each pixel and its horizontal and vertical neighbors, and accumulating these differences, which is used to evaluate C t or C t-1Smoothness or degree of variation throughout the image.

[0094] In order to further solve the problem that the agent generates coverage holes and needs a high time cost to complete complete coverage in the area coverage process, the unknown front reward R frontier As follows:

[0095]

[0096] Where d old is the distance of the agent from the nearest unknown front at the previous time step, d new is the distance of the agent from the nearest unknown front at the current time step, S is an adaptive coefficient, S = 1 when the proportion of the covered area is greater than a given threshold β, otherwise S = 0, R fmax is the maximum penalty reward set. By constructing an adaptive unknown front reward, the agent is encouraged to cover the location of the hole when the coverage rate is high, improving the efficiency of coverage.

[0097] Further, as shown in Figure 4 In order to fully extract and utilize the features contained in the map information, the present application first extracts features from multi-scale maps and laser radar observation information through convolution operation, then fuses features through multi-head attention mechanism, and finally outputs the control signal of the agent. Among them, the linear speed of the agent is constant, and the control signal is the steering direction of the agent, which is a continuous action.

[0098] Specifically, the feature extraction module used in the present application includes a feature extraction unit, a fully connected layer, a convolution unit, an activation unit, and a multi-head attention mechanism feature fusion unit. The feature extraction unit includes a plurality of feature extraction subunits, and each feature extraction subunit includes a convolution subunit and an activation subunit. The number of feature extraction subunits is the same as the number of maps in the multi-scale map.

[0099] The method for obtaining the map information feature f f is as follows:

[0100] Step one, each map in the multi-scale map is input into a feature extraction subunit for feature extraction operation as follows:

[0101] f m = ReLU(Conv(r))

[0102] Where r is any scale of map information, each type of map can be used as a separate input data r, Conv(·) is the convolution operation of the convolution subunit, ReLU(·) is the activation operation of the activation subunit, and f m is the single-scale map feature obtained by performing feature extraction operation on any scale of map information.

[0103] That is, for the n different precision and size of the map contained in the multi-scale map information, the application first extracts the features of the input map information through convolution operation; after repeating the convolution operation for many times, the n scale map features are converted into fixed dimension features f m .

[0104] Step two, the distance between the agent and the obstacle is obtained after the operation of the full connection layer and the convolution unit in turn, and the distance feature f s ;

[0105] That is, for the sensor information, the application directly adopts full connection and ReLU activation function to obtain the contained features f s .

[0106] Step three, the single scale map features f m and the distance features f s of each map in the multi-scale map are combined into the overall feature f f ;

[0107] Step four, the multi-head attention mechanism feature fusion unit is used to extract and fuse the overall feature f f , and the final map information feature f′ f is obtained.

[0108] That is, for the obtained multi-scale map features and sensor information features, the multi-head attention mechanism is used for feature fusion to obtain the overall feature representation f′ f , which is as follows:

[0109] f f =Concat(f m ,f s )

[0110] f′ f =MultiheadAttention(f f ,f f ,f f )

[0111] Wherein, MultiheadAttention is a multi-head self-attention mechanism, and Q, K and V required in the multi-head self-attention mechanism are input map information features f f .

[0112] Next, in combination with specific agent parameters, a multi-scale front perception reinforcement learning area coverage method for an agent in an unknown environment provided by the application will be described in detail.

[0113] (1) Confirm the parameters of the mobile agent. The sensor carried by the agent is a laser radar with an observation angle α of 360° horizontally and a detection range l of 3.5 meters. The agent coverage radius r is 0.15 meters.

[0114] (2) Construct training maps with increasingly complex obstacles, gradually larger maps, and increasing difficulty levels, totaling 58 maps, as the training environment for reinforcement learning.

[0115] (3) The agent's position is randomly initialized within the map. Combined with the agent's observation information, a multiscale map of the agent's local neighborhood is constructed, with increasing size and decreasing resolution, to form the observation space. Each map contains information about obstacles and coverage areas. For the multiscale map, four scales are used, with a resolution of 32 × 32 pixels. The finest scale is 0.0375 meters per pixel, and the map spans a square with a side length of 76.8 meters.

[0116] (4) For the four maps of different precision and size contained in the multi-scale map information, the input map information is first subjected to feature extraction through convolution operation:

[0117] f m =ReLU(Conv(f))

[0118] Among them, Conv represents the convolution operation, using a 3×3 convolution kernel. After repeating the convolution operation four times, the map features of the four scales are converted into fixed-dimensional features f through full connection. m For sensor information, directly use full connection and ReLU activation function to obtain the included features f s .

[0119] For the obtained multi-scale map features and sensor information features, a multi-head attention mechanism is used to perform feature fusion to obtain the overall feature representation f′ f , as follows:

[0120] f f =Concat(f m ,f s )

[0121] f′ f =MultiheadAttention(f f ,f f ,f f )

[0122] Among them, MultiheadAttention is a multi-head self-attention mechanism, and the required Q, K, and V are all input map information features f f .

[0123] (5) The reward function is constructed as follows, including the reward R for covering new areas new , the penalty R for obstacle collision collision , the penalty R for moving cost energy , the reward R for variation variation , and the reward R for unknown frontier frontier :

[0124] R = λ new R new + λ energy R energy + λ collision R collision + λ variation R variation + λ frontier R frontier

[0125] wherein λ is the weight coefficient of each reward.

[0126] During training, λ new is set to 1, λ variation is set to 1, λ collision is set to 1, R collision is set to -10, the threshold β is set to 0.5, when the proportion of covered areas is greater than 50%, λ frontier is set to 1, λ energy is set to 1, and R energy is set to -0.1. R fmax is set to 5, v max = 0.26 m / s, and Δt = 0.5 s.

[0127] (6) The Soft Actor-critic reinforcement learning method is used for training, and the Actor (policy network) and Critic (value network) are trained, wherein the Actor and Critic networks share the structure but not the weights.

[0128] During the movement of the agent, the environment constructed in (2) is used, and the state of the agent in the environment is obtained through (3). The overall features are obtained by extracting and fusing the map and lidar information through (4). The policy network in the SAC algorithm is processed through a linear layer, and then the control signal f c ,

[0129] f c = ReLU(Linear(f′ f )

[0130] wherein f′ fFor the whole feature extracted in step four, the multi-scale map information and sensor information are fused. The reward mechanism in (5) is used to guide the agent to effectively explore and cover the target area. The experience replay area size is set to 500000. The learning rate is set to 10 -5 , the batch size is 256, the size of the experience replay area is 500000, the reward discount factor is 0.99, the training step is 1000000, and the optimizer is Adam optimizer.

[0131] The curriculum learning method is used to gradually improve the difficulty of the map to be covered by the agent and the requirements on the agent. With the deepening of curriculum learning, the map difficulty and the requirement on the coverage completion rate are gradually increased. The initial map level is 1, and when the proportion of the area covered by the agent reaches 90%, it is considered that the current map learning is completed, then the level is increased, the map difficulty is increased, the map size is increased, and the obstacle difficulty is increased. When the level is greater than or equal to n=5, the proportion of the area covered by the agent is required to be 99%, and the agent is required to learn to cover higher difficulty maps in the training process, so as to improve the generalization of the area coverage.

[0132] Of course, the present application can have other various embodiments, and those skilled in the art can certainly make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application. However, these corresponding changes and modifications should belong to the protection scope of the claims attached to the present application.

Claims

1. A method for multi-scale frontier perception reinforcement learning area coverage of an intelligent agent in an unknown environment, characterized by: The following steps are involved: S1: Obtain a multi-scale map of the target area to be covered at the current time step of the agent, and the distance between the agent and the obstacle. Each scale map contains information about obstacles and covered areas. S2: Input the multi-scale map and the distance between the agent and the obstacle into the feature extraction module to obtain the map information feature f′ f ; S3: map information feature f′ f Input the policy network to get the control signal f for the next time step c ; S4: The agent executes the control signal f for the next time step c After executing an action once, steps S1 to S4 are executed again until the coverage of the target area reaches the set threshold.

2. The method for multi-scale frontier perception reinforcement learning area coverage of an intelligent agent in an unknown environment according to claim 1, characterized in that: The training method of the policy network is: Step 1: Place the agent in the training map and initialize the network parameters of the policy network and value network. The policy network and value network have the same network structure but different network parameters. Step 2: Obtain a multi-scale map of the target area to be covered at the current time step of the agent, input the multi-scale map and the distance between the agent and the obstacle into the feature extraction module to obtain the map information feature f′ f ; Step 3: Map information feature f′ f Input the policy network to get the control signal f for the next time step c ; Step 4: The agent executes the control signal f for the next time step c After the corresponding angular velocity and linear velocity are obtained, the environment state and the agent state are updated, and the reward function of the feedback agent's progress in covering the target area is obtained; at the same time, the environment state and agent state before the update, the environment state and agent state after the update, and the control signal f are calculated. c , the six-tuple data consisting of the reward function is stored in the experience replay buffer; repeat steps 2 to 4 until the experience replay buffer is filled; Step 5: Sample a batch of six-tuple data from the experience replay buffer, where each six-tuple data corresponds to a value target value. At the same time, for each six-tuple data, input the agent state before and after the update into the value network to obtain the value prediction value. Based on the value target value and the value prediction value corresponding to all six-tuple data, a loss function is constructed to represent the difference between the two values. Then, the backpropagation algorithm is used to iteratively update the network parameters of the value network according to the loss function. Step 6: Based on all the value prediction values ​​obtained in step 5, the gradient weights in the gradient optimization algorithm used to update the network parameters of the policy network are obtained, and then the network parameters of the policy network are updated using the gradient optimization algorithm; Step 7: Use the updated policy network and value network to re-execute steps 2 to 6 until the ratio of the current agent's coverage area on the training map to the total coverage area of ​​the training map is greater than the coverage ratio set for the current training map. Step 8: Use the curriculum learning method to place the agent on the next more complex training map and repeat steps 1 to 7 until the accumulated time steps of the agent reach the set maximum time steps, and then end the training.

3. The method for multi-scale frontier perception reinforcement learning area coverage of an intelligent agent in an unknown environment according to claim 2, characterized in that: The reward function R that provides feedback on the agent's progress in covering the target area is obtained as follows: R=λ new R new +λ energy R energy +λ collision R collision +λ variation R variation +λ frontier R frontier Among them, R new is the reward for covering new areas, λ new R new The corresponding weight, R energy is the movement cost penalty, λ energy R energy The corresponding weight, R collision is the obstacle collision penalty, λ collision R collision The corresponding weight, R variation is the variational reward, λ variation R variation The corresponding weight, R frontier is the unknown frontier reward, λ frontier R frontier The corresponding weight.

4. The method for multi-scale frontier perception reinforcement learning area coverage of an intelligent agent in an unknown environment according to claim 3, characterized in that: The reward for covering new areas is calculated as follows: R new =A new / A max Among them, A new The control signal f for executing the previous time step c The covered area of ​​the global coverage map obtained after the corresponding angular velocity and linear velocity, A max The control signal f for executing the previous time step c The theoretical maximum coverage area obtained after the corresponding angular velocity and linear velocity.

5. The method for multi-scale frontier perception reinforcement learning area coverage of an intelligent agent in an unknown environment as claimed in claim 3, characterized in that: Movement cost penalty R energy Collision penalty R collision The method for determining is as follows: Each time the agent takes an action, it will receive a fixed movement cost penalty R energy ; Each time the agent collides with an obstacle, it will receive an obstacle collision penalty R collision .

6. The method for multi-scale frontier perception reinforcement learning area coverage of an intelligent agent in an unknown environment as claimed in claim 3, characterized in that: Variational reward R variation The method to determine is: Where V(x) is the total variation of x, and the value of x is C t or C t-1 , C t The control signal f for executing the current time step c The global coverage map obtained after the corresponding angular velocity and linear velocity, C t-1 The control signal f for executing the previous time step c The global coverage map obtained after the corresponding angular velocity and linear velocity, v max is the maximum moving speed of the agent, Δt is the time step between two time steps; x i,j Indicates C t or C t-1 The pixel value at the upper position coordinate (i, j); x i+1.j Indicates C t or C t-1 The pixel value at the upper position coordinate (i+1, j); x i,j+1 Indicates C t or C t-1 The pixel value at position (i, j+1) on the image.

7. The method for multi-scale frontier perception reinforcement learning area coverage of an intelligent agent in an unknown environment as claimed in claim 3, characterized in that: Unknown Frontier Reward R frontier The method to determine is: Among them, d old is the distance between the agent and the nearest unknown frontier at the previous time step, d new is the distance between the agent and the nearest unknown frontier at the current time step, S is the adaptive coefficient, when the proportion of the covered area is greater than the given threshold β, S = 1, otherwise S = 0, R fmax is the maximum penalty reward set.

8. The method for multi-scale frontier perception reinforcement learning area coverage of an intelligent agent in an unknown environment according to claim 1 or 2, characterized in that: The feature extraction module includes a feature extraction unit, a fully connected layer, a convolution unit, an activation unit, and a multi-head attention mechanism feature fusion unit, wherein the feature extraction unit includes multiple feature extraction subunits, and each feature extraction subunit includes a convolution subunit and an activation subunit, and the number of feature extraction subunits is the same as the number of maps in the multi-scale map; Map information features f The method to obtain is as follows: Each map in the multi-scale map is input into a feature extraction subunit for feature extraction as follows: f m =ReLU(Conv(r)) Among them, r is the map information of any scale, Conv(·) is the convolution operation of the convolution subunit, ReLU(·) is the activation operation of the activation subunit, and f m Single-scale map features are obtained by performing feature extraction operations on map information of any scale; The distance between the agent and the obstacle is processed through the fully connected layer and the convolution unit in sequence to obtain the distance feature f s ; The single-scale map features f corresponding to each map in the multi-scale map m And the distance feature f s Combined into the overall feature f f ; The multi-head attention mechanism feature fusion unit is used to fusion the overall feature f f Perform feature extraction and fusion to obtain the final map information feature f′ f .

9. The method for multi-scale frontier perception reinforcement learning area coverage of an intelligent agent in an unknown environment as claimed in claim 1, characterized in that: The distance between the agent and the obstacle is measured by the lidar carried on the agent.

Images