Search Path Planning Method, Device and Evaluation Method Based on Deep Reinforcement Learning

Abstract

This application relates to the technical field of search path planning, and specifically relates to a search path planning method, device and evaluation method based on deep reinforcement learning. The search path planning method includes the following steps: based on search environment parameters, an environment matrix regarding detection capabilities is established, and a starting point is set; the trained neural network outputs Q-values corresponding to each action based on the current position of the agent, the reward function and the environment matrix, and the reward function is designed according to the current position and its distance from the target point, the detection capabilities of the neighborhood of the current position, and the number of unvisited areas; the agent executes each next action according to the action corresponding to the maximum Q-value. It can solve the problem in the prior art that the reinforcement learning algorithm often has difficulty in simultaneously considering efficiency and stability when facing complex tasks, which may lead to too long path planning time or the result deviating from the global optimum, affecting the actual application effect.

Description

Technical Field

[0001] This application relates to the technical field of search path planning, and particularly relates to a search path planning method, device, and evaluation method based on deep reinforcement learning. Background Art

[0002] Path planning in complex environments is a core issue in technical applications such as UAV inspections, robot navigation, and automated search and rescue.

[0003] Existing path planning technologies are mainly divided into two categories: traditional algorithms and reinforcement learning methods based on artificial intelligence. Traditional path planning algorithms (such as the A* algorithm, Dijkstra algorithm, etc.) optimize paths through static environment modeling. Although the computational efficiency is relatively high, their adaptability and robustness are poor in dynamic environments or large-scale complex scenarios. In contrast, due to its adaptive learning ability and efficient processing of dynamic environments, deep reinforcement learning has gradually become an important research direction in the field of path planning.

[0004] There are still the following two major problems in the existing technologies in practical applications: First, most reinforcement learning path planning methods aim to find the optimal path and lack the ability to cover the target area. Especially when comprehensive information collection of the environment or large-scale target search is required, there are obvious limitations and it is easy to fall into local optima. Second, existing reinforcement learning algorithms often have difficulty balancing efficiency and stability when facing complex tasks, which may lead to too long path planning time or results deviating from the global optimum, affecting the actual application effect. Summary of the Invention

[0005] This application provides a search path planning method, device, and evaluation method based on deep reinforcement learning, which can solve the problem in the existing technologies that reinforcement learning algorithms often have difficulty balancing efficiency and stability when facing complex tasks, which may lead to too long path planning time or results deviating from the global optimum, affecting the actual application effect.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is:

[0007] On the one hand, the present invention provides a search path planning method based on deep reinforcement learning, including the following steps:

[0008] Based on the search environment parameters, establish an environment matrix regarding the detection ability and set a starting point;

[0009] The trained neural network outputs the Q values corresponding to each action based on the current position of the intelligent agent, the reward function, and the environment matrix, and the reward function is designed according to the current position and its distance from the target point, the detection ability of the neighborhood of the current position, and the number of unvisited areas;

[0010] The agent executes each next action according to the action corresponding to the maximum Q value.

[0011] In some alternative solutions, the reward function is designed based on the current position, its distance from the target point, the detection ability of the neighborhood of the current position, and the number of unvisited areas, including:

[0012] Design the current position detection ability reward according to the current position;

[0013] Design the target distance reward according to the distance between the current position and the target point;

[0014] Design the neighborhood detection ability reward according to the detection ability of the neighborhood of the current position;

[0015] Design the unvisited area reward according to the number of unvisited areas;

[0016] Design the reward function according to the current position detection ability reward, target distance reward, neighborhood detection ability reward, and unvisited area reward.

[0017] In some alternative solutions, the reward function is:

[0018] ;

[0019] Among them, represents the current position of the agent, is the multi-factor immediate reward corresponding to the position at S after executing the action a , is the current position detection ability reward, is the target distance reward, is the neighborhood detection ability reward, is the unvisited area reward, is the target distance reward weight, is the neighborhood detection ability reward weight, is the unvisited area reward weight.

[0020] In some alternative solutions, the current position detection ability reward , among which, is the detection ability value of the current position, and i and j are the row and column numbers of the environment matrix respectively;

[0021] The target distance reward , among which, is the Euclidean distance between the current position of the agent and the target point, is the weight parameter of the distance reward, is the current position of the agent, is the target point position;

[0022] Neighborhood Detection Ability Reward , where is the neighborhood range of the current position , n is the sequence number of the current execution time step, is the position of the agent at time step n detection ability;

[0023] Unvisited Area Reward , where is the hyperparameter for adjusting the reward, is the multi-factor immediate reward obtained according to the reward function at time step t, is the record of whether the position is an unvisited area, t is the time step number, is the current position detection ability value.

[0024] In some alternative solutions, when the agent executes an action according to an action instruction, if the execution of the action will cause the agent to exceed the boundary condition or enter a searched position, the agent remains at the current position unchanged.

[0025] In some alternative solutions, before the neural network outputs the Q value corresponding to each action based on the current position, reward function, and environmental matrix of the agent, the neural network is further trained to obtain a trained neural network. During training, each update of the neural network parameters includes the following steps:

[0026] Determine the target Q value corresponding to the maximum reward value according to the reward values of each action;

[0027] Determine the mean square error loss value according to the target Q value;

[0028] Update the parameters of the neural network according to the mean square error loss value.

[0029] In some alternative solutions, according to the Bellman equation , determine the target Q value; where is the multi-factor immediate reward obtained according to the reward function at time step t, is the discount factor, is the target Q value, is the maximum Q value of all possible actions in the next state, is the position of the agent at time step t + 1, is the action to be executed corresponding to the maximum Q value, is the target parameter of the neural network, indicates that the task is completed, indicates that the task has not been completed;

[0030] According to the mean square error loss function , mean squared error loss value;

[0031] Among them, is the target Q value, is the predicted Q value of the current Q network for the given position state and action ; is the position of the agent at time step t, is the action of the agent at time step t, is the current parameter of the neural network, represents the mean squared error loss function, is the mean squared error loss value.

[0032] In a second aspect, the present invention provides a search path planning device based on deep reinforcement learning, including:

[0033] An environment establishment module, which is used to establish an environment matrix regarding detection capabilities based on search environment parameters;

[0034] A neural network module, which is used to output the Q value corresponding to each action based on the current position of the agent, the reward function, and the environment matrix, and the reward function is designed according to the current position and its distance from the target point, the detection capabilities in the neighborhood of the current position, and the number of unvisited areas;

[0035] An agent, which is used to execute the next action according to the action corresponding to the maximum Q value.

[0036] In a third aspect, a search path planning evaluation method of the present invention is characterized in that it is used to evaluate the search path planning method of any one of the above, and the evaluation method includes the following steps:

[0037] Initialize the environment matrix and set the starting point and the target point;

[0038] In each path planning step, based on the current position of the agent, the reward function, and the environment matrix, use the neural network to select the optimal action;

[0039] Synchronously update the recorded path and the cumulative detection capabilities, and mark the searched positions until the maximum coverage area is reached or the detection capabilities do not increase after the agent has taken several steps, and output the planned path;

[0040] Based on the searched position indexes in the planned path, evaluate the effectiveness of the planned path.

[0041] In some alternative solutions, the evaluation of the effectiveness of the planned path based on the searched position indexes in the planned path includes:

[0042] According to the path cumulative detection capability value , evaluate the coverage degree;

[0043] According to the cumulative search coverage rate , evaluate the search coverage rate;

[0044] Among them, is the path cumulative detection ability value, t is the number of time steps for path planning, k is the time step serial number of path planning, is the position of the agent corresponding to the k-th time step of the detection ability, is the cumulative search coverage rate, indicates whether the position of the i-th row and j-th column is visited, 1 means visited, 0 means not visited, m is the number of rows of the environment matrix, and n is the number of columns of the environment matrix.

[0045] Compared with the prior art, the advantages of the present invention are as follows: By designing a reward function according to the current position, its distance from the target point, the detection ability of the neighborhood of the current position, and the number of unvisited areas, an improved reward function that comprehensively considers the coverage range and path optimization is proposed. By introducing multi-factors such as neighborhood detection ability reward, unvisited point incentive, and dynamic target distance reward, the coverage uniformity and global performance of path planning are improved. To solve the problem that the existing reinforcement learning algorithms often have difficulty in simultaneously considering efficiency and stability when facing complex tasks, which may lead to too long path planning time or the result deviating from the global optimum, affecting the actual application effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0047] Figure 1 is the flowchart of the search path planning method based on deep reinforcement learning in the embodiment of the present invention;

[0048] Figure 2 is the reinforcement learning planned path of the first environment map in the embodiment of the present invention;

[0049] Figure 3 is the cumulative curve of the coverage rate of the reinforcement learning planned path of the first environment map in the embodiment of the present invention;

[0050] Figure 4 is the reinforcement learning planned path of the second environment map in the embodiment of the present invention;

[0051] Figure 5 is the cumulative curve of the coverage rate of the reinforcement learning planned path of the second environment map in the embodiment of the present invention. Detailed implementation manners

[0052] To enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the protection scope of this application.

[0053] As Figure 1 shown, on the one hand, the present invention provides a search path planning method based on deep reinforcement learning, which is characterized by including the following steps:

[0054] S1: Based on the search environment parameters, establish an environment matrix regarding the detection ability, and set the starting point.

[0055] In this example, the rasterization method is used to map the search environment area into an environment matrix , where each element represents the detection ability of the raster . The larger the value of the matrix element, the larger the detection range of the agent at this position. m is the number of rows of the environment matrix, and n is the number of columns of the environment matrix.

[0056] In this example, a detection instrument is used to detect and scan the area to be searched to obtain the parameters of the area to be searched environment, that is, the search environment parameters. The parameters of the area to be searched environment are the visible range or the detectable range at each position, and the visible range or the detectable range corresponds to the detection ability.

[0057] The target point is, in the initialization stage, for all position points in the current environment matrix , calculate the point with the maximum detection ability according to the detection ability value . This point will be used as the target point for calculating the target distance reward. The target point is the target raster

[0058] representing the raster with row number i and column number j.

[0059] In addition, the following definitions are also made for the reinforcement learning framework:

[0060] Definition of the action space: .

[0061] Each action represents a two-dimensional offset vector:

[0062] .

[0063] Among them, … respectively represent the execution actions of the agent in 8 directions.

[0064] The state transition function is defined as , representing the position of the agent after executing the action. The state transition of the agent is affected by the time step position state at the moment and the time step execution action at the moment , is the current time step agent position at the moment.

[0065] S2: The trained neural network outputs the Q value corresponding to each action based on the current position of the agent, the reward function, and the environment matrix. The reward function is designed according to the current position, its distance from the target point, the detection ability of the neighborhood of the current position, and the number of unvisited areas.

[0066] When the neural network outputs the action instruction corresponding to the maximum Q value, based on the current position of the agent, the reward function, and the environment matrix, calculate the reward value of each action, and output the action instruction corresponding to the maximum reward value, that is, the maximum Q value.

[0067] The agent is guided by two goals: maximizing the cumulative detection ability and covering as many grids as possible :

[0068]

[0069] Among them, represents the access record matrix, , the initial element is set to 0. When a certain grid is visited, , is the total number of steps for the agent to execute the action, t is the time step number, is the time step detection ability of the agent position at the moment.

[0070] To maximize the cumulative detection ability and cover as many grids as possible two goals, design the reward function. Specifically:

[0071] In some alternative embodiments, the reward function is designed according to the current position, its distance from the target point, the detection ability of the neighborhood of the current position, and the number of unvisited areas, including:

[0072] A: Design the detection ability reward for the current position according to the current position.

[0073] Detection ability reward for the current position Among them, is the detection ability value at the current position, where i and j are the row and column numbers of the environment matrix respectively.

[0074] The detection ability value at the current position is the basic reward when the agent is at the current position indicating the importance of detection at this position, which aims to encourage the agent to preferentially explore areas with higher detection capabilities.

[0075] B: Design the target distance reward according to the distance between the current position and the target point.

[0076] The target distance reward , where is the Euclidean distance between the agent's current position and the target point, is the weight parameter of the distance reward, is the agent's current position, is the target point position.

[0077] The target distance reward is used to adjust the importance of distance in the multi-factor immediate reward. The selection of the target point follows the principle of the point with the largest value in the current detection ability matrix. The agent will preferentially choose actions to shorten the distance. If it is close to the target point, the reward value tends to decrease to avoid over-concentration near the target point.

[0078] C: Design the neighborhood detection ability reward according to the detection ability of the neighborhood of the current position.

[0079] The neighborhood detection ability reward , where is the current position is the neighborhood range, n is the serial number of the current execution time step, is the agent's position at time step n is the detection ability.

[0080] The neighborhood detection ability reward can encourage the agent to choose actions that can explore neighborhoods with high detection capabilities, and if the detection capabilities of the surrounding areas are all low, the agent will preferentially leave this area.

[0081] D: Design the unvisited area reward according to the number of unvisited areas.

[0082] The unvisited area reward , where is the hyperparameter for adjusting the reward, is the multi-factor immediate reward obtained according to the reward function at time step t, is the record of whether the position is an unvisited area, t is the time step serial number, is the current position is the detection ability value.

[0083] The design of the unvisited area reward is to reduce the repetition of paths. The step size decay encourages the agent to explore more areas in the initial stage and focus on completing the goal in the later stage.

[0084] E: Design the reward function based on the current position detection ability reward, target distance reward, neighborhood detection ability reward, and unvisited area reward.

[0085] ;

[0086] Among them, represents the current position of the agent, is the multi-factor immediate reward, is the current position detection ability reward, is the target distance reward, is the neighborhood detection ability reward, is the unvisited area reward, is the target distance reward weight, is the neighborhood detection ability reward weight, is the unvisited area reward weight.

[0087] The improved reward function that comprehensively considers the coverage range and path optimization, by introducing multi-factors such as neighborhood detection ability reward, unvisited point incentive, and dynamic target distance reward, improves the coverage uniformity and global performance of path planning.

[0088] S3: The agent executes the next action according to the action corresponding to the maximum Q value.

[0089] In this example, according to the reward values of each action, determine the target Q value corresponding to the reward value, and select the action corresponding to the maximum Q value to execute the next action.

[0090] In this example, when the agent executes an action according to the action instruction, when the execution of the action will cause the agent to exceed the boundary condition or enter the searched position, the agent keeps the current position unchanged.

[0091] Specifically, the state transition function includes boundary conditions and access constraints. If the position of the agent after executing the action exists , , or , then keep the current position unchanged; if has been visited, that is, has been searched, then abandon the current action and keep the position unchanged.

[0092] In this example, the neural network adopts a double-network structure, where the main network parameters are and the target network parameters are , the input is the current position of the agent , and the output is the Q-value estimate of each action in the action space.

[0093] S0: The network outputs the Q-values corresponding to each action based on the current position of the agent, the reward function, and the environment matrix. That is, before applying this method, the neural network is also trained. During training, an experience replay mechanism is adopted to store historical experience tuples into the buffer, and small batches of data are uniformly sampled to break temporal correlations. At the same time, a policy is adopted to ensure that the agent efficiently learns the optimal path policy in a dynamic environment.

[0094] Each update of the neural network parameters includes the following steps:

[0095] A: Determine the target Q-value corresponding to the maximum reward value according to the reward values of each action.

[0096] Specifically, according to the Bellman equation , determine the target Q-value;

[0097] Among them, is the multi-factor immediate reward obtained according to the reward function at time step t, is the discount factor, is the target Q-value, is the maximum Q-value of all possible actions in the next state; is the position of the agent at time step t + 1, is the action to be executed corresponding to the maximum Q-value, is the target parameter of the neural network, indicates that the task is completed, indicates that the task has not been completed.

[0098] B: Determine the mean squared error loss value according to the target Q-value.

[0099] Specifically, according to the mean squared error loss function , the mean squared error loss value;

[0100] Among them, is the multi-factor immediate reward obtained according to the reward function, is the discount factor, is the target Q-value, is the predicted Q-value of the current Q-network for the given position state and action , is the position of the agent at time step t, is the action of the agent at time step t, is the current parameter of the neural network, represents the mean squared error loss function, is the mean squared error loss value.

[0101] C: Update the parameters of the neural network according to the mean squared error loss value.

[0102] Specifically, the main network updates the parameters by minimizing the mean squared error loss function The optimizer performs gradient descent, and the target network parameters are synchronized with the main network every fixed number of steps to stabilize the training process.

[0103] In a second aspect, the present invention also provides a search path planning device based on deep reinforcement learning, including: an environment establishment module, a neural network module, and an agent. Among them, the environment establishment module is used to establish an environment matrix regarding detection capabilities based on search environment parameters; the neural network module is used to output Q values corresponding to each action based on the current position of the agent, the reward function, and the environment matrix, and the reward function is designed according to the current position and its distance from the target point, the detection capabilities in the neighborhood of the current position, and the number of unvisited areas; the agent is used to execute each next action according to the action corresponding to the maximum Q value.

[0104] Among them, the functional implementation of each module in the above-mentioned search path planning device based on deep reinforcement learning corresponds to each step in the above-mentioned embodiment of the search path planning method based on deep reinforcement learning, and its functions and implementation processes will not be elaborated here one by one.

[0105] In summary, in this solution, by designing the reward function according to the current position and its distance from the target point, the detection capabilities in the neighborhood of the current position, and the number of unvisited areas, an improved reward function that comprehensively considers the coverage range and path optimization, by introducing multi-factors such as neighborhood detection ability reward, unvisited point incentive, and dynamic target distance reward, improves the coverage uniformity and global performance of path planning. To solve the problem that existing reinforcement learning algorithms often have difficulty in simultaneously considering efficiency and stability when facing complex tasks, which may lead to too long path planning time or the result deviating from the global optimum, affecting the actual application effect.

[0106] In a third aspect, the present invention also provides a search path planning evaluation method for evaluating the above-mentioned search path planning method, and this evaluation method includes the following steps:

[0107] S10: Initialize the environment matrix and set the starting point and the target point.

[0108] Specifically, , where the initialized path is used to store each point in the path, represents the initial position state of the agent, is the initial coordinate point of the agent, Represents the position state of each point in the final generated path of the agent. Is the search coverage rate of the initial state. Is the detection ability corresponding to the initial position state.

[0109] In this solution, the path starting point, as a variable input variable, can be changed according to the actual situation.

[0110] S20: In each path planning step, based on the current position of the agent, the reward function, and the environment matrix, use the neural network to select the optimal action.

[0111]

[0112] Is the position of the agent after executing the action. Indicates that at time step t, the agent selects the position state The action with the largest Q value among all possible actions , if Is out of bounds or has been visited, then abandon the execution of the action And retain the current position.

[0113] S30: Synchronously update the recorded path and the cumulative detection ability, and mark the searched positions until the maximum coverage area is reached or the detection ability does not increase after several steps of the agent, and output the planned path.

[0114] S40: Based on the searched position indices in the planned path, evaluate the effectiveness of the planned path.

[0115] Specifically include:

[0116] According to the path cumulative detection ability value , evaluate the coverage degree;

[0117] According to the cumulative search coverage rate , evaluate the search coverage rate;

[0118] Among them, Is the path cumulative detection ability value, t is the number of time steps of path planning, k is the time step number of path planning, Is the detection ability of the agent position corresponding to the k-th time step, Is the cumulative search coverage rate, Indicates whether the position in the i-th row and j-th column is visited, 1 for visited, 0 for unvisited, m is the number of rows of the environment matrix, and n is the number of columns of the environment matrix.

[0119] In this solution, and The higher the value, the better the path planning effect.

[0120] According to the above evaluation method, a simulation experiment was carried out. The size of the simulation detection ability map was set to , and two different maps were selected. According to two different environments, two environment matrices based on detection ability were constructed respectively. The path planning results of the algorithm were generated under the same starting point, and the starting point was randomly generated to generate the path. The results are shown in Figures 2 - 5 as follows. As Figure 2 shown, the path planning on the first environment matrix. This path planning can effectively cover the target area and optimize the path efficiency. Through the selection of the agent, the path planning avoids repeated and invalid paths and improves the detection efficiency. As Figure 3 shown, the cumulative curve of the path coverage rate under the first environment matrix. As time goes by, the agent continuously increases the covered area, and the coverage rate gradually increases. This shows the ability of this method to quickly cover the area in the initial stage and steadily improve and finally tend to be stable. As Figure 4 shown, the path planning on the second environment matrix. The complexity of this figure is relatively higher than that of the first one, reflecting the adaptability of this method in complex or dynamic environments. Despite the different environments, the agent can still respond flexibly and quickly adjust its path planning strategy. As Figure 5 shown, the cumulative curve of the coverage rate under the second environment matrix. Although the path planning may encounter more challenges in complex environments, Figure 5 it still shows that the agent can continuously improve the area coverage rate in a relatively long time and finally maintain at a relatively high probability level, indicating the effectiveness of this method in complex scenarios. To sum up, this method not only has high efficiency in conventional environments, but also demonstrates applicability and stability when dealing with complex and dynamic environment problems. Through comparative experiments, it is found that the paths generated by this model perform excellently in key indicators such as coverage rate, cumulative detection ability, and path efficiency. While achieving a high coverage rate, the path planning efficiency is significantly improved. By improving the design of the reward function, the decision-making of the agent in complex environments is more accurate, especially in covering areas with higher detection ability values. In addition, this method shows strong robustness and adaptability under special conditions and complex tasks, and can quickly respond to changes and generate reasonable path planning results. Generally speaking, this scheme achieves a good balance among coverage rate, path efficiency, and calculation accuracy, and has obvious advantages especially in efficiently covering the target area and maximizing the detection ability value, providing an efficient and intelligent solution for path planning in complex task scenarios.

[0121] Fourthly, an embodiment of the present application provides a search path planning device based on deep reinforcement learning. The search path planning device based on deep reinforcement learning can be a device with data processing functions such as a personal computer (PC), a laptop computer, a server, etc.

[0122] In an embodiment of the present application, the search path planning device based on deep reinforcement learning may include a processor, a memory, a communication interface, and a communication bus.

[0123] Among them, the communication bus can be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0124] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces, etc., which are used to implement the interconnection of components inside the search path planning device based on deep reinforcement learning, and interfaces for interconnecting the search path planning device based on deep reinforcement learning with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, an optical fiber interface, an ATM interface, etc.; the user device can be a display screen (Display), a keyboard (Keyboard), etc.

[0125] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical memory, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0126] The processor can be a general-purpose processor. The general-purpose processor can call the search path planning program stored in the memory and execute the search path planning method provided by the embodiment of the present application. For example, the general-purpose processor can be a central processing unit (CPU). Among them, the method executed when the search path planning program based on deep reinforcement learning is called can refer to the various embodiments of the search path planning method based on deep reinforcement learning of the present application, which will not be elaborated here.

[0127] Fifthly, an embodiment of the present application further provides a computer-readable storage medium.

[0128] A search path planning program based on deep reinforcement learning is stored on a computer-readable storage medium of the present application. When the search path planning program based on deep reinforcement learning is executed by a processor, the steps of the search path planning method based on deep reinforcement learning as described above are implemented.

[0129] Among them, the method implemented when the search path planning program based on deep reinforcement learning is executed can refer to each embodiment of the search path planning method based on deep reinforcement learning of the present application, which will not be elaborated here.

[0130] It should be noted that the serial numbers of the embodiments of the present application above are only for description and do not represent the advantages and disadvantages of the embodiments.

[0131] The terms "including" and "having" and any variations thereof in the specification, claims and drawings of the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices. The descriptions of "first", "second", "third", etc. are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit that "first", "second" and "third" are different types.

[0132] In the description of the embodiments of the present application, "exemplary", "for example" or "for instance" are used to mean for example, illustration or explanation. Any embodiment or design solution described as "exemplary", "for example" or "for instance" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary", "for example" or "for instance" is intended to present relevant concepts in a specific manner.

[0133] In the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality of" means two or more than two.

[0134] In some processes described in the embodiments of the present application, there are multiple operations or steps that appear in a specific order. However, it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or may be executed in parallel. The serial numbers of the operations are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0135] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium as described above (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal device to execute the methods described in various embodiments of the present application.

[0136] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A search path planning method based on deep reinforcement learning, characterized in that, It includes the following steps: Based on the search environment parameters, establish an environmental matrix regarding the detection ability and set a starting point; The trained neural network outputs the Q values corresponding to each action based on the current position of the agent, the reward function, and the environmental matrix, and the reward function is designed according to the current position, its distance from the target point, the detection ability of the neighborhood of the current position, and the number of unvisited areas; The agent executes each next action according to the action corresponding to the maximum Q value; The reward function is: ； Among them, represents the current position of the agent, is the multi-factor immediate reward corresponding to the position S after performing the action a , is the current position detection ability reward, is the target distance reward, is the neighborhood detection ability reward, is the unvisited area reward, is the target distance reward weight, is the neighborhood detection ability reward weight, is the unvisited area reward weight; Current position detection ability reward , where is the detection ability value of the current position, and i and j are the row and column numbers of the environment matrix respectively; Target distance reward , where is the Euclidean distance between the agent's current position and the target point, is the weight parameter of the distance reward, is the agent's current position, is the target point position; Neighborhood Detection Ability Reward , where is the neighborhood range of the current position , n is the sequence number of the current execution time step is the agent position at time step n is the detection ability Unvisited Area Reward , where is the hyperparameter for adjusting the reward, is the multi-factor immediate reward obtained according to the reward function at time step t, is the record of whether the position is an unvisited area, and t is the time step number, is the current position is the detection ability value of.

2. The search path planning method according to claim 1, wherein When the agent executes an action according to the action instruction, if the execution of the action will cause the agent to exceed the boundary condition or enter the searched position, the agent remains at the current position unchanged.

3. The search path planning method according to claim 1, wherein Before the neural network outputs the Q values corresponding to each action based on the current position of the agent, the reward function, and the environmental matrix, the neural network is also trained to obtain the trained neural network. During training, each update of the neural network parameters includes the following steps: Determine the target Q value corresponding to the maximum reward value according to the reward values of each action; Determine the mean squared error loss value according to the target Q value; Update the parameters of the neural network according to the mean squared error loss value.

4. The search path planning method according to claim 3, wherein: According to the Bellman equation , determine the target Q-value; where is the multi-factor immediate reward obtained according to the reward function at time step t, is the discount factor, is the target Q-value, is the maximum Q-value of all possible actions in the next state, is the position of the agent at time step t+1, is the action to be executed corresponding to the maximum Q-value, is the target parameter of the neural network, indicates that the task is completed, indicates that the task has not been completed; According to the mean square error loss function , the mean square error loss value; Among them, is the target Q value, is the predicted Q value of the current Q network for the given position state and action . is the position of the agent at time step t, is the action of the agent at time step t, is the current parameter of the neural network, represents the mean squared error loss function, is the mean squared error loss value.

5. A search path planning device based on deep reinforcement learning, characterized in that, It includes: An environment establishment module, which is used to establish an environmental matrix regarding the detection ability based on the search environment parameters; A neural network module, which is used to output the Q values corresponding to each action based on the current position of the agent, the reward function, and the environmental matrix, and the reward function is designed according to the current position, its distance from the target point, the detection ability of the neighborhood of the current position, and the number of unvisited areas; An agent, which is used to execute each next action according to the action corresponding to the maximum Q value; The reward function is: ； Among them, represents the current position of the agent, is the multi-factor immediate reward corresponding to the position at S after performing the action a , is the current position detection ability reward, is the target distance reward, is the neighborhood detection ability reward, is the unvisited area reward, is the target distance reward weight, is the neighborhood detection ability reward weight, is the unvisited area reward weight; Current position detection ability reward , where is the detection ability value of the current position, and i and j are the row and column numbers of the environment matrix respectively; Target distance reward , where is the Euclidean distance between the current position of the agent and the target point, is the weight parameter of the distance reward, is the current position of the agent, is the position of the target point; Neighborhood Detection Ability Reward , where is the neighborhood range of the current position , n is the sequence number of the current execution time step is the detection ability of the agent at time step n . Unvisited Area Reward , where is the hyperparameter for adjusting the reward, is the multi-factor immediate reward obtained according to the reward function at time step t, is the record of whether the position is an unvisited area, and t is the time step number, is the current position is the detection ability value of 6. A search path planning evaluation method, characterized in that For evaluating the search path planning method according to any one of claims 1-4, the evaluation method includes the following steps: Initialize the environmental matrix and set the starting point and the target point; In each path planning step, based on the current position of the agent, the reward function, and the environmental matrix, use the neural network to select the optimal action; Synchronously update the recorded path and the cumulative detection ability, and mark the searched positions until the maximum coverage area is reached or the detection ability does not increase after several steps of the agent, and output the planned path; Based on the searched position indexes in the planned path, evaluate the effectiveness of the planned path.

7. The search path planning evaluation method according to claim 6, wherein The evaluation of the effectiveness of the planned path based on the searched position indexes in the planned path includes: Evaluate the coverage based on the path cumulative detection ability value , and evaluate the coverage; According to the cumulative search coverage rate , evaluate the search coverage rate; Among them, is the path cumulative detection ability value, t is the number of time steps for path planning, k is the time step sequence number of path planning, is the position of the agent corresponding to the k-th time step of the detection ability, is the cumulative search coverage rate, indicates whether the position of the i-th row and j-th column is visited, 1 means visited, 0 means not visited, m is the number of rows of the environment matrix, and n is the number of columns of the environment matrix.

Images