Path planning method, device, equipment, medium and product
Through the deep learning Q network combined with task difficulty level update path rewards, the problem that existing path planning algorithms cannot match fire conditions is solved, and a more efficient fire task execution path is achieved.
Patent Information
- Application Number
- CN202411999106.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-04-18
AI Technical Summary
The existing fire robot path planning algorithm cannot dynamically adapt to path cost changes and cannot determine the path matching the fire situation, resulting in the shortest path not necessarily the most efficient path to perform fire protection tasks.
By obtaining the task to be executed and the target task map, combining the deep learning Q network to predict the action value of each task action of the task execution object at each time step, and perform action selection, update the path rewards to match the task difficulty level, and determine the optimal task path.
In the process of task path planning, it realizes that the path reward is updated in combination with the task difficulty level to obtain paths that are more matched with the task to be executed, thereby improving the task execution efficiency.
Smart Images
Figure CN119847154A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular, to a path planning method, device, equipment, medium and product. Background Art
[0002] Due to its automation and intelligent features, firefighting robots can be used in fire environments to remove fire-fighting equipment and perform fire-fighting tasks.
[0003] However, the path planning algorithms currently used in firefighting robots cannot dynamically adapt to the changes in path costs of robots in specific working scenarios, nor can they determine the path that matches the fire situation. That is, although the path planning results obtained by the existing path planning algorithms can obtain the shortest path, they cannot guarantee that the shortest path is the most efficient path for executing firefighting tasks. Summary of the invention
[0004] Embodiments of the present invention provide a path planning method, apparatus, equipment, medium and product, which can update the path reward in combination with the task difficulty level requirements during the task path planning process, obtain a path that better matches the task to be performed, and obtain a task path that can perform the task with higher efficiency.
[0005] In a first aspect, an embodiment of the present invention provides a path planning method, the method comprising:
[0006] Obtaining a map of the tasks to be performed and the target task, wherein the target task map is marked with the target task location and the location of the task equipment, and the task information of the tasks to be performed includes the task difficulty level;
[0007] Based on the deep learning Q network, the task execution object is predicted to perform the action value of each task action in the preset action space under the state corresponding to each task execution time step, and the action selection is performed to obtain the action selection result; wherein the task execution time step is each time step from the starting point of the task to be executed to the target task location of the task execution object; the state includes the position of the task execution object in the target task map, the task equipment pickup situation, the target task location and the task difficulty level; the task actions in the preset action space include moving in any preset direction, picking up task equipment and executing the target task;
[0008] Determine the target task action value corresponding to each task execution time step according to the action selection result and the preset reward function; wherein the preset reward function includes a mapping relationship between the action result of the task action and the matching degree of the task difficulty level and the reward value;
[0009] According to the target task action value of each task execution time step, the task cumulative value of each task path corresponding to the task to be executed is determined, and the task path corresponding to the task cumulative value with the highest value among the task cumulative values is determined as the target task path for executing the task to be executed.
[0010] In a second aspect, an embodiment of the present invention further provides a path planning device, the device comprising:
[0011] A path planning task information acquisition module is used to obtain the task to be performed and the target task map, wherein the target task map is marked with the target task location and the task equipment location, and the task information of the task to be performed includes the task difficulty level;
[0012] The task action analysis module is used to predict the action value of each task action in the preset action space executed by the task execution object under the state corresponding to each task execution time step based on the deep learning Q network, and perform action selection to obtain the action selection result; wherein the task execution time step is each time step from the starting point of the task to be executed to the target task location of the task execution object; the state includes the position of the task execution object in the target task map, the task equipment pickup situation, the target task location and the task difficulty level; the task actions in the preset action space include moving in any preset direction, picking up task equipment and executing the target task;
[0013] An action reward analysis module is used to determine the target task action value corresponding to each task execution time step according to the action selection result and the preset reward function; wherein the preset reward function includes a mapping relationship between the action result of the task action and the matching degree of the task difficulty level and the reward value;
[0014] The path generation module is used to determine the task cumulative value of each task path corresponding to the task to be executed according to the target task action value of each task execution time step, and determine the task path corresponding to the task cumulative value with the highest value among the task cumulative values as the target task path for executing the task to be executed.
[0015] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:
[0016] one or more processors;
[0017] A memory for storing one or more programs;
[0018] When one or more programs are executed by one or more processors, the one or more processors implement the path planning method provided by any embodiment of the present invention.
[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a path planning method as provided in any embodiment of the present invention.
[0020] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements a path planning method as provided in any embodiment of the present invention.
[0021] The embodiments of the above invention have the following advantages or beneficial effects:
[0022] In an embodiment of the present invention, by obtaining a map of a task to be performed and a target task, wherein the target task location and the location of task equipment are marked in the target task map, and the task information of the task to be performed includes the task difficulty level; based on a deep learning Q network, the action value of each task action in the preset action space executed by the task execution object in the state corresponding to each task execution time step is predicted, and action selection is performed to obtain an action selection result; wherein the task execution time step is each time step from the starting point of the task to be performed to the target task location of the task execution object; the state includes the position of the task execution object in the target task map, the task equipment picking situation, the target task location and the task difficulty level; the task action in the preset action space includes moving in any preset direction, picking up task equipment and performing the target task; the target task action value corresponding to each task execution time step is determined according to the action selection result and a preset reward function; wherein the preset reward function includes a mapping relationship between the matching degree of the action result of the task action and the task difficulty level and the reward value; according to the target task action value of each task execution time step, the task cumulative value corresponding to each task path corresponding to the task to be performed is determined, and the task path corresponding to the task cumulative value with the highest value in the task cumulative value is determined as the target task path for performing the task to be performed. The technical solution of the embodiment of the present invention solves the problem that although the current path planning result is the shortest path, it is not the path that can make the task execution most efficient. In the process of task path planning, the path reward can be updated in combination with the requirements of the task difficulty level to obtain a path that better matches the task to be executed, so as to obtain a task path that can execute the task with higher efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 is a flow chart of a path planning method provided by an embodiment of the present invention;
[0024] Figure 2 is a flow chart of a path planning method provided by an embodiment of the present invention;
[0025] Figure 3 It is a schematic diagram of a task map provided by an embodiment of the present invention;
[0026] Figure 4 is a schematic structural diagram of a path planning device provided by an embodiment of the present invention;
[0027] Figure 5 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0028] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0029] Figure 1 This is a flow chart of a path planning method provided by an embodiment of the present invention. This embodiment can be applied to the scenario of task path planning, especially the path planning situation based on a neural network agent. The method can be executed by a path planning device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.
[0030] like Figure 1 As shown, the path planning method of this embodiment includes the following steps:
[0031] S110, obtaining a map of the task to be performed and a target task, wherein the target task location and the location of task equipment are marked in the target task map, and the task information of the task to be performed includes the task difficulty level.
[0032] The task to be performed can be any task that needs to be performed by moving in a spatial distance. The target task map corresponds to the movable spatial range when performing the task to be performed.
[0033] The target mission location and mission equipment position are marked in the target mission map. The target mission location is the destination of the movement path of the mission execution object in the target mission map. The mission equipment position can be a waypoint before reaching the destination. At the corresponding waypoint, the mission execution object can pick up the mission equipment, and the picked up mission equipment can be used to perform the mission at the destination.
[0034] Among them, the task difficulty level indicates the difficulty of executing the task to be executed. Different task difficulty levels can correspond to different types of task equipment, which will also affect the execution efficiency of the task. If the task execution object can pick up task equipment that matches the task difficulty level during the path planning process, the task execution efficiency can be improved. If the task equipment that does not match the task difficulty level is picked up, the task execution efficiency will be affected and the task execution time will be increased.
[0035] S120, predicting the action value of each task action in the preset action space executed by the task execution object in the state corresponding to each task execution time step based on the deep learning Q network, and performing action selection to obtain an action selection result.
[0036] Among them, the task execution time step is each time step of the task execution object from the starting point of executing the task to be executed to the target task location; the state includes the position of the task execution object in the target task map, the task equipment pickup situation, the target task location and the task difficulty level; the task actions in the preset action space include moving in any preset direction, picking up task equipment and executing the target task.
[0037] The deep learning Q network is also called the action value function. Q(s,a) represents the expected value of the future cumulative reward that the agent can obtain after executing action a when it is in state s, which is the action value of action a.
[0038] In this step, for each task execution time step, the state corresponding to the time step is combined with each task action in the preset action space, and the combination result is input into the deep learning Q network to predict the action value corresponding to each task action under the corresponding state. Then, the action is selected through the preset action selection strategy, such as the ε-greedy strategy or the Softmax strategy to obtain the action selection result.
[0039] S130. Determine the target task action value corresponding to each task execution time step according to the action selection result and the preset reward function.
[0040] The preset reward function is a function that maps from the environment state and the agent action to real numbers, usually expressed as R(s,a), which is used to quantify the feedback that the agent receives immediately after performing action a in state s. This feedback value can be positive or negative. Positive rewards encourage the agent to repeat the corresponding actions, while negative rewards warn the agent to avoid such actions.
[0041] Among them, the preset reward function includes the mapping relationship between the action result of the task action and the matching degree of the task difficulty level and the reward value. For example, if the task action is to pick up the task equipment, and the matching degree of the picked up equipment and the task difficulty level is the highest level, then it can be mapped to the highest reward value, so that the intelligent agent learns to select the location of the task equipment that matches the task difficulty level when the waypoints of the planned path planning include the task equipment points. Of course, the preset reward function can also include other positive or negative action reward items.
[0042] In a specific implementation, the action value corresponding to the action selection result can be used as the initial task action value of the corresponding task execution time step; when the task action corresponding to the action selection result is picking up task equipment, the matching degree between the equipment type of the picked up task equipment and the task difficulty level is input into the preset reward function to obtain the action reward value; then, the action reward value is superimposed on the initial task action value to obtain the target task action value.
[0043] In an optional embodiment, when the task action is picking up the task equipment, the action cost value corresponding to the task action after the action of picking up the task equipment can also be updated according to the equipment type of the task equipment picked up by the task action; wherein the action cost value corresponds to the negative reward value of the action. The goal is to minimize the load-bearing operation of the task execution object during the path planning process, and let the task execution object pick up the task equipment at the task equipment position that is close to the target task location as much as possible. The heavier the weight of the picked up task equipment, the larger the corresponding action cost value. The action cost value can be subtracted from the above target task action value to obtain the latest target task action value.
[0044] S140. Determine the task cumulative value of each task path corresponding to the task to be executed based on the target task action value of each task execution time step, and determine the task path corresponding to the task cumulative value with the highest value among the task cumulative values as the target task path for executing the task to be executed.
[0045] The technical solution of this embodiment is to obtain the map of the task to be executed and the target task, wherein the target task location and the location of the task equipment are marked in the target task map, and the task information of the task to be executed includes the task difficulty level; based on the deep learning Q network, the action value of each task action in the preset action space executed by the task execution object in the state corresponding to each task execution time step is predicted, and action selection is performed to obtain the action selection result; wherein the task execution time step is each time step from the starting point of the task to be executed to the target task location of the task execution object; the state includes the position of the task execution object in the target task map, the task equipment picking situation, the target task location and the task difficulty level; the task action in the preset action space includes moving in any preset direction, picking up task equipment and executing the target task; the target task action value corresponding to each task execution time step is determined according to the action selection result and the preset reward function; wherein the preset reward function includes the mapping relationship between the matching degree of the action result of the task action and the task difficulty level and the reward value; according to the target task action value of each task execution time step, the task cumulative value corresponding to each task path corresponding to the task to be executed is determined, and the task path corresponding to the task cumulative value with the highest value in the task cumulative value is determined as the target task path for executing the task to be executed. The technical solution of the embodiment of the present invention solves the problem that although the current path planning result is the shortest path, it is not the path that can make the task execution most efficient. In the process of task path planning, the path reward can be updated in combination with the requirements of the task difficulty level to obtain a path that better matches the task to be executed, so as to obtain a task path that can execute the task with higher efficiency.
[0046] In addition, in the process of obtaining the path planning results, the state, task action, target task action value and new state after the action execution corresponding to each task execution time step can be stored in the preset experience playback buffer; the deep learning Q network is updated according to the data in the preset experience playback buffer and the preset objective function. That is, the Q table is updated, and the update process can be expressed as:
[0047] Q(s,a)←Q(s,a)+α[R+γmax a′ Q(s ′ ,a ′ )-Q(s,a)]. Where s ′ is the new state after taking action a, max a′ Q(s ′ ,a ′ ) means in state s ′ Next, perform action a ′α is the learning rate, which controls the magnitude of each update and is usually between 0.1 and 0.9; γ is the discount factor, which is set to about 0.9 to balance the importance of current and future rewards. In each interaction, the task execution object uses new data to improve the Q table so that the Q value is close to the real action value.
[0048] Figure 2 This is a flow chart of a path planning method provided in an embodiment of the present invention. This embodiment and the path planning method in the above embodiment belong to the same inventive concept, and further describes the path planning process when the task to be executed is a fire-fighting task and the task execution object is a fire-fighting robot. The method can be executed by a path planning device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.
[0049] like Figure 2 As shown, the path planning method of this embodiment includes the following steps:
[0050] S210, obtaining a fire fighting task and a target task map, wherein the target task location and the location of fire fighting equipment are marked in the target task map, and the task information of the fire fighting task includes the fire size level.
[0051] Figure 3 is a schematic diagram of the target mission map. The map information can be obtained by inputting the floor plan of the building to ensure that the map accurately reflects the building layout and equipment distribution. The map can be discretized based on the obtained floor plan, and the locations of fire extinguishers, fire hydrant water guns, fire blankets and other fire-fighting equipment in the building can be marked, as well as the location of the fire source (corresponding to the target mission location) and the distribution of fire intensity (corresponding to the mission difficulty level, the intensity of the fire is proportional to the difficulty of the mission).
[0052] S220, predicting the action value of each task action in the preset action space executed by the task execution object in the state corresponding to each task execution time step based on the deep learning Q network, and performing action selection to obtain an action selection result.
[0053] Among them, the task execution time step is each time step from the starting point of the task to be executed to the target task location of the task execution object. The state includes the position of the task execution object in the target task map, the task equipment pickup situation, the target task location and the task difficulty level, such as the current location information of the task execution object, the current status of the equipment carried, the location of the fire source and the size of the fire. The state can be represented in the form of a vector, and one state information corresponds to a multi-dimensional state vector.
[0054] The task actions in the preset action space include moving in any preset direction (such as moving forward, backward, left or right), picking up task equipment (obtaining a fire extinguisher, water gun or fire blanket) and performing the target task (performing the fire extinguishing task).
[0055] In each task time step, the corresponding state and action are input into the deep learning Q network to generate a reachable path network and a dynamic value (cost) table. Then, the action is selected through the preset action selection strategy, such as the ε-greedy strategy or the Softmax strategy to obtain the action selection result.
[0056] S230. Determine the target task action value corresponding to each task execution time step according to the action selection result and the preset reward function.
[0057] The preset reward function includes a mapping relationship between the matching degree of the action result of the task action and the task difficulty level and the reward value. The mapping relationship between the matching degree of the action result of the task action and the task difficulty level and the reward value includes a mapping relationship between the matching degree of the equipment type of the fire extinguishing equipment and the fire size level and the reward value.
[0058] For example, when the size of the fire corresponds to a micro fire (very weak fire), the action of obtaining fire extinguishing equipment is to select a fire blanket, and the corresponding reward is set to +1000; the action of obtaining fire extinguishing equipment is to select a fire extinguisher, and the corresponding reward is set to +500. The fire blanket has a higher matching degree with the corresponding micro fire. The firefighting robot gives priority to the fire blanket as a firefighting tool because it is suitable for handling small fire sources, has high efficiency and does not require weight to change the path cost. When the size of the fire corresponds to a small fire (weak fire), the action of obtaining fire extinguishing equipment is to select a fire extinguisher, and the corresponding reward is set to +1000; the action of obtaining fire extinguishing equipment is to select a water gun, and the corresponding reward is set to +500. The firefighting robot gives priority to the fire extinguisher because its portability and fire extinguishing effect are more suitable for small fire scenarios. When the fire is large (strong fire), the action of obtaining fire extinguishing equipment is to select a fire extinguisher, and the corresponding reward is set to -1000; the action of obtaining fire extinguishing equipment is to select a water gun, and the corresponding reward is set to +1000. It is mandatory to select a fire hydrant water gun to ensure that the fire-fighting tools can cope with scenarios with stronger fires.
[0059] In addition, the preset reward function can also update the action cost value corresponding to the task action after the task action of picking up the fire-fighting equipment according to the equipment load parameter corresponding to the fire-fighting equipment picked up by the task action. Among them, the value of the equipment load parameter is proportional to the action cost value. The cost value of the path network will be recalculated in real time after each device is picked up to ensure that the robot still chooses the global optimal path in the new state.
[0060] For example, the initial cost of the robot walking 1 meter is set to 1. After the firefighting robot picks up the fire extinguisher, the travel speed becomes half of the original speed, and the path cost doubles. For example, the cost of walking 5 meters with a loaded fire extinguisher is 5×2=10. After the firefighting robot picks up the fire hydrant water gun, the travel speed becomes one-third of the original speed, and the path cost becomes three times the original speed. For example, the cost of walking 5 meters with a loaded water gun is 5×3=15. This method can ensure that the firefighting robot always chooses the path with the lowest cost when carrying a load, effectively improving the efficiency and adaptability of task execution.
[0061] In the process of implementing S220 and S230, the deep learning Q network can also be updated according to the target task action value corresponding to each task execution time step. The update process can be expressed as:
[0062] Q(s,a)←Q(s,a)+α[R+γmax a′ Q(s ′ ,a ′ )-Q(s,a)]. Where s ′ is the new state after taking action a, max a′ Q(s ′ ,a ′ ) means in state s ′ Next, perform action a ′ α is the learning rate, which controls the amplitude of each update and is usually between 0.1 and 0.9; γ is the discount factor, which is set to about 0.9 to balance the importance of current and future rewards. In each interaction, the task execution object uses new data to improve the Q table so that the Q value is close to the real action value. By repeating the interactive learning process and after a large number of iterations, the Q table gradually stabilizes. At this time, the firefighting robot can rely on the Q table to start from the starting state, select the action with the maximum Q value each time, and efficiently plan the optimal path to the target point.
[0063] S240. Determine the task cumulative value of each task path corresponding to the task to be executed based on the target task action value of each task execution time step, and determine the task path corresponding to the task cumulative value with the highest value among the task cumulative values as the target task path for executing the task to be executed.
[0064] The technical solution of this embodiment adjusts the path cost in real time by adding a dynamic path cost update mechanism, so that it can complete the task efficiently under load conditions. The specific method includes: dynamically updating the path cost according to the load conditions of the robot carrying different equipment (such as fire extinguishers and fire hoses), for example, the path cost doubles when carrying a heavy fire extinguisher, and the path cost triples when carrying a heavy hose, and combining the reinforcement learning algorithm to achieve real-time replanning of the path. Moreover, the reward mechanism of the preset reward function in this embodiment can guide the fire-fighting robot to intelligently select the optimal fire-fighting equipment in different fire scenarios. This mechanism senses the fire in real time, judges whether the fire is a micro-fire, a small fire or a large fire, and assigns differentiated reward values to different fire-fighting equipment. Finally, based on the multi-task joint optimization mechanism of deep Q-Learning, path planning, equipment picking and fire-fighting tasks are integrated, and multiple indicators such as path length, task completion efficiency and fire-fighting effect are comprehensively considered; so that the robot can efficiently and intelligently complete complex fire-fighting tasks, and has strong adaptability and scalability.
[0065] Figure 4 A structural schematic diagram of a path planning device provided for an embodiment of the present invention. This embodiment is applicable to scenarios, especially situations. The path planning device can be implemented by software and / or hardware and integrated into a computer terminal device with application development function.
[0066] like Figure 4 The path planning device shown includes: an instruction acquisition module 410 and an instruction execution module 420 .
[0067] Among them, the path planning task information acquisition module 310 is used to obtain the map of the task to be executed and the target task, wherein the target task location and the location of the task equipment are marked in the target task map, and the task information of the task to be executed includes the task difficulty level; the task action analysis module 320 is used to predict the action value of each task action in the preset action space executed by the task execution object under the state corresponding to each task execution time step based on the deep learning Q network, and perform action selection to obtain the action selection result; wherein the task execution time step is each time step from the starting point of the task execution object to the target task location; the state includes the location of the task execution object in the target task map, the task equipment pickup situation, the target task location, and the status of the task execution object. point and task difficulty level; the task actions in the preset action space include moving in any preset direction, picking up task equipment and performing target tasks; an action reward analysis module 330, which is used to determine the target task action value corresponding to each task execution time step according to the action selection result and the preset reward function; wherein the preset reward function includes a mapping relationship between the action result of the task action and the matching degree of the task difficulty level and the reward value; a path generation module 340, which is used to determine the task cumulative value of each task path corresponding to the task to be performed according to the target task action value of each task execution time step, and determine the task path corresponding to the task cumulative value with the highest value in the task cumulative value as the target task path for performing the task to be performed.
[0068] The technical solution of this embodiment is to obtain the map of the task to be executed and the target task, wherein the target task location and the location of the task equipment are marked in the target task map, and the task information of the task to be executed includes the task difficulty level; based on the deep learning Q network, the action value of each task action in the preset action space executed by the task execution object in the state corresponding to each task execution time step is predicted, and action selection is performed to obtain the action selection result; wherein the task execution time step is each time step from the starting point of the task to be executed to the target task location of the task execution object; the state includes the position of the task execution object in the target task map, the task equipment picking situation, the target task location and the task difficulty level; the task action in the preset action space includes moving in any preset direction, picking up task equipment and executing the target task; the target task action value corresponding to each task execution time step is determined according to the action selection result and the preset reward function; wherein the preset reward function includes the mapping relationship between the matching degree of the action result of the task action and the task difficulty level and the reward value; according to the target task action value of each task execution time step, the task cumulative value corresponding to each task path corresponding to the task to be executed is determined, and the task path corresponding to the task cumulative value with the highest value in the task cumulative value is determined as the target task path for executing the task to be executed. The technical solution of the embodiment of the present invention solves the problem that although the current path planning result is the shortest path, it is not the path that can make the task execution most efficient. In the process of task path planning, the path reward can be updated in combination with the requirements of the task difficulty level to obtain a path that better matches the task to be executed, so as to obtain a task path that can execute the task with higher efficiency.
[0069] In an optional implementation, the action reward analysis module 330 is specifically used to:
[0070] The action value corresponding to the action selection result is used as the initial task action value of the corresponding task execution time step;
[0071] When the task action corresponding to the action selection result is picking up task equipment, the matching degree between the equipment type of the picked up task equipment and the task difficulty level is input into the preset reward function to obtain the action reward value;
[0072] Add the action reward value to the initial task action value to get the target task action value.
[0073] In an optional implementation, the action reward analysis module 330 is specifically used to:
[0074] When the task action is to pick up the task equipment, the action cost value corresponding to the task action after the action of picking up the task equipment is updated according to the equipment type of the task equipment picked up by the task action;
[0075] Among them, the action cost value corresponds to the negative reward value of the action.
[0076] In an optional implementation, the task action analysis module 320 may also be used to:
[0077] The state, task action, target task action value and new state after action execution corresponding to each task execution time step are stored in the preset experience playback buffer;
[0078] The deep learning Q network is updated based on the data in the preset experience replay buffer and the preset objective function.
[0079] In an optional implementation, the task to be performed is a fire-fighting task, and the task execution object is a fire-fighting robot; accordingly, the task difficulty level is the fire size level, and the task equipment is fire-fighting equipment;
[0080] The mapping relationship between the matching degree and reward value of the action result of the task action and the task difficulty level includes the mapping relationship between the matching degree and reward value of the equipment type of the fire-fighting equipment and the fire size level.
[0081] In an optional implementation, the action reward analysis module 330 is specifically used to:
[0082] According to the equipment weight parameter corresponding to the fire-fighting equipment picked up by the task action, the action cost value corresponding to the task action after the task action of picking up the fire-fighting equipment is updated;
[0083] Among them, the value of the equipment weight parameter is proportional to the action cost value.
[0084] The path planning device provided in the embodiment of the present invention can execute the path planning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0085] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 5 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 5 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities, such as an intelligent controller and server, a mobile phone and other terminal devices.
[0086] like Figure 5As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).
[0087] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0088] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.
[0089] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.
[0090] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0091] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0092] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing the path planning method provided in the embodiment of the present invention, which includes:
[0093] Obtaining a map of the tasks to be performed and the target task, wherein the target task map is marked with the target task location and the location of the task equipment, and the task information of the tasks to be performed includes the task difficulty level;
[0094] Based on the deep learning Q network, the task execution object is predicted to perform the action value of each task action in the preset action space under the state corresponding to each task execution time step, and the action selection is performed to obtain the action selection result; wherein the task execution time step is each time step from the starting point of the task to be executed to the target task location of the task execution object; the state includes the position of the task execution object in the target task map, the task equipment pickup situation, the target task location and the task difficulty level; the task actions in the preset action space include moving in any preset direction, picking up task equipment and executing the target task;
[0095] Determine the target task action value corresponding to each task execution time step according to the action selection result and the preset reward function; wherein the preset reward function includes a mapping relationship between the action result of the task action and the matching degree of the task difficulty level and the reward value;
[0096] According to the target task action value of each task execution time step, the task cumulative value of each task path corresponding to the task to be executed is determined, and the task path corresponding to the task cumulative value with the highest value among the task cumulative values is determined as the target task path for executing the task to be executed.
[0097] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the path planning method provided by any embodiment of the present invention is implemented. The method includes:
[0098] Obtaining a map of the tasks to be performed and the target task, wherein the target task map is marked with the target task location and the location of the task equipment, and the task information of the tasks to be performed includes the task difficulty level;
[0099] Based on the deep learning Q network, the task execution object is predicted to perform the action value of each task action in the preset action space under the state corresponding to each task execution time step, and the action selection is performed to obtain the action selection result; wherein the task execution time step is each time step from the starting point of the task to be executed to the target task location of the task execution object; the state includes the position of the task execution object in the target task map, the task equipment pickup situation, the target task location and the task difficulty level; the task actions in the preset action space include moving in any preset direction, picking up task equipment and executing the target task;
[0100] Determine the target task action value corresponding to each task execution time step according to the action selection result and the preset reward function; wherein the preset reward function includes a mapping relationship between the action result of the task action and the matching degree of the task difficulty level and the reward value;
[0101] According to the target task action value of each task execution time step, the task cumulative value of each task path corresponding to the task to be executed is determined, and the task path corresponding to the task cumulative value with the highest value among the task cumulative values is determined as the target task path for executing the task to be executed.
[0102] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0103] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0104] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0105] Computer program code for performing the operation of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0106] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0107] An embodiment of the present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements a path planning method as provided in any embodiment of the present disclosure.
[0108] In the process of implementation, the computer program product can be written in one or more programming languages or a combination thereof to perform the computer program code for the disclosed operation, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).
[0109] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A path planning method, characterized in that: include: Obtaining a map of tasks to be performed and a target task, wherein the target task map is marked with a target task location and a task equipment location, and the task information of the task to be performed includes a task difficulty level; Based on the deep learning Q network, the action value of each task action in the preset action space is predicted by the task execution object in the state corresponding to each task execution time step, and action selection is performed to obtain the action selection result; wherein, the task execution time step is each time step of the task execution object from the starting point of the task to be executed to the target task location; the state includes the position of the task execution object in the target task map, the task equipment pickup situation, the target task location and the task difficulty level; the task actions in the preset action space include moving in any preset direction, picking up task equipment and executing the target task; Determine the target task action value corresponding to each task execution time step according to the action selection result and the preset reward function; wherein the preset reward function includes a mapping relationship between the action result of the task action and the matching degree of the task difficulty level and the reward value; According to the target task action value of each task execution time step, the task cumulative value of each task path corresponding to the task to be executed is determined, and the task path corresponding to the task cumulative value with the highest value among the task cumulative values is determined as the target task path for executing the task to be executed.
2. The method according to claim 1, characterized in that: The determining the target task action value corresponding to each task execution time step according to the action selection result and the preset reward function includes: The action value corresponding to the action selection result is used as the initial task action value of the corresponding task execution time step; When the task action corresponding to the action selection result is picking up the task equipment, the matching degree between the equipment type of the picked up task equipment and the task difficulty level is input into the preset reward function to obtain the action reward value; The action reward value is added to the initial task action value to obtain the target task action value.
3. The method according to claim 2, characterized in that When the task action is picking up task equipment, the method further includes: According to the equipment type of the task equipment picked up by the task action, updating the action cost value corresponding to the task action after the action of picking up the task equipment; Among them, the action cost value corresponds to the negative reward value of the action.
4. The method according to claim 1, characterized in that: The method further comprises: The state, task action, target task action value and new state after the action execution corresponding to each task execution time step are stored in a preset experience playback buffer; The deep learning Q network is updated according to the data in the preset experience replay buffer and the preset objective function.
5. The method according to any one of claims 1 to 4, characterized in that: The task to be performed is a fire-fighting task, and the task execution object is a fire-fighting robot; accordingly, the task difficulty level is the fire size level, and the task equipment is fire-fighting equipment; The mapping relationship between the action result of the task action and the matching degree of the task difficulty level and the reward value includes the mapping relationship between the matching degree of the equipment type of the fire extinguishing equipment and the fire size level and the reward value.
6. The method according to any one of claim 5, characterized in that: The method further comprises: According to the equipment weight parameter corresponding to the fire-fighting equipment picked up by the task action, updating the action cost value corresponding to the task action after the task action of picking up the fire-fighting equipment; Among them, the value of the equipment weight parameter is proportional to the action cost value.
7. A path planning device, characterized in that: include: A path planning task information acquisition module is used to acquire the task to be performed and the target task map, wherein the target task map is marked with the target task location and the task equipment location, and the task information of the task to be performed includes the task difficulty level; The task action analysis module is used to predict the action value of each task action in the preset action space executed by the task execution object under the state corresponding to each task execution time step based on the deep learning Q network, and perform action selection to obtain the action selection result; wherein, the task execution time step is each time step of the task execution object from the starting point of executing the task to be executed to the target task location; the state includes the position of the task execution object in the target task map, the task equipment pickup situation, the target task location and the task difficulty level; the task actions in the preset action space include moving in any preset direction, picking up task equipment and executing the target task; An action reward analysis module, used to determine the target task action value corresponding to each task execution time step according to the action selection result and a preset reward function; wherein the preset reward function includes a mapping relationship between the action result of the task action and the matching degree of the task difficulty level and the reward value; The path generation module is used to determine the task cumulative value of each task path corresponding to the task to be executed according to the target task action value of each task execution time step, and determine the task path corresponding to the task cumulative value with the highest value among the task cumulative values as the target task path for executing the task to be executed.
8. A computer device, characterized in that: The computer device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the path planning method as described in any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the path planning method as described in any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the path planning method according to any one of claims 1 to 6.