A target search method and device based on incremental reinforcement learning

Abstract

The application discloses a target searching method based on incremental reinforcement learning, comprising the following steps: performing grid processing on a target space where a first searching task of a UAV is performed, and generating an environment data network; designing a reward function based on a current state of the UAV in the environment data network, an action taken by the current state, and a next state; generating a reinforcement learning training model based on the reward function and a preset searching strategy; in the case that the UAV performs a second searching task, if an evaluation result of evaluating the reinforcement learning training model by the second searching task is that the model needs to be retrained, taking the reinforcement learning training model as a first network branch, taking a model generated by training model parameters corresponding to the second searching task as a second network branch, designing a network structure, and generating a new task adaptation model based on incremental learning; and searching for a target based on the new task adaptation model. By applying the method, efficient target searching can be realized.

Description

Technical Field

[0001] This application relates to the field of target search and tracking, and more particularly to a target search method based on incremental reinforcement learning. Background Technology

[0002] Goal search refers to the process of finding the optimal solution in a given environment, which involves defining the search space, designing search algorithms, and building an environment model. Reinforcement learning is a machine learning method that achieves goal search through the interaction between an agent and its environment.

[0003] The application of traditional reinforcement learning methods in the field of target search typically includes the following steps: (1) Environment modeling: Modeling the environment of the search task, including defining the state space, action space, and reward function. (2) Policy design: Designing the agent's policy, i.e., choosing the optimal action given a state. This may involve various reinforcement learning algorithms, such as Q-learning, deep Q-networks (DQN), etc. (3) Training and optimization: Training and optimization are carried out through the interaction between the agent and the environment, so that it learns the optimal policy. This usually involves backpropagation algorithms based on reward signals, as well as parameter tuning and model updates. Although traditional reinforcement learning methods perform well in some cases, they also have some problems, such as the problem of "catastrophic forgetting" when facing new search tasks, which causes the learned knowledge to be forgotten, thus affecting its application effect.

[0004] Therefore, how to provide a target search method that can better adapt to new search environments while ensuring the search effect of the current task has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a target search method based on incremental reinforcement learning, a target search device based on incremental reinforcement learning, a computer-readable storage medium, and a computer device to solve the problems of low search efficiency and poor adaptability to environmental changes in traditional methods.

[0006] In a first aspect of this application, a target search method based on incremental reinforcement learning is provided, comprising:

[0007] The target space where the UAV performs its first search mission is gridded to generate an environmental data network.

[0008] Based on the current state of the UAV in the environmental data network, the action taken in the current state, and the next state, a reward function is designed. Based on the reward function and the preset search strategy, a reinforcement learning training model is generated. The search strategy is the optimal search strategy defined for the UAV when searching for a target in the target space.

[0009] When the UAV performs a second search task, in response to the evaluation result of the second search task evaluating the reinforcement learning training model, which is to retrain the model, the reinforcement learning training model is used as the first network branch, and the model generated by training with the model parameters corresponding to the second search task is used as the second network branch. The network structure is designed to generate a new task-adaptive model based on incremental learning.

[0010] The target is searched based on the new task adaptation model.

[0011] In a second aspect of this application, a target search apparatus based on incremental reinforcement learning is provided, comprising:

[0012] The processing module is configured to perform gridding on the target space where the UAV performs its first search mission, generating an environmental data network;

[0013] The first model generation module is configured to design a reward function based on the current state of the UAV in the environmental data network, the action taken in the current state, and the next state, and generate a reinforcement learning training model based on the reward function and a preset search strategy. The search strategy is the optimal search strategy defined for the UAV when searching for a target in the target space.

[0014] The second model generation module is configured to, when the UAV performs the second search task, in response to the evaluation result of the second search task evaluating the reinforcement learning training model, which is to retrain the model, design the network structure using the reinforcement learning training model as the first network branch and the model generated by training with the model parameters corresponding to the second search task as the second network branch, and generate a new task-adaptive model based on incremental learning.

[0015] The search module is configured to search for targets based on the new task adaptation model.

[0016] In a third aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described prefix mask quintuple keyword matching method.

[0017] In a fourth aspect of this application, a computer device is provided, comprising:

[0018] Memory and processor;

[0019] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-described prefix mask five-tuple keyword matching method.

[0020] This application provides a target search method based on incremental reinforcement learning, comprising: First, performing gridding processing on the target space where the UAV performs a first search task to generate an environmental data network; then, designing a reward function based on the current state of the UAV, the action taken in the current state, and the next state in the environmental data network, and generating a reinforcement learning training model based on the reward function and a preset search strategy, wherein the search strategy is a defined optimal search strategy corresponding to the UAV searching for targets in the target space; Second, when the UAV performs a second search task, in response to the evaluation result of the second search task evaluating the reinforcement learning training model as retraining the model, designing a network structure using the reinforcement learning training model as a first network branch and the model generated by training with the model parameters corresponding to the second search task as a second network branch, generating a new task-adaptive model based on incremental learning; Finally, searching for targets based on the new task-adaptive model.

[0021] By applying the method provided in the embodiments of this application, leveraging the advantages of reinforcement learning and introducing incremental learning, the search performance of the current task can be better adapted to the new search environment, thereby achieving efficient target search.

[0022] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0023] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0024] Figure 1 This is a flowchart illustrating a target search method based on incremental reinforcement learning provided in an embodiment of this application;

[0025] Figure 2This is a schematic diagram of the reward function corresponding to a target search method based on incremental reinforcement learning provided in an embodiment of this application;

[0026] Figure 3 This is a flowchart illustrating the reinforcement learning training model in an incremental reinforcement learning-based target search method provided in this application embodiment;

[0027] Figure 4 This is a schematic diagram of the reinforcement learning structure in an incremental reinforcement learning-based target search method provided in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the incremental learning network connection mechanism in an incremental reinforcement learning-based target search method provided in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the new task adaptation process of the incremental learning training model in a target search method based on incremental reinforcement learning provided in this application embodiment;

[0030] Figure 7 This is a schematic diagram of the structure of a target search device based on incremental reinforcement learning provided in an embodiment of this application;

[0031] Figure 8 This is a structural block diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0032] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0033] This application provides a target search method based on incremental reinforcement learning, a target search device based on incremental reinforcement learning, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0034] In the target search method based on incremental reinforcement learning provided in this application embodiment, the UAV conducts target search in target space 1. First, the target space 1 needs to be gridded to facilitate subsequent data collection. Then, the collected data is used to train the model to obtain model 1. Next, the target search effect of model 1 on task 2 is tested. If the effect is good, no further training is needed. If the effect is poor, model 2 is redesigned and trained. Here, model 2 is a network architecture based on the incremental idea, which can enable task 2 to achieve good results, while also ensuring that task 1 achieves good results.

[0035] See Figure 1 , Figure 1 This is a flowchart illustrating a target search method based on incremental reinforcement learning provided in an embodiment of this application. Figure 1 As shown, the specific steps include:

[0036] Step S102: Grid the target space where the UAV performs the first search task to generate an environmental data network.

[0037] Specifically, the step of gridding the target space where the UAV performs the first search task to generate an environmental data network includes: dividing the target space into grid cells of the same size based on a preset grid cell specification, wherein each grid cell has a unique identifier, and each grid cell corresponds to a current state of the UAV when it moves to a different grid cell; designing identifiers for the UAV's movement positions within the target space, wherein the movement positions include: moving up, moving down, moving left, and moving right; and generating an environmental data network based on the grid cells and the identifiers.

[0038] In practical applications, gridding the target space is beneficial for subsequent UAV search missions.

[0039] In this embodiment, the target space is first defined as follows: assuming the target space has a spatial range of W*H, it is divided into N*M grid cells, each with a unique identifier (i, j). Each grid cell is considered a current state of the UAV, and its state space is represented by S, where S = {(i,j)|i∈[1,N],j∈[1,M]}. Therefore, the grid cell in which the UAV is located is its current state.

[0040] Next, the action space (i.e., the aforementioned movement position) of the UAV in the target space is designed. This action space includes the UAV's movement directions within the target space, such as upward, downward, leftward, and rightward. In this embodiment, the action space is represented by A, where A = {a1, a2, a3, a4}, a1 represents upward movement, a2 represents downward movement, a3 represents leftward movement, and a4 represents rightward movement.

[0041] Step S104: Based on the current state of the UAV in the environmental data network, the action taken in the current state, and the next state, design a reward function. Based on the reward function and the preset search strategy, generate a reinforcement learning training model. The search strategy is the optimal search strategy defined for the UAV when searching for a target in the target space.

[0042] In this embodiment of the application, for reinforcement learning-based training models, it is necessary to construct a learning environment, define a policy, and a reward function, as detailed below:

[0043] Specifically, the step of designing a reward function based on the current state of the UAV in the environmental data network, the action taken in the current state, and the next state, and generating a reinforcement learning training model based on the reward function and a preset search strategy, includes:

[0044] Design a reward function based on the current state of the drone, the action taken in the current state, and the next state;

[0045] Value functions are used to define the search strategy for the UAV to search for targets in the target space;

[0046] Starting from the initial state, based on the current state of the drone, determine the action to be taken in the current state, observe the reward and the next state after the drone performs the action, until the reinforcement learning training model converges;

[0047] The reinforcement learning training model is tested, and test results are generated, wherein the test results represent the effectiveness of the UAV in searching for targets in the scene.

[0048] It should be noted that the reward function is used to evaluate the quality of a drone performing a specific action under a specific state. In this embodiment, the reward function is R(s,a,s'), where s represents the current state, a represents the action taken, and s' represents the next state.

[0049] In this embodiment of the application, the reward function is first set, which specifically includes the following steps.

[0050] The reward function is designed based on the current state of the drone, the action taken in the current state, and the next state, including:

[0051] In response to the drone moving to a range of one preset cell from the target, the target search is determined to be successful, and the reward is set to a first positive value, wherein the first preset range is the range within which the drone moves to a distance of one cell from the target;

[0052] If the drone moves to a second preset range from the cell where the search target is located, the target search is determined to have failed, and the reward is set to a negative value. The second preset range is the range where the drone moves to a distance of more than one cell from the search target, and the first preset range and the second preset range are non-intersecting spaces.

[0053] More specifically, the response that the drone moves to a second preset range from the cell where the search target is located, and determines that the target search has failed and sets the reward to a negative value, includes:

[0054] In response to the drone moving to a first spatial range within the second preset range, a first negative value is set; in response to the drone moving within the first spatial range, a second negative value is set based on the number of steps the drone moves. The first spatial range includes at least one grid cell within the second preset range, and the second negative value is determined based on the number of steps the drone moves within the first spatial range.

[0055] In response to the drone moving to the spatial boundary of the target space, the reward is set to a first negative value.

[0056] Furthermore, in response to the drone moving from the first spatial range within the second preset range to the second spatial range, the reward is set to a second positive value.

[0057] In practical applications, if the drone moves to within one grid of the target, it is considered to have found the target, an additional positive reward is given, and the search process ends. The reward is set to a positive value, indicating that the drone has successfully found the target.

[0058] If a drone touches a spatial boundary or illegal area, the reward is set to a negative value, indicating that the drone should avoid the collision.

[0059] In other cases, the reward is a small negative value to encourage the drone to find the target as quickly as possible.

[0060] Find the target +5 Entering unreached areas +0.1 Enter the reached area -0.1 Hitting a wall or obstacle -0.1 Each step consumes -0.01

[0061] It should be noted that the event "Target Found" occurs when the drone moves to the first preset range from the cell containing the target, indicating a successful target search and a reward of +5. The event "Entering Reached Area" occurs when the drone moves to the first spatial range from the second preset range from the cell containing the target, even though the drone did not find the target and moved within the first spatial range, resulting in a negative reward of -0.1. The event "Entering Unreached Area" occurs when the drone moves to the second spatial range from the second preset range from the cell containing the target, even though the drone did not find the target but moved from the first spatial range to the second spatial range, resulting in a positive reward of +0.1. The event "Hit a Wall or Obstacle" occurs when the drone moves to the spatial boundary of the target space, resulting in a negative reward of -0.1. The event "Step Cost" refers to the number of steps the drone takes within each spatial range (e.g., the first spatial range, the second spatial range, etc.) of the second preset range, resulting in a negative reward of -0.01.

[0062] See Figure 2 , Figure 2This is a schematic diagram of the reward function corresponding to a target search method based on incremental reinforcement learning provided in an embodiment of this application.

[0063] like Figure 2 As shown, the search target is represented by "Q"; the first preset range is represented by "P"; the second preset range is represented by "O"; the first spatial range of the second preset range is represented by "S"; and the second spatial range of the second preset range is represented by "T". It should be noted that the division of the spatial range in the second preset range can be made according to actual needs. In this embodiment, the second preset range is divided into two spatial ranges, the first spatial range and the second spatial range, as an example (for example, a third spatial range U may also be included).

[0064] Then, in reinforcement learning, the value function Q(s, a) is used to guide the decision-making process. It should be noted that the algorithm's update rule is as follows:

[0065]

[0066] Where Q(s) i ,a i ) is state s i Take action a i The Q-value is given by ∝, where γ is the learning rate and γ is the discount factor.

[0067] Secondly, the model training process specifically includes the following steps: starting from the initial state s0, select action a according to the current state, execute the action, and observe the reward R(s,a,s') and the next state s'. Repeat this process until convergence or the predetermined number of training rounds is reached.

[0068] Next, the generated model is tested: After training, the drone's performance in searching for targets in the scene can be tested. Based on the learned policy, the drone selects and executes actions, and the rewards and search results are observed.

[0069] Finally, we need to determine whether the current model is feasible for the new task, that is, whether the currently trained model is suitable for the new task. The specific steps are as follows:

[0070] (1) Performance evaluation: Evaluate the performance of the current model in the new task, including indicators such as search efficiency.

[0071] (2) Model training judgment: Based on the model's performance in the current task, determine whether the model needs to be retrained.

[0072] See Figure 3 , Figure 3 This is a flowchart illustrating the reinforcement learning training model in a target search method based on incremental reinforcement learning, as provided in an embodiment of this application.

[0073] like Figure 3 As shown, the training process of the Task 1 model includes: spatial gridding; data collection; data processing; reinforcement learning model training and output of Model 1.

[0074] The target search process includes: updating target object information; determining target search behavior based on output model 1; executing target search behavior; determining whether it is a target, and if so, outputting the target object location; if not, updating the target object information and re-executing the target search process.

[0075] See Figure 4 , Figure 4 This is a schematic diagram of the reinforcement learning structure in a target search method based on incremental reinforcement learning provided in an embodiment of this application.

[0076] like Figure 4 As shown, the reinforcement learning structure consists of two parts: an agent and an environment. The environment determines the current state (S) and reward (R) by receiving the agent's action (a), and sends them to the agent so that the agent can search for the target based on the received current state and reward.

[0077] The policy optimization algorithm for the target search task in the incremental reinforcement learning target search method provided in the embodiments of this application can dynamically adjust the agent's policy to achieve better search results.

[0078] Step S106: When the UAV performs the second search task, in response to the evaluation result of the second search task evaluating the reinforcement learning training model, which is to retrain the model, the reinforcement learning training model is used as the first network branch, and the model trained with the model parameters corresponding to the second search task is used as the second network branch. The network structure is designed to generate a new task-adaptive model based on incremental learning.

[0079] In practical applications, when the evaluation result of the second search task on the reinforcement learning training model is to retrain the model, the environmental data gridding, the reinforcement learning training model, and the reward function are consistent with the above-mentioned generation of the environmental data grid, generation of the reinforcement learning training model, and design of the reward function. The embodiments of this application will not be elaborated here.

[0080] Specifically, in the network design process, the network structure design focuses primarily on learning new tasks during training. It utilizes shared parts of the network structure and newly added modules to adapt to and optimize for new tasks. The network structure designed in this embodiment includes two branches: a first branch (Task A) and a second branch (Task B). The Task A branch maintains the original network structure and model parameters, handling the learning and decision-making for the original task. During training, this branch primarily focuses on optimizing the original task, maintaining the model's performance on that task. The Task B branch has the same network structure as the Task A branch, but its model parameters are retrained to adapt to the characteristics and requirements of the new task.

[0081] See Figure 5 , Figure 5 This is a schematic diagram of the incremental learning network connection mechanism in a target search method based on incremental reinforcement learning provided in an embodiment of this application.

[0082] like Figure 5 As shown, this incremental learning network connection mechanism includes an input layer and an output layer. The output layer includes Task A and Task B. Task A includes: output 1, network layer 2, and network layer 1; Task B includes: output 2, network layer 2, and network layer 1. The file 'a' corresponding to network layer 2 of Task A is added to the output layer 2 of Task B, and the file 'a' corresponding to network layer 1 of Task A is added to the network layer 2 of Task B.

[0083] In the new task adaptation model, the first and second network branches are connected across layers to enable information sharing and transmission between Task A and Task B within the network. By dynamically adjusting the weights of the first and second network branches, task balancing and optimization are achieved based on the importance of the tasks.

[0084] During training, in a new task, start from the initial state s0, select action a based on the current state, execute the action, and observe the reward and the next state s'. Repeat this process until convergence or the predetermined number of training rounds is reached.

[0085] Experimental Verification: The performance of models using the modified network and those without was compared in the experiment. The model's effectiveness in searching for targets in both new and old scenarios was tested.

[0086] See Figure 6 , Figure 6 This is a schematic diagram of the new task adaptation process of the incremental learning training model in a target search method based on incremental reinforcement learning provided in this application embodiment.

[0087] like Figure 6As shown, after starting Task 2, the target search behavior is first determined, then Model 1 is adapted to perform the target search behavior, and then the search effect is evaluated. If the search efficiency is high, retraining is not required. The model continues to determine whether the target is found. If it is, the target object location is output; otherwise, Task 2 is restarted. If the search efficiency is poor, retraining is required. The training process of the Task 2 model is as follows: first, the Task 2 space is gridded; then, data is collected; second, the data is processed; third, a new network module is added for reinforcement learning model training; finally, Model 2 is output. The target search process is as follows: first, Task 2 is started; then, based on Model 2, the target search behavior is determined; second, Model 2 is used to perform the target search behavior; third, it is determined whether the target is found. If it is, the target object location is output; otherwise, Task 2 is restarted.

[0088] By applying the incremental reinforcement learning target search method provided in this application embodiment, the incremental learning algorithm and memory module are synthesized, enabling the agent to retain learned knowledge while learning new tasks, effectively solving the "catastrophic forgetting" problem in traditional reinforcement learning methods. Furthermore, this application embodiment also introduces a performance evaluation and adjustment mechanism to monitor the agent's performance in different tasks and adjust and optimize the strategy based on feedback information. This mechanism improves the agent's adaptability and robustness.

[0089] Step S108: Search for the target based on the new task adaptation model.

[0090] The incremental reinforcement learning-based target search method provided in this application first addresses the inefficiency of traditional target search strategies in complex environments. By introducing incremental reinforcement learning, the agent can dynamically learn and optimize its strategy during the search task, thereby improving search efficiency. This advantage helps users find the optimal solution faster in complex environments, saving time and resources. Second, it aims to improve the adaptability of the search method, enabling it to better adapt to new search environments. Traditional reinforcement learning methods may suffer from "catastrophic forgetting" when facing new tasks, where learned knowledge is forgotten, affecting performance. This invention, through the design of an incremental learning algorithm, allows the agent to retain existing knowledge while learning new tasks, improving adaptability. This advantage helps users cope with constantly changing environments, maintaining the stability and robustness of the search method.

[0091] Corresponding to the above method embodiments, this specification also provides embodiments of a target search device based on incremental reinforcement learning. Figure 7 This is a schematic diagram of the structure of a target search device based on incremental reinforcement learning provided in an embodiment of this application. Figure 7 As shown, the device includes:

[0092] Processing module 702 is configured to perform gridding processing on the target space where the UAV performs the first search task, generating an environmental data network;

[0093] The first model generation module 704 is configured to design a reward function based on the current state of the UAV in the environmental data network, the action taken in the current state, and the next state, and to generate a reinforcement learning training model based on the reward function and a preset search strategy. The search strategy is the optimal search strategy defined for the UAV when searching for a target in the target space.

[0094] The second model generation module 706 is configured to, when the UAV performs a second search task, in response to the evaluation result of the second search task evaluating the reinforcement learning training model, which is to retrain the model, design the network structure using the reinforcement learning training model as the first network branch and the model generated by training with the model parameters corresponding to the second search task as the second network branch, and generate a new task-adaptive model based on incremental learning.

[0095] The search module 708 is configured to search for the target based on the new task adaptation model.

[0096] Furthermore, the processing module 702 is further configured to:

[0097] Based on a preset grid cell specification, the target space is divided into grid cells of the same specification. Each grid cell has a unique identifier, and when the UAV moves to each grid cell, it corresponds to a current state of the UAV.

[0098] Design an identifier for the movement position of the UAV within the target space, wherein the movement position includes: moving up, moving down, moving left, and moving right;

[0099] An environmental data network is generated based on the grid cells and the identifier symbols.

[0100] Furthermore, the first model generation module 704 is further configured as follows:

[0101] Design a reward function based on the current state of the drone, the action taken in the current state, and the next state;

[0102] Value functions are used to define the search strategy for the UAV to search for targets in the target space;

[0103] Starting from the initial state, based on the current state of the drone, determine the action to be taken in the current state, observe the reward and the next state after the drone performs the action, until the reinforcement learning training model converges;

[0104] The reinforcement learning training model is tested, and test results are generated, wherein the test results represent the effectiveness of the UAV in searching for targets in the scene.

[0105] Furthermore, the first model generation module 704 is further configured as follows:

[0106] In response to the drone moving to a range of one preset cell from the target, the target search is determined to be successful, and the reward is set to a first positive value, wherein the first preset range is the range within which the drone moves to a distance of one cell from the target;

[0107] If the drone moves to a second preset range from the cell where the search target is located, the target search is determined to have failed, and the reward is set to a negative value. The second preset range is the range where the drone moves to a distance of more than one cell from the search target, and the first preset range and the second preset range are non-intersecting spaces.

[0108] Furthermore, the first model generation module 704 is further configured as follows:

[0109] In response to the drone moving to a first spatial range within the second preset range, a first negative value is set; in response to the drone moving within the first spatial range, a second negative value is set based on the number of steps the drone moves. The first spatial range includes at least one grid cell within the second preset range, and the second negative value is determined based on the number of steps the drone moves within the first spatial range.

[0110] In response to the drone moving to the spatial boundary of the target space, the reward is set to a first negative value.

[0111] Furthermore, the first model generation module 704 is further configured as follows:

[0112] In response to the drone moving from the first spatial range within the second preset range to the second spatial range, the reward is set to a second positive value.

[0113] Furthermore, the second model generation module 706 is further configured as follows:

[0114] In the new task adaptation model, the first network branch and the second network branch are connected across layers. The new task adaptation model is optimized by dynamically adjusting the weights corresponding to the first network branch and the second network branch.

[0115] The incremental reinforcement learning-based target search device provided in this application first addresses the problem of low efficiency in traditional target search strategies when facing various complex environments. By introducing an incremental reinforcement learning method, the agent can dynamically learn and optimize its own strategy in the search task, thereby improving search efficiency. This advantage helps users find the optimal solution faster in complex environments, saving time and resources. Second, it aims to improve the adaptability of the search method, enabling it to better adapt to new search environments. Traditional reinforcement learning methods may suffer from "catastrophic forgetting" when facing new tasks, where learned knowledge is forgotten, thus affecting performance. This invention, through the design of an incremental learning algorithm, enables the agent to retain existing knowledge while learning new tasks, improving adaptability. This advantage helps users cope with constantly changing environments, maintaining the stability and robustness of the search method.

[0116] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the incremental reinforcement learning-based target search apparatus is described simply because it is fundamentally similar to the incremental reinforcement learning-based target search method embodiments; relevant details can be found in the descriptions of the incremental reinforcement learning-based target search method embodiments.

[0117] Figure 8 This is a structural block diagram of a computing device provided in an embodiment of this application. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0118] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0119] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0120] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.

[0121] The processor 820 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described target search method based on incremental reinforcement learning.

[0122] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computing device embodiments are relatively simple in description because they are fundamentally similar to the incremental reinforcement learning-based target search method embodiments; relevant parts can be found in the descriptions of the incremental reinforcement learning-based target search method embodiments.

[0123] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described target search method based on incremental reinforcement learning.

[0124] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiment is described simply because it is substantially similar to the incremental reinforcement learning-based target search method embodiment; relevant details can be found in the description of the incremental reinforcement learning-based target search method embodiment.

[0125] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described target search method based on incremental reinforcement learning.

[0126] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer program embodiments are relatively simple in description because they are fundamentally similar to the incremental reinforcement learning-based target search method embodiments; relevant parts can be found in the descriptions of the incremental reinforcement learning-based target search method embodiments.

[0127] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0128] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0129] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0130] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0131] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A target search method based on incremental reinforcement learning, characterized in that, include: The target space where the UAV performs its first search mission is gridded to generate an environmental data network. Based on the current state of the UAV in the environmental data network, the action taken in the current state, and the next state, a reward function is designed. Based on the reward function and the preset search strategy, a reinforcement learning training model is generated. The search strategy is the optimal search strategy defined for the UAV when searching for a target in the target space. When the UAV performs a second search task, in response to the evaluation result of the second search task evaluating the reinforcement learning training model, which is to retrain the model, the reinforcement learning training model is used as the first network branch, and the model generated by training with the model parameters corresponding to the second search task is used as the second network branch. The network structure is designed to generate a new task adaptation model based on incremental learning. In the new task adaptation model, the first network branch and the second network branch adopt cross-layer connection. The target is searched based on the new task adaptation model.

2. The method according to claim 1, characterized in that, The step of performing gridded processing on the target space where the UAV performs the first search task to generate an environmental data network includes: Based on a preset grid cell specification, the target space is divided into grid cells of the same specification. Each grid cell has a unique identifier, and when the UAV moves to each grid cell, it corresponds to a current state of the UAV. Design an identifier for the movement position of the UAV within the target space, wherein the movement position includes: moving up, moving down, moving left, and moving right; An environmental data network is generated based on the grid cells and the identifier symbols.

3. The method according to claim 1, characterized in that, The step involves designing a reward function based on the current state of the UAV in the environmental data network, the action taken in the current state, and the next state. Based on the reward function and a preset search strategy, a reinforcement learning training model is generated, including: Design a reward function based on the current state of the drone, the action taken in the current state, and the next state; Value functions are used to define the search strategy for the UAV to search for targets in the target space; Starting from the initial state, based on the current state of the drone, determine the action to be taken in the current state, observe the reward and the next state after the drone performs the action, until the reinforcement learning training model converges; The reinforcement learning training model is tested, and test results are generated, wherein the test results represent the effectiveness of the UAV in searching for targets in the scene.

4. The method according to claim 3, characterized in that, The reward function is designed based on the current state of the drone, the action taken in the current state, and the next state, including: In response to the drone moving to a range of one preset cell from the target, the target search is determined to be successful, and the reward is set to a first positive value, wherein the first preset range is the range within which the drone moves to a distance of one cell from the target; If the drone moves to a second preset range from the cell where the search target is located, the target search is determined to have failed, and the reward is set to a negative value. The second preset range is the range where the drone moves to a distance of more than one cell from the search target, and the first preset range and the second preset range are non-intersecting spaces.

5. The method according to claim 4, characterized in that, The response is that if the drone moves to a distance of a second preset range from the cell where the search target is located, the target search is determined to have failed, and the reward is set to a negative value, including: In response to the drone moving to a first spatial range within the second preset range, a first negative value is set; in response to the drone moving within the first spatial range, a second negative value is set based on the number of steps the drone moves. The first spatial range includes at least one grid cell within the second preset range, and the second negative value is determined based on the number of steps the drone moves within the first spatial range. In response to the drone moving to the spatial boundary of the target space, the reward is set to a first negative value.

6. The method according to claim 5, characterized in that, The design of the reward function based on the current state of the drone, the action taken in the current state, and the next state also includes: In response to the drone moving from the first spatial range to the second spatial range within the second preset range, the reward is set to a second positive value.

7. The method according to claim 1, characterized in that, The method further includes: optimizing the new task adaptation model by dynamically adjusting the weights corresponding to the first network branch and the second network branch.

8. A target search device based on incremental reinforcement learning, characterized in that, include: The processing module is configured to perform gridding on the target space where the UAV performs its first search mission, generating an environmental data network; The first model generation module is configured to design a reward function based on the current state of the UAV in the environmental data network, the action taken in the current state, and the next state, and generate a reinforcement learning training model based on the reward function and a preset search strategy. The search strategy is the optimal search strategy defined for the UAV when searching for a target in the target space. The second model generation module is configured to, when the UAV performs a second search task, in response to the evaluation result of the second search task evaluating the reinforcement learning training model as retraining the model, design the network structure using the reinforcement learning training model as the first network branch and the model generated by training with the model parameters corresponding to the second search task as the second network branch, and generate a new task-adaptive model based on incremental learning. In the new task-adaptive model, the first network branch and the second network branch adopt cross-layer connection. The search module is configured to search for targets based on the new task adaptation model.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

10. A computer device, characterized in that, The computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 7.

Images