Unmanned system strategy virtual-real enhancement method based on expert knowledge-decision data fusion

Abstract

The invention discloses an unmanned system strategy virtual-real enhancement method based on expert knowledge-decision data fusion, and belongs to the field of unmanned system intelligent decision. The method comprises the following steps: acquiring decision tracks of experts and unmanned system agents through a real environment man-machine interaction and virtual environment autonomous deduction mode; fusing expert trajectory data and unmanned system agent trajectory data to form a mixed decision trajectory; complementing the mixed decision trajectory reward information based on the reward rule of the task scene; storing the complete mixed decision trajectory and unmanned system agent trajectory data in an experience pool; obtaining trajectory training data from the experience pool, and calculating a virtual-real mixed weight; and calculating the loss of the unmanned system agent strategy network by using the virtual-real mixed weight, and training the unmanned system agent strategy network for track decision of the unmanned system in a real scene. According to the invention, the decision-making efficiency of the unmanned system agent is improved, and the problem that the decision-making model trained in the virtual scene is difficult to directly apply to the real scene when being migrated to the real scene is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous decision-making of unmanned systems and deep reinforcement learning, and specifically refers to a method for virtual-reality enhancement of unmanned system strategies based on expert knowledge-decision data fusion. Background Art

[0002] Autonomous behavior strategy learning for unmanned systems is an important research topic in the field of intelligent unmanned systems. In the process of applying the decision model of unmanned systems from virtual to real, the difference in physical parameters between the virtual and real environments leads to the difference in state transition probabilities, making it difficult for the intelligent agents trained in the virtual environment to achieve the same decision-making efficiency in the real environment as in the virtual environment. Therefore, overcoming the inconsistency in the state transition probabilities between the virtual and real environments during the virtual-to-real migration process is a difficult problem in the autonomous behavior strategy learning of unmanned systems. The current methods mainly alleviate the problem of low decision-making efficiency of unmanned system strategy models caused by the difference between virtual and real environments by automatically modifying simulator parameters and combining reinforcement learning with expert data.

[0003] The method of automatically modifying simulator parameters collects a large amount of observation data in the real environment, uses a classification network to predict whether the simulator parameters are too high, too low, or moderate, and dynamically modifies the simulator parameters to make the state transition probability of the virtual environment close to the real environment, thereby reducing the difference between the virtual and real environments. However, this method requires a large amount of real environment data support and consumes a lot of computing resources for parameter search, so it has poor usability when real data is scarce.

[0004] The method of combining reinforcement learning with expert data can effectively combine expert demonstration data and reinforcement learning to speed up the learning process, but the processing of perceptual input is relatively limited and may not fully utilize the information of perceptual input. For complex tasks in real environments, especially those that require flexible response to perceptual information in the environment, this method may not be suitable.

[0005] In summary, most of the current unmanned system autonomous behavior strategy learning technologies for the virtual-to-reality migration process focus on a large amount of real-world environment data support and reinforcement learning, and often rely on a large amount of task environment data collection and retraining. In the context of unmanned systems, the operation is cumbersome and inefficient, and the usability is poor in practical applications. Summary of the invention

[0006] Due to the large difference in perceived image pixels between virtual scenes and real scenes, it is difficult to directly apply the decision model trained in the virtual scene to the real scene. In order to solve these problems, the present invention proposes a virtual-reality enhancement method for unmanned system strategies based on expert knowledge-decision data fusion. By introducing domain expert knowledge and using expert demonstration data collected from the real environment, the unmanned system decision model trained in the virtual environment is fine-tuned to enable it to better adapt to the state transition model of the real environment, thereby improving decision-making efficiency.

[0007] The unmanned system strategy virtual reality enhancement method based on expert knowledge-decision data fusion of the present invention comprises the following steps:

[0008] Step 1: For the current task, obtain the expert decision trajectory sequence through real-world human-computer interaction and virtual-environment autonomous deduction and the decision trajectory sequence of the unmanned system agent ; The trajectory sequence and Each step consists of a series of state triplets, and the state triplet of each step includes the current system state, the current unmanned system action, and the next system state;

[0009] Step 2: Fusion of expert trajectory data and unmanned system agent trajectory data to form a hybrid decision trajectory sequence; first, Each state triple in Find the tuple that has the same current system state and next system state as the tuple, match the two state triples, and set the step number of the matching tuples to be consistent; then, The current unmanned system action of the agent decision in each state triplet is replaced by the current unmanned system action of the matching expert decision, and a mixed decision trajectory sequence is obtained. ;

[0010] Step 3: Complete the reward information of the hybrid decision trajectory based on the reward rule of the unmanned system mission scenario; for each state triplet in the hybrid decision trajectory sequence, calculate the reward of the current system state based on the reward rule of the unmanned system mission scenario, and complete the reward of the state triplet to obtain a quadruple;

[0011] Step 4: Complete the mixed decision trajectory sequence after the reward and the trajectory sequence of the unmanned system agent The data is saved to the unmanned system strategy virtual reality enhancement experience pool middle;

[0012] Step 5: Enhance the experience pool from the unmanned system strategy Obtain the agent decision trajectory training data and mixed trajectory training data, and calculate the virtual-real mixed weight for each quadruple in the mixed trajectory training data;

[0013] Step 6: Use the virtual-real mixed weight to calculate the loss of the unmanned system intelligent agent policy network, update the policy network parameters, train the unmanned system intelligent agent policy network, and use the trained unmanned system intelligent agent policy network for trajectory decision-making of the unmanned system in real scenarios.

[0014] In the step 1, the reward value threshold H of the trajectory, the minimum number of tuples Min, and the reward function of the state triple are pre-set; the trajectory sequence is generated include:

[0015] Step a1) For the trajectory sequence U decided by the unmanned system agent, use the reward function to calculate the state reward value of each state triple in the sequence U, remove the state triple according to the state reward value, and accumulate the state reward value of the state triple that has not been removed in the sequence to obtain the trajectory reward value; suppose the sequence U' is obtained after the removal operation, and judge whether U' meets the conditions: the trajectory reward value is greater than the threshold H, and the number of tuples in the trajectory sequence U' is greater than Min and less than The number of tuples in the tuple; if satisfied, U' is used as the trajectory sequence ; If not satisfied, go to step a2;

[0016] Step a2) Generate a trajectory sequence U again through the unmanned system agent and execute step a1 until a trajectory sequence that meets the conditions is found. .

[0017] The step 2 comprises:

[0018] Step 2-1) Match the state triples: For the decision trajectory sequence of the unmanned system agent Each state triple in , traverse the expert decision trajectory sequence in sequence For each state triple in , find the state triple ,in and ;in, The unmanned system representing the agent's decision is The action of the step, Indicates the agent's decision The system status of the step, Indicates the agent's decision The system status of the step, The unmanned system representing expert decision making is The action of the step, Indicates the expert decision The system status of the step, Indicates the expert decision The system status of the step;

[0019] Step 2-2) Fusion trajectory data: Each state triple in Actions in Replace with matching state triples Actions in , and obtain the triplet of fused trajectory data , forming a mixed decision trajectory sequence .

[0020] The step 5 comprises:

[0021] Step 5-1) From the experience pool In the complete mixed decision trajectory sequence data, random sampling is used Extract hybrid trajectory training data , from the decision trajectory sequence data of the unmanned system agent Extract agent decision trajectory training data from ;

[0022] Step 5-2) Training data on mixed trajectories Each quadruple in Calculate the virtual-real mixing weight of the tuple and mark the quadruple , calculate the virtual-real mixed weight of the quaternary as follows:

[0023]

[0024] in, and is a hyperparameter, is the number of fine-tuning trainings for decision actions, represents the advantage function, Represents the decision network model of the unmanned system agent, and the network parameters are .

[0025] The advantages and positive effects of the present invention are as follows: compared with the method of automatically modifying simulator parameters and combining reinforcement learning with expert data, the method of the present invention integrates the trajectory data generated by the knowledge body in the real environment and the decision trajectory data of the intelligent body itself, and uses these mixed data to fine-tune the decision model obtained by virtual environment training, so that it can make actions that are closer to the actual environment state transition probability, reducing the degree of dependence on real data. The method of the present invention can not only effectively improve the generalization ability of the model and overcome the problems caused by the difference in virtual and real state transition probabilities, but also greatly reduce the dependence on real environment data, and is more suitable for practical application scenarios. Experiments have also proved that the use of the method of the present invention can effectively improve the decision-making efficiency of unmanned vehicles by introducing expert knowledge. Unmanned vehicles do not need to use trial and error methods to learn the skills of staying in the lane, which solves the problem that the decision model trained in the virtual scene is difficult to directly apply in the real scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a method for enhancing the virtual reality of an unmanned vehicle strategy based on expert knowledge-decision data fusion according to an embodiment of the present invention;

[0027] Figure 2 This is the simulation effect diagram of the lane keeping task of an unmanned vehicle in a real scenario. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below using an unmanned vehicle as an example of an unmanned system, combined with a specific unmanned vehicle lane keeping task example, and with reference to the accompanying drawings.

[0029] The present invention proposes a virtual-reality enhancement method for unmanned system strategies based on expert knowledge-decision data fusion. First, expert decision data and agent decision data are obtained through human-computer interaction in real environment and autonomous deduction in virtual environment, and then the mixed decision data is formed; then the mixed decision trajectory reward information is completed based on the reward rules of the unmanned system mission scenario, and the virtual-reality mixed weight is calculated; finally, the loss of the agent strategy network is calculated under the reinforcement learning training framework, and the strategy network parameters are updated. The basic features of the present invention are mainly the following aspects: First, by fusing the trajectory data generated by experts in the real environment and the agent's own decision trajectory data; second, the decision model obtained by virtual environment training is enhanced with virtual-reality mixed data.

[0030] like Figure 1 As shown, the specific implementation steps of the unmanned vehicle strategy virtual-reality enhancement method based on expert knowledge-decision data fusion in an embodiment of the present invention include the following 6 steps, and each step is described below.

[0031] Step 1: Obtain the expert decision trajectory sequence and the unmanned system agent decision trajectory sequence through human-computer interaction in the real environment and autonomous deduction in the virtual environment. The specific process is as follows:

[0032] (1-1) Definition of unmanned system trajectory sequence: The trajectory sequence of an unmanned system is defined as a set , it is assumed that Secondary trajectory collection, Included Unmanned system tracks: ;in, Indicates Each trajectory consists of a series of state tuples from the start of the task to the end of the task: ,in, The triple represents the Step status, including System status of the step ,action and System status of the step , is the initial state tuple. The system state is set according to different task requirements, and the action refers to the action performed by the unmanned system, which is set according to different tasks.

[0033] In the lane keeping task of the unmanned vehicle in the embodiment of the present invention, the system state Represents unmanned vehicle The RGB image data of the visual sensor of the step, the image size is 84*84*3 pixels; Represented as drone The two-dimensional continuous action of the step, including speed and direction, is normalized so that the value range of speed and direction is [0,1] and [-1,1] respectively.

[0034] (1-2) Generation of decision trajectory sequence: In a real environment, through human-computer interaction, experts directly operate the unmanned system through wireless remote control, perform tasks and collect decision trajectories, thereby obtaining the expert decision trajectory sequence. Run the unmanned system agent model in a virtual simulation environment, perform tasks and collect decision trajectory data of the unmanned system, and obtain the decision trajectory sequence of the unmanned system agent .

[0035] In the unmanned vehicle lane keeping task of the embodiment of the present invention, for each unmanned vehicle, that is, the trajectory sequence and Each contains only one trajectory of the unmanned vehicle performing the task, and the trajectory sequence and It can be directly composed of state triplets in the trajectory.

[0036] Step 2: Fuse the expert decision trajectory data and the unmanned system agent decision trajectory data to form a hybrid decision trajectory sequence. The specific process is as follows:

[0037] (2-1) State triplet matching: For the decision trajectory sequence of the unmanned system agent Each state triple in , traverse the expert decision trajectory sequence in sequence For each state triple in , find the state triple ,in, and Among them, the superscript Indicates the decision-making of the unmanned system agent, with the superscript represents the expert decision-making situation, The unmanned system representing the agent's decision is The action of the step, Indicates the agent's decision The system status of the step, Indicates the agent's decision The system status of the step, The unmanned system representing expert decision making is The action of the step, Indicates the expert decision The system status of the step, Indicates the expert decision The system status of the step.

[0038] The purpose of this step is to and Align the state triples in The state triplet of each step in Find the triples that match the system state in the matching sequence, so that the step number of each step state tuple of the two sequences after matching is consistent, and the state information in each step is consistent.

[0039] In the unmanned vehicle lane keeping task, after triple matching processing, the Matching new , after matching Middle Step triple and Middle Step triple Number of steps Remain consistent and the state information (image) currently perceived by the unmanned vehicle and Maintain consistency and the next step of the driverless car's perception status information and Be consistent.

[0040] Since the unmanned system agent makes a decision trajectory sequence It is randomly generated by the unmanned system agent in the virtual environment, which may have poor results. right When matching tuples in the virtual environment, there may be a small number of matched tuples. In this case, a new unmanned system trajectory will be generated in the unmanned system agent in the virtual environment and reused. right The method of the present invention pre-sets the reward value threshold H and the minimum number of tuples Min of the trajectory, and sets the reward calculation function of the state tuple. For each trajectory sequence generated by the unmanned system intelligent agent, the reward value of each state tuple in the sequence is calculated, and the state tuple is eliminated. The reward value of the state tuple is accumulated for the trajectory sequence after the elimination operation to determine whether it meets the requirements: the accumulated trajectory reward value is greater than the threshold H, the number of tuples in the trajectory sequence meets the number Min and is less than If the requirement is met, the trajectory sequence after the removal operation is used as , perform triple matching operation. If the requirements are not met, the unmanned system agent will re-determine a trajectory sequence. The threshold H and the minimum number of tuples Min can be selected based on experience.

[0041] The embodiment of the present invention can also use the unmanned system agent to generate multiple trajectories in advance, count the reward values ​​of the tuple nodes in each trajectory, and if the system state is out of the lane, set the state reward to -1, and if it is not out of the lane, set the state reward to 1, and accumulate the reward values ​​of each trajectory. Directly eliminate trajectories with poor effects according to the preset reward value threshold H. Select the trajectory with the highest cumulative reward value from the trajectories that meet the reward value threshold, and eliminate the tuples in the trajectory according to the state reward, so that the number of tuples in the trajectory sequence meets the minimum number of tuples Min requirement and is less than If the highest cumulative reward value still does not meet the preset reward value threshold, the tuples in the trajectory with the highest cumulative reward value can be further eliminated according to the state reward, so that the path meets the threshold H requirement, and the number of tuples in the trajectory sequence meets the number Min requirement and is less than If a path that meets the requirements cannot be obtained after tuple elimination, the unmanned system agent is used again to generate a set of trajectories and the above judgment and selection are repeated.

[0042] In addition, it should be noted that when judging whether the current system state or the next system state of two pairs of tuples is the same, if the system states are basically the same and there is no big difference, they are considered to be the same. For example, in the lane keeping task of the embodiment of the present invention, when the current state images in the two pairs of tuples are basically the same, such as the scenes in the two images are the same and they are both located on the same side of the road, it can be considered that If the scenes in the two images are the same but located on different sides of the road, then it is considered Specifically, the standard for judging whether the status is consistent can be pre-set according to the task requirements.

[0043] (2-2) Trajectory data fusion: transform the state triplet Agent actions in Replace with matching state triples Expert Actions in , and obtain the mixed decision trajectory triplet . The decision trajectory sequence of the unmanned system agent Each state triplet in is subjected to trajectory data fusion to form a mixed decision trajectory sequence .

[0044] Step 3: Complete the hybrid decision trajectory reward information based on the reward rules of the unmanned system mission scenario. The specific process is as follows:

[0045] (3-1) Setting of unmanned system task reward rules: According to the task objectives of the unmanned system, set the unmanned system decision reward rules to evaluate the value of each unmanned system state. In the unmanned system lane keeping task, the state reward of the unmanned system leaving the lane can be set to -1, and the state reward of not leaving the lane can be set to 1.

[0046] (3-2) For mixed decision trajectory sequence Each mixed decision trajectory triplet in , according to the current task reward rules of the unmanned system, calculate the state The following rewards , thus forming a mixed decision trajectory quadruple .

[0047] Step 4: Save the complete hybrid decision trajectory data after the reward and the unmanned system agent trajectory data into the experience pool. The specific process is as follows:

[0048] The mixed decision trajectory sequence data after reward completion and unmanned system agent decision trajectory sequence data , uniformly saved to the unmanned system strategy virtual reality enhancement experience pool middle.

[0049] Step 5: Obtain trajectory data from the unmanned system strategy virtual-real augmentation experience pool and calculate the virtual-real mixed weight. The specific process is as follows:

[0050] (5-1) Trajectory training data acquisition: from the experience pool In the mixed decision trajectory sequence data, random sampling is used. and unmanned system agent decision trajectory sequence data Extract the mixed trajectory training data , agent decision trajectory training data .in, Represents the decision network model of the unmanned system agent, and its network parameters are .

[0051] (5-2) Virtual-real mixed weight calculation: For each mixed trajectory training data The data tuple in , calculate the virtual-real mixed weight of the data tuple according to the following formula :

[0052]

[0053] in, and is a hyperparameter, is the number of fine-tuning trainings for decision actions, represents the advantage function. Controls the weight of the virtual-real blend The weight size, The weights gradually decay as the number of fine-tuning increases. By controlling the hyperparameters, in the early stages of training, the mixed trajectory training data is given a larger weight to guide the strategy to quickly adapt to the real environment; as the training progresses, the role of the mixed trajectory training data gradually weakens, and the algorithm relies more on reinforcement learning objectives to improve the strategy in the real environment.

[0054] Step 6: Use the virtual-real mixed weight to calculate the loss of the unmanned system agent policy network and update the policy network parameters. The specific process is as follows:

[0055] (6-1) Calculation of unmanned system strategy network parameter loss: Mix the virtual and real weights Join the Unmanned Systems Agent Policy Network During the training process, the parameter loss of the unmanned system strategy network is calculated by the following formula :

[0056]

[0057] in, Indicates the calculation strategy network parameters The gradient of Indicates status Get the action The natural logarithm of the probability.

[0058] (6-2) Policy network parameter update: Policy network for unmanned systems , update the parameters through the gradient descent algorithm , to minimize the parameter loss , the network training times is 1000 times.

[0059] It can be seen from the above steps 5 and 6 that the method of the present invention can effectively improve the training efficiency of the intelligent agent (unmanned system) by introducing expert knowledge, and reduce the ineffective exploration of the intelligent agent. For example, in the task of training the unmanned vehicle to keep driving in the lane in the embodiment of the present invention, when expert knowledge is not introduced, the unmanned vehicle needs to interact with the environment for tens of thousands of experiments to learn to drive in the lane. However, after the expert knowledge is introduced in the present invention, the unmanned vehicle only needs one experiment to learn to drive in the lane, because the current decision state and action of the unmanned vehicle are aligned with the expert knowledge, and after alignment, a correct trajectory is selected. It can be seen from this that the mixed decision trajectory sequence introducing expert knowledge is obtained through the early steps 1 and 2 of the method of the present invention, and then the data is stored in the experience pool through steps 3 and 4, so that the policy network of the unmanned system intelligent agent is trained using steps 5 and 6, thereby improving the decision-making performance of the unmanned vehicle in migrating to the real scene.

[0060] Because it is difficult to transfer the interaction data in the virtual scene to the real scene at one time to achieve significant results, a fine-tuning method is needed to optimize the decision-making performance in the real scene. The trained unmanned system agent strategy network is used to make trajectory decisions for the unmanned system in the real environment.

[0061] The present invention applies the unmanned system intelligent decision network to the lane keeping task of a real unmanned vehicle, and the decision effect is significant.

[0062] Experimental description and results: The experimental scenario is the unmanned system lane keeping task scenario described in the implementation steps. The scenario is virtually simulated by Unity3D, and a real one-to-one scene is built indoors. Table 1 shows the comparison results of the method of the present invention and other benchmark methods in the task scenario.

[0063] Table 1 Comparison of experimental results

[0064]

[0065] Among them, the two benchmark methods for comparison are PPO and NO_KDF. PPO represents the proximal policy optimization deep reinforcement learning model, without virtual and real model enhancement, that is, only the policy network is used without expert knowledge. NO_KDF means that the model is directly enhanced using expert data, without using the policy network for decision-making. This model is used to prove the effect of the method of the present invention not only using expert knowledge but also using the policy network to make decisions together. The two models were compared by ablation experiments relative to the method of the present invention.

[0066] The evaluation index is the number of successful tasks. The number of successful tasks in Table 1 refers to the number of successes of the agent in 100 rounds. The closer the number is to 100, the better the virtual-real migration performance of the model. The unmanned system strategy virtual-real enhancement method based on expert knowledge-decision data fusion of the present invention fuses the trajectory data generated by the knowledge body in the real environment and the decision trajectory data of the agent itself, and uses these mixed data to perform virtual-real enhancement on the decision model trained in the virtual environment, so that it can make actions that are closer to the actual environment state transition probability, reducing the degree of dependence on real data. It can be seen from the experimental results that the present invention has the highest number of successful tasks in the unmanned vehicle lane keeping task scenario.

[0067] like Figure 2 As shown, this is a simulation result of the method of the present invention in a real scenario of an unmanned vehicle lane keeping task. From the states of the unmanned vehicle at the three moments (a), (b), and (c), it can be seen that the method of the present invention can maintain the lane during the obstacle avoidance process and has practical applicability.

[0068] Except for the technical features described in the specification, all of them are known technologies to those skilled in the art. The present invention omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.

Claims

1. A method for enhancing the virtual reality of unmanned system strategy based on expert knowledge-decision data fusion, characterized in that: The steps include: Step 1: For the current task, obtain the expert decision trajectory sequence through real-world human-computer interaction , obtain the decision trajectory sequence of the unmanned system agent through autonomous deduction in the virtual environment ; The trajectory sequence and They are composed of a series of state triplets, each of which contains the current system state, the current unmanned system action, and the next system state; Step 2: Fusion expert decision trajectory sequence and unmanned system agent decision trajectory sequence to form a hybrid decision trajectory sequence; first, Each state triple in Find the tuple that has the same current system state and next system state as the tuple, match the two state triples, and set the step number to be consistent; then, The current unmanned system action of the agent decision in each state triplet is replaced by the current unmanned system action of the matching expert decision, and a mixed decision trajectory sequence is obtained. ; Step 3: Mixed decision trajectory sequence For each state triplet in the system, the reward of the current system state is calculated based on the reward rule of the unmanned system mission scenario, and the state triplet is completed to obtain a quadruple. Step 4: Complete the mixed decision trajectory sequence after the reward and the decision trajectory sequence of the unmanned system agent Save to the unmanned system strategy virtual reality enhancement experience pool middle; Step 5: Enhance the experience pool from the unmanned system strategy Obtain the agent decision trajectory training data and mixed trajectory training data, and calculate the virtual-real mixed weight for each quadruple in the mixed trajectory training data; Step 6: Use the virtual-real mixed weights to calculate the loss of the unmanned system intelligent agent policy network, update the policy network parameters, and train the unmanned system intelligent agent policy network.

2. The method according to claim 1, characterized in that: The step 1 comprises: Step 1-1) Define the trajectory sequence of the unmanned system: Define the trajectory sequence of the unmanned system as a set , if the unmanned system is Second trajectory collection, then the collection Include The trajectory is represented by , among which Tracks It consists of a series of state tuples from the start of the task to the end of the task, expressed as , is the initial state tuple, Step state triplet Contains Step system status ,action and Step system status ; Step 1-2) Generate decision trajectory sequence: In a real environment, through human-computer interaction, experts directly operate the unmanned system, perform tasks and collect unmanned system trajectories to obtain expert decision trajectory sequence ; Run the unmanned system agent model in a virtual simulation environment, perform tasks and collect unmanned system trajectories, and obtain the unmanned system agent decision trajectory sequence .

3. The method according to claim 1, characterized in that: In the step 1, the reward value threshold H of the trajectory, the minimum number of tuples Min, and the reward function of the state triple are pre-set; the trajectory sequence is generated include: Step a1) For the trajectory sequence U decided by the unmanned system agent, use the reward function to calculate the state reward value of each state triple in the sequence U, remove the state triple according to the state reward value, and accumulate the state reward values ​​of the state triples that have not been removed in the sequence to obtain the trajectory reward value; suppose the sequence U' is obtained after the removal operation, and judge whether U' meets the conditions: the trajectory reward value is greater than the threshold H, and the number of tuples in the trajectory sequence U' is greater than Min and less than The number of tuples in the tuple; if satisfied, U' is used as the trajectory sequence ; If not satisfied, go to step a2; Step a2) Generate a trajectory sequence U again through the unmanned system agent and execute step a1 until a trajectory sequence that meets the conditions is found. .

4. The method according to claim 1, characterized in that: The step 2 comprises: Step 2-1) Match the state triples: For the decision trajectory sequence of the unmanned system agent Each state triple in , traverse the expert decision trajectory sequence in sequence For each state triple in , find the state triple ,in and ;in, The unmanned system representing the agent's decision is The action of the step, Indicates the agent's decision The system status of the step, Indicates the agent's decision The system status of the step, The unmanned system representing expert decision making is The action of the step, Indicates the expert decision The system status of the step, Indicates the expert decision The system status of the step; Step 2-2) Fusion trajectory data: Each state triple in Actions in Replace with matching state triples Actions in , and obtain the triplet of fused trajectory data , forming a mixed decision trajectory sequence .

5. The method according to claim 1 or 4, characterized in that: The step 5 comprises: Step 5-1) From the experience pool In the complete mixed decision trajectory sequence data, random sampling is used Extract hybrid trajectory training data , from the decision trajectory sequence data of the unmanned system agent Extract agent decision trajectory training data from ; Step 5-2) Training data on mixed trajectories Each quadruple in Calculate the virtual-real mixing weight of the tuple and mark the quadruple , calculate the virtual-real mixed weight of the quaternary as follows: in, and is a hyperparameter, is the number of fine-tuning trainings for decision actions, represents the advantage function, Represents the decision network model of the unmanned system agent, and the network parameters are .

6. The method according to claim 5, characterized in that In step 5, when calculating the virtual-real mixed weight of the quadruple, set the hyperparameter , .

7. The method according to claim 5, characterized in that In step 6, the virtual-real mixed weight Join the Unmanned Systems Agent Policy Network During the training process, the parameter loss of the unmanned system strategy network is calculated by the following formula : in, Indicates the calculation strategy network parameters gradient.

Images