Unmanned ship multi-level task decision-making method combining large model and reinforcement learning

Abstract

The invention discloses an unmanned ship multi-level task decision-making method combining a large model and reinforcement learning, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining an environment observation value of an unmanned ship in a current state; taking the environment observation value and the current state of the unmanned ship as the environment state of the unmanned ship, and inputting the environment state into a strategy model to obtain an execution strategy of the unmanned ship; wherein the strategy model is obtained based on a large language model and reinforcement learning training. According to the unmanned ship multi-level task decision-making method combining the large model and the reinforcement learning, a teacher-student method is utilized, and the executive ability of an unmanned ship in a complex confrontation environment is remarkably improved by fusing the confrontation rule guidance of the large language model and the dynamic decision-making ability of the reinforcement learning.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a multi-level task decision-making method for unmanned boats that combines a large model with reinforcement learning. Background Art

[0002] In modern maritime confrontation training, the opposing island chain often relies on a multi-layered dynamic defense system based on radar, communications, and command posts, creating a highly challenging confrontation training environment. Taking a typical island chain penetration mission as an example, the unmanned vehicle must penetrate three layers of defense: an outer layer of electronic barriers, a middle layer of encrypted communications vessels, and a core underground command post physically protected by tidal shelters. The unmanned vehicle must complete a nested task chain of "electronic deception, signal decryption, and dynamic strike" within a single mission. This complexity is reflected in the triple coupling constraints: the dynamic upgrade of the opponent's encryption protocol and radar recognition algorithm (iteration every 30 minutes), the subsequent path blocking caused by the failure of the previous mission, and the strict physical limitations of the unmanned vehicle's own computing power and endurance.

[0003] Current reinforcement learning approaches encompass a variety of approaches to address diverse learning tasks and objectives. For example, deep reinforcement learning utilizes deep learning models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), to handle tasks with high-dimensional input spaces. Through interaction with the environment, the intelligent agent network autonomously learns control policies.

[0004] However, for modern maritime adversarial training scenarios, traditional reinforcement learning algorithms rely on a single, sparse reward, resulting in inefficient early exploration. In simulated training, unmanned boats, lacking phased goal guidance, often wander randomly within the first defense zone until their energy is exhausted. Furthermore, static training strategies struggle to cope with the dynamics of adversarial environments. These issues make existing reinforcement learning techniques difficult to apply to complex mission scenarios. Summary of the Invention

[0005] To address the aforementioned issues in the prior art, the present invention provides a multi-level task decision-making method for unmanned vehicles that combines a large model with reinforcement learning. The technical problems to be solved by the present invention are achieved through the following technical solutions:

[0006] The present invention provides a multi-level task decision-making method for unmanned boats that combines a large model with reinforcement learning, including:

[0007] Step 1: Obtain the environmental observation value of the unmanned boat in its current state;

[0008] Step 2: The environmental observation value and the current state of the unmanned boat are used as the environmental state of the unmanned boat, and input into the strategy model to obtain the execution strategy of the unmanned boat; wherein, the strategy model is obtained based on a large language model and reinforcement learning training.

[0009] In one embodiment of the present invention, the environmental observation values ​​include basic environmental observation values ​​and target observation values.

[0010] In one embodiment of the present invention, the training process of the policy model includes:

[0011] S1: The interaction process between the unmanned boat and the environment is abstracted into a sequential decision problem of a Markov decision process, expressed as<S,A,τ,R,γ> , where S is the state space, A is the action space, τ is the transition probability function, R is the single-step reward obtained from the environment, and γ is the discount factor;

[0012] S2: Collect policy experience data using the student model after policy initialization;

[0013] S3: Optimizing the strategy network of the student model using the strategy experience data to obtain a preliminarily trained student model;

[0014] S4: Build and initialize a teacher model based on a large language model;

[0015] S5: Utilize the teacher model to perform strategy network optimization on the preliminarily trained student model to obtain a trained student model, and use the trained student model as the strategy model.

[0016] In one embodiment of the present invention, the S2 includes:

[0017] S21: If there is a pre-trained student model, input the environmental state of the unmanned vehicle into the pre-trained student model to obtain the action and action reward of the unmanned vehicle, and store the environmental state of the unmanned vehicle and its corresponding action and action reward as the strategy experience data in the experience buffer;

[0018] S22: If there is no pre-trained student model, a policy neural network Actor module and a value neural network Critic module are constructed as the student model based on the proximal policy optimization algorithm, the environmental state of the unmanned boat is input into the policy neural network Actor module and the value neural network Critic module, the action and action reward of the unmanned boat are obtained, and the environmental state, action, action reward and environmental state after the action is executed of the unmanned boat are stored in the experience buffer as the policy experience data.

[0019] In one embodiment of the present invention, when there is a pre-trained student model, S3 includes:

[0020] S311: extracting a preset amount of experience data from the experience buffer;

[0021] S312: Calculating a first loss value of the extracted experience data according to a preset first loss function;

[0022] S313: Performing strategy network optimization on the pre-trained student model according to the first loss value to obtain the preliminarily trained student model.

[0023] In one embodiment of the present invention, the first loss function is expressed as:

[0024] L total =βL policy +(1-β)L value ;

[0025] Among them, L policy =-E[log(π θ (a t |s t ))A t ]+λ(π θ (a t |s t )) α -1) 2 ;

[0026] L value =E[(V new (s t )-V old (s t )) 2 ];

[0027] Where, L total is the first loss function, L policy is the policy loss function, L value is the value loss function, β is the weight parameter, s t represents the environmental state of the unmanned boat at time t, a t Represents the environmental state s t The corresponding action, π θ (a t |s t ) indicates that in the environmental state s t Next, perform action a t The probability of A t is the advantage function, λ is the regularization parameter, α is a constant greater than 1, E represents the expectation, V new (s t ) represents the new value function for the environment state st The estimated value of V old (s t ) represents the old value function for the environment state s t estimated value.

[0028] In one embodiment of the present invention, when there is no pre-trained student model, S3 includes:

[0029] S321: extracting a preset amount of experience data from the experience buffer;

[0030] S322: Input the extracted experience data into the value neural network critic module for training, and update the network parameters of the value neural network critic module according to the temporal difference error;

[0031] S323: Optimizing the policy network of the policy neural network Actor module according to the estimated advantage function obtained by the value neural network Critic module to obtain the preliminarily trained student model.

[0032] In one embodiment of the present invention, the timing differential error is expressed as:

[0033]

[0034] The estimated advantage function is expressed as:

[0035]

[0036] The objective of the policy network optimization is expressed as:

[0037]

[0038] Where, δ t represents the timing difference error, r t represents the reward parameter, γ represents the discount factor, s t represents the environmental state of the unmanned boat at time t, s t+1 represents the environmental state of the unmanned boat at time t+1, In the strategy π θ The environment is in state s t+1 The value function of In the strategy π θ The environment is in state s t The value function of To estimate the advantage function, k represents the number of steps backtracking in the calculation of the estimated advantage function, σ t+l represents the time series difference error at time t+l, β represents the trade-off parameter, represents the optimization goal of the policy network, is the network parameter of the policy neural network Actor module, clip is the truncation function, θ′ represents the network parameter of the value neural network Critic module, Indicates that the policy parameters are The strategy ratio when ∈ represents the truncation parameter, Represents an estimate of expectation.

[0039] In one embodiment of the present invention, the S4 includes:

[0040] Building a teacher model including a translator and a pre-trained large language model, and setting a set of strategy options in the teacher model;

[0041] The translator is used to generate a text description according to the environmental status of the unmanned boat, and the pre-trained large language model is used to output the execution strategy of the unmanned boat according to the text description, user intention instructions and the strategy option set.

[0042] In one embodiment of the present invention, the S5 includes:

[0043] The environmental state of the unmanned boat is input into the teacher model and the preliminarily trained student model respectively, and the policy network optimization of the preliminarily trained student model is achieved by minimizing a preset second loss function, where the second loss function is expressed as:

[0044]

[0045] Where, is the second loss function, is the loss function used to encode environmental feedback in reinforcement learning algorithms, represents the expectation of the environment state, H(π T (·|s)||π θ (·|s)) represents the regularization term for the difference between the execution strategies generated by the teacher model and the initially trained student model, s represents the environment state, represents the degradation parameter, θ represents the policy network parameter of the student model, π T (·|s) represents the execution strategy generated by the teacher model, π θ (·|s) represents the execution strategy generated by the student model.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention's multi-level task decision-making method for unmanned boats that combines a large model with reinforcement learning utilizes a teacher-student approach and significantly improves the execution capability of unmanned boats in complex adversarial environments by integrating the adversarial rule guidance of a large language model with the dynamic decision-making capability of reinforcement learning. By generating interpretable adversarial logs through a large language model, the decision-making process is made transparent, allowing for real-time adjustment of strategy weights through natural language instructions to ensure consistency between the unmanned boat's artificial intelligence decisions and adversarial intentions. A dynamic adversarial learning framework is introduced to perceive changes in the adversary's defense strategy in real time, adjust the decision-making model online, predict the adversary's behavior pattern through a large language model, and quickly generate adaptive execution strategies in combination with reinforcement learning, thereby significantly improving the unmanned boat's penetration success rate.

[0048] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of a multi-level task decision-making method for an unmanned boat combining a large model and reinforcement learning, provided by an embodiment of the present invention;

[0050] Figure 2 This is a mission diagram of an unmanned boat provided by an embodiment of the present invention;

[0051] Figure 3 This is a block diagram of a training process of a strategy model provided by an embodiment of the present invention;

[0052] Figure 4 This is a flow chart of a teacher model generation execution strategy provided by an embodiment of the present invention;

[0053] Figure 5 This is an example diagram of a strategy model training for an unmanned boat-based mission provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of a multi-level task decision-making method for an unmanned boat combining a large model and reinforcement learning proposed in accordance with the present invention, in conjunction with the accompanying drawings and specific implementation methods.

[0055] The aforementioned and other technical contents, features, and effects of the present invention are clearly presented in the following detailed description of the specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the intended purpose can be obtained. However, the accompanying drawings are provided for reference and illustration purposes only and are not intended to limit the technical solutions of the present invention.

[0056] The embodiment of the present invention provides a multi-level task decision-making method for unmanned boats that combines a large model with reinforcement learning. Figure 1 , Figure 1 Schematic diagram of a multi-level task decision-making method for an unmanned boat combining a large model and reinforcement learning provided by an embodiment of the present invention. Figure 1 As shown, the multi-level task decision-making method for unmanned boats combining a large model and reinforcement learning in this embodiment includes:

[0057] Step 1: Obtain the environmental observation values ​​of the unmanned vehicle in its current state.

[0058] In this embodiment, the environmental observation values ​​include basic environmental observation values ​​and target observation values. Optionally, the environmental observation values ​​of the unmanned boat in the current state can be obtained through tools such as an onboard camera.

[0059] Among them, the basic environment observation values ​​include geographic information and obstacle information, and the target observation values ​​include target category and target status.

[0060] For example, geographic information can be the current coordinates of the UAV, and obstacle information can be the coordinates, speed, and heading of the obstacle. Target status can be the target's location and navigation information. For example, target categories include {other ships, obstacles}, and target status includes {position, speed, heading, size, and shape}.

[0061] In this embodiment, the target is the task target in the multi-level task.

[0062] Step 2: The environmental observation value and the current state of the unmanned boat are used as the environmental state of the unmanned boat and input into the strategy model to obtain the execution strategy of the unmanned boat; among them, the strategy model is obtained based on the large language model and reinforcement learning training.

[0063] In this embodiment, the current state of the unmanned boat includes the state of the unmanned boat itself and the mission progress at the current moment.

[0064] For example, the self-status can be the position, navigation state, and payload state of the unmanned boat, that is, the latitude and longitude, speed, heading angle, and payload quantity. Mission progress can be the completion of the hierarchical target, the confidence level of the strike coordinates, and the threat assessment index.

[0065] In this embodiment, the execution strategy of the unmanned boat is the actions that the unmanned boat can perform, including: maneuver control, electronic countermeasures, and strike system.

[0066] For example, maneuver control can be heading adjustment and speed control; electronic countermeasures can be active interference and signal decryption; and the strike system can be payload launch.

[0067] Further, the training process of the policy model of this embodiment is described in detail. Figure 3 , Figure 3 This is a training process diagram of a strategy model provided by an embodiment of the present invention, such as Figure 3 As shown, the training process of the policy model of this embodiment may include the following steps:

[0068] S1: The interaction process between the unmanned boat and the environment is abstracted into a sequential decision problem of a Markov decision process, expressed as<S,A,τ,R,γ> , where S is the state space, A is the action space, τ is the transition probability function, R is the single-step reward obtained from the environment, and γ is the discount factor.

[0069] In this embodiment, the state space S represents the real-time interaction information between the UAV and the environment, including {environmental observations; own state; mission progress}, and the action space A represents the set of all actions that the UAV can perform, including {maneuver control; electronic countermeasures; strike system}. The transition probability function τ can be written as τ = S × A → P (S); the single-step reward R obtained from the environment can be written as R = S × A × S → R. The main goal is to learn an optimal policy π: S → P (A) that maximizes the expected cumulative reward over time, that is, max π E[∑ t γ t r t ]. Letting the parameters of the policy network be θ, a standard gradient-based reinforcement learning algorithm minimizes a surrogate loss function using gradient descent with respect to θ. This loss is estimated by sampling trajectories, where each trajectory consists of a sequence of tuples of state, action, and reward.

[0070] S2: Use the student model after policy initialization to collect policy experience data.

[0071] In this embodiment, the student model can be initialized in different ways according to whether the strategy is provided in advance. In this embodiment, S2 includes:

[0072] S21: If there is a pre-trained student model, the environmental state of the unmanned boat is input into the pre-trained student model to obtain the action and action reward of the unmanned boat, and the environmental state of the unmanned boat and its corresponding action and action reward are stored as strategy experience data in the experience buffer.

[0073] Specifically, if a pre-trained student model exists, then the policy will load the pre-trained student model from the path provided. The policy will then be trained over multiple iterations, with data collection, policy updates, and evaluations performed during each iteration.

[0074] It is understandable that the current timestamp can be obtained and the start time of the current strategy optimization process can be recorded as a benchmark for subsequent calculation of the time consumption of the optimization process.

[0075] S22: If there is no pre-trained student model, a policy neural network Actor module and a value neural network Critic module are constructed as the student model based on the proximal policy optimization algorithm, and the environmental state of the unmanned boat is input into the policy neural network Actor module and the value neural network Critic module to obtain the action and action reward of the unmanned boat. The environmental state, action, action reward and environmental state of the unmanned boat after executing the action are stored in the experience buffer as policy experience data.

[0076] Specifically, if there is no pre-trained student model, then create a new instance and use a proximal policy optimization algorithm (PPO algorithm) strategy. First, build the PPO policy neural network Actor module and the value neural network Critic module, and initialize the corresponding parameters of the two modules respectively. And ω, where the input of the policy neural network Actor module is the environment state, and the output probability distribution of the unmanned boat's action is used to select the unmanned boat's execution action. Its goal is to learn the optimal strategy under the current environment state; the value neural network Critic module also receives the environment state as input, and outputs the expected value of the cumulative reward of the current environment state, which is used to measure the quality of the environment state of the unmanned boat under the current environment state. The current environment state is input into the student model to output the action, and the selected action is executed and rewarded in the synaesthesia integrated data collection environment under interference, and transformed into the environment state at the next moment. The environment state, action, action reward and environment state after the action of the unmanned boat are packaged into a four-tuple (s t ,a t ,r t ,s t+1 ) is stored in the experience buffer, where s t Indicates the current environment state, a t is the currently selected action, r t ,To obtain the reward, s t+1 The above steps are repeated until the amount of experience data in the experience buffer reaches the preset threshold.

[0077] S3: Use the policy experience data to optimize the policy network of the student model to obtain a preliminarily trained student model.

[0078] In this embodiment, when there is a pre-trained student model, S3 includes:

[0079] S311: extracting a preset amount of experience data from the experience buffer;

[0080] S312: Calculating a first loss value of the extracted experience data according to a preset first loss function;

[0081] Among them, the first loss function is expressed as:

[0082] L total =βL policy +(1-β)L value ;

[0083] Among them, L policy =-E[log(π θ (a t |s t ))A t ]+λ(π θ (a t |s t )) a -1) 2 ;

[0084] L value =e[(V new (s t )-V old (s t )) 2 ];

[0085] Where, L total is the first loss function, L policy is the policy loss function, L value is the value loss function, β is the weight parameter, s t represents the environmental state of the unmanned boat at time t, a t Represents the environmental state s t The corresponding action, π θ (a t |s t ) indicates that in the environmental state s t Next, perform action a t The probability of a t is the advantage function, λ is the regularization parameter, α is a constant greater than 1, E represents the expectation, V new (s t ) represents the new value function for the environment state s t The estimated value of V old(s t ) represents the old value function for the environment state s t estimated value.

[0086] In this embodiment, the policy loss is used to optimize the policy network of the student model. The first term of the policy loss function is the standard policy loss, and the second term is unique to the proximal policy optimization algorithm and is used to limit the step size of the policy update. α is used to scale the target probability.

[0087] In this embodiment, the value loss function is used to optimize the value function, which measures the difference between the predicted return and the actual return, V new (s t ) That is, the value network (Critic network) currently being trained is concerned with the environment state s t The parameters of the value network are updated in real time during training to reflect the latest strategies and environment dynamics.

[0088] In this embodiment, β is a weight used to balance strategy loss and value loss.

[0089] S313: Optimize the strategy network of the pre-trained student model according to the first loss value to obtain a preliminarily trained student model.

[0090] Optionally, the parameters of the policy network can be updated according to the gradient of the loss function, which can be expressed as: Where α is the learning rate. The loss value and other related indicators in the optimization process are then calculated and returned, such as the entropy of the policy network. The entropy can be calculated using the following formula: H(π) = -E a～π [log(π(α))].

[0091] In this embodiment, when there is no pre-trained student model, S3 includes:

[0092] S321: extracting a preset amount of experience data from the experience buffer;

[0093] S322: Input the extracted experience data into the value neural network critic module for training, and update the network parameters of the value neural network critic module according to the temporal difference error;

[0094] S323: Optimize the policy network of the policy neural network Actor module according to the estimated advantage function obtained by the value neural network Critic module to obtain a preliminarily trained student model.

[0095] In this embodiment, a batch of experience data is randomly sampled from the experience buffer pool. The environment state, actions, and action rewards in the experience data are then input into the value neural network Critic module for training. Finally, the policy network is updated, and the environment state is input into the policy neural network Actor module. Simultaneously, the policy neural network Actor module is trained in conjunction with the estimated advantage function obtained in the value neural network Critic module to update the action selection strategy. Training stops when the maximum number of update cycles is reached. This cyclic training process helps improve anti-interference decision-making capabilities, allowing the system to gradually learn more optimized behavioral strategies.

[0096] Among them, the timing difference error is expressed as:

[0097]

[0098] The estimated advantage function is expressed as:

[0099]

[0100] The goal of policy network optimization is expressed as:

[0101]

[0102] Where, δ t represents the timing difference error, r t represents the reward parameter, which is used to measure the quality of the action taken by the unmanned boat at time t, γ represents the discount factor, and s t represents the environmental state of the unmanned boat at time t, s t+1 represents the environmental state of the unmanned boat at time t+1, In the strategy π θ The environment is in state s t+1 The value function of In the strategy π θ The environment is in state s t The value function of is the estimated advantage function of multi-step temporal difference estimation, k represents the number of steps backtracking in the calculation of the estimated advantage function, σ t+l represents the time series difference error at time t+l, β represents the trade-off parameter, represents the optimization goal of the policy network, is the network parameter of the policy neural network Actor module, clip is the truncation function, θ′ represents the network parameter of the value neural network Critic module, Indicates that the strategy parameters are The strategy ratio when ∈ represents the truncation parameter, Represents an estimate of the expected value.

[0103] In this embodiment, r t is a manually set hyperparameter, which is the relative probability ratio of the strategy, used to help control the update amplitude of the strategy, expressed as:

[0104] S4: Build and initialize the teacher model based on the large language model.

[0105] In this embodiment, S4 includes constructing a teacher model comprising a translator and a pre-trained large language model, and setting a set of policy options in the teacher model. The translator is used to generate a text description based on the environmental state of the UAV, and the pre-trained large language model is used to output the UAV's execution strategy based on the text description, the user's intended instructions, and the set of policy options.

[0106] In this embodiment, the large language model is learned through historical data of the unmanned boat and a knowledge base is constructed by combining expert knowledge and experience to form a pre-trained large language model.

[0107] Optionally, relevant models and model interfaces can be set up to generate a series of options using the description of observations and tasks, and the teacher model is responsible for providing high-level instructions to guide the operation.

[0108] Optionally, the user's intended instructions can be constraints and policy output requirements. In this embodiment, relevant prompt words can be preset in advance according to the target task, and the pre-trained large language model is used to constrain and guide the large language model to generate relevant strategies based on the standardized template of [environmental state] + [constraints] + [output requirements]. As the unmanned boat continues to explore, its environmental state and actions are constantly adjusted, and its environmental observations will also change. That is, according to the standardized template, a prompt with a text description c(s) is generated for the change of (s, a, r)∈D, where D represents the experience buffer.

[0109] It should be noted that when initializing the teacher model, the system prompt words are set by sending a request to the specified URL (Uniform Resource Locator). First, the appropriate prompt is input to the large language model, and a set of K strategy options related to the current task is input, which consists of a strategy option set π: {π k :S→P(A)}, where k∈{1, 2, ..., K} represents the option index. When the environment state s is presented, the large language model is requested to guide the next action, initially selecting a policy option π from the policy option set π k , prompted by the text description c(s) of the environment state s. Subsequently, an execution strategy α~π is generated based on the selected strategy option k (s), as guidance provided by the teacher model.

[0110] In this embodiment, when sending a query to the teacher model to obtain the execution strategy, the specific process is to send a request text with prompts to the URL of the large language model and parse the returned JSON response to extract the action plan. If the text contains new environmental observations, the large language model is queried multiple times to generate an action plan, and the generated behavior list and strategy probabilities are stored in the skill dictionary for later extraction as action guidance for the unmanned boat. The flow chart of the teacher model generating the execution strategy is shown below. Figure 4 shown.

[0111] It's no secret that large language models possess unique advantages in simulating human creative work, demonstrating their potential for assisting engineers in designing and developing control algorithms for unmanned systems. On the one hand, their ability to understand natural language descriptions and mimic human thinking and creative processes makes them extremely valuable for exploring new domains and designing innovative solutions. For example, they can be used to generate new control strategies, design novel sensor configurations, or propose creative path planning methods. On the other hand, addressing the poor interpretability of large language models, through the supervision and guidance of professional technicians, we can fully leverage their creative and generative capabilities, incorporating expert knowledge and experience into the generation process while ensuring that the resulting control algorithms are interpretable, controllable, and comply with relevant safety standards. This, in turn, enables safe and reliable application in the unmanned systems sector. Therefore, this combination will provide a new avenue for the development of autonomous interpretable algorithms, promoting innovation and progress in the field of unmanned systems.

[0112] S5: Use the teacher model to optimize the strategy network of the initially trained student model to obtain the trained student model, and use the trained student model as the strategy model.

[0113] In this embodiment, the process by which the student model learns a policy from the teacher model can be viewed as a process of policy transfer and the extraction of task-related knowledge from the teacher model. When the student model sends the environment state to the teacher model, the teacher model responds by providing guidance strategies, thereby acquiring the teacher model's strategy based on the large language model.

[0114] For example, the teacher model outputs an execution strategy based on the received task name, action space, environment observation value, and user intention instruction, which can be expressed as:

[0115] π T (·|s)=∑ k P rLLM (k|c(s))π k (·|s);

[0116] Among them, P rLLM(k|c(s)) represents the probability that the teacher model selects the kth policy option given the text description c(s) of the current environment state s, π k (·|s) denotes the policy associated with the kth policy option.

[0117] Optionally, S5 includes: inputting the environmental state of the unmanned boat into the teacher model and the preliminarily trained student model respectively, and optimizing the policy network of the preliminarily trained student model by minimizing a preset second loss function, where the second loss function is expressed as:

[0118]

[0119] Where, is the second loss function, It is the loss function used to encode environmental feedback in the reinforcement learning algorithm, that is, the strategy loss function L policy , represents the expectation of the environment state, H(π T (·|s)||π θ (·|s)) represents the regularization term for the difference between the execution strategies generated by the teacher model and the initially trained student model, s represents the environment state, represents the degradation parameter, θ represents the policy network parameter of the student model, π T (·|s) represents the execution strategy generated by the teacher model, π θ (·|s) represents the execution strategy generated by the student model.

[0120] In this embodiment, represents the traditional loss used in proximal policy optimization algorithms to encode feedback from the environment, which is usually designed to maximize the expected return or reward obtained. Represents the expectation of the environment state s, where the environment state s is distributed by the current policy π θ Generate. By introducing a regular term H(π T (·|s)||π θ (·|s)) incorporates the teacher model's guidance into the student model's learning process. This term captures the KL divergence between the student model's policy and the teacher model's policy. To control the student model's dependence on the teacher model, a degradation parameter λ is introduced. When λ is set to zero, the learning process degenerates to a standard reinforcement learning process, unaffected by the teacher model.

[0121] As training progresses, we gradually decay λ, allowing the student model to shift its attention to maximizing its expected return. By reducing the influence of the teacher model's guidance, the student model becomes more independent in its decision-making process and places more emphasis on its own learning strategy.

[0122] Preferably, the degradation system is designed as follows:

[0123]

[0124] Where i represents the index of training iteration, k represents the decay rate, λ0 is the initial value of λ, and λ c is a constant value less than λ0 and remains unchanged from iteration i1 to iteration i2, where i2 represents the point at which the connection with the teacher model based on the large language model is closed.

[0125] The multi-level task decision-making method for unmanned boats that combines a large model with reinforcement learning in an embodiment of the present invention uses a teacher-student method, which significantly improves the execution capability of unmanned boats in complex adversarial environments by integrating the adversarial rule guidance of a large language model and the dynamic decision-making capability of reinforcement learning. By generating explainable adversarial logs through a large language model, the decision-making process is made transparent, allowing real-time adjustment of strategy weights through natural language instructions to ensure the consistency of the unmanned boat's artificial intelligence decision-making and adversarial intentions. A dynamic adversarial learning framework is introduced to perceive changes in the adversary's defense strategy in real time, adjust the decision-making model online, predict the adversary's behavior pattern through a large language model, and quickly generate an adaptive execution strategy in combination with reinforcement learning, thereby significantly improving the unmanned boat's penetration success rate.

[0126] Furthermore, the specific implementation process of the unmanned boat multi-level task decision-making method combining a large model and reinforcement learning in this embodiment is explained through specific simulation experiments.

[0127] First, the environment is constructed as a multi-layer authority sea area infiltration, including minefields, communication stations and core command. The environment is explored with continuous actions (moving, scanning, interfering, unlocking), and natural language instructions need to be parsed and hierarchical tasks must be completed in sequence.

[0128] The mission space of the unmanned boat is divided into three layers: the first-level mission (peripheral breakthrough): paralyze the opponent's offshore radar station and avoid threats; the second-level mission (middle-level interference): intercept and decipher the secret messages of the communication ship and avoid the electromagnetic trap area; the third-level mission (inner-level strike): generate dynamic strike coordinates and destroy the underground command post in the tidal window. The mission diagram of the unmanned boat is as follows: Figure 2 As shown in Figure 2, the action space consists of {maneuver control; electronic countermeasures; strike system}. Maneuver control includes {course adjustment; speed control}, electronic countermeasures includes {active jamming; signal decryption}, and the strike system includes {payload launch}. The state space is a triplet of {environmental observations; own state; mission progress}.

[0129] The reward design includes core mission rewards and confrontation behavior rewards. In the core mission rewards, the unmanned boat is driven to advance the mission in the order of "paralyzing radar → deciphering communications → destroying the command post", reflecting the hierarchical dependency. The reward for paralyzing the radar station is +800, the reward for deciphering communications is +600, the reward for matching the strike window with the command post and destroying it is +5000, and the reward for entering a new mission area is +300. The confrontation behavior rewards include: maneuvering when not being scanned by radar +2 / second, being continuously scanned by radar -20 per second, after optical lock -50 per second, entering the minefield -30 per second, triggering a mine -500, repeating the same action sequence (such as turning left 15 degrees five times in a row) -100 each time, and confirming the remaining payload at the end of the mission, with +100 for each remaining payload.

[0130] Based on the multi-level tasks of the unmanned boat, the training example of the strategy model of this embodiment is shown in the figure Figure 5 The specific training process is as follows:

[0131] Step i: The Red Team’s UAV agent interacts with the environment and abstracts the interaction process into a sequential decision problem of a Markov decision process.<S,A,τ,R,γ> The main goal is to learn an optimal policy π: S→P(A) that maximizes the expected cumulative benefit over time: max π E[∑ t γ t r t ].

[0132] Step ii: Set the random seed to ensure the repeatability of the experiment. Setting the seed for the random number generation engine ensures that all random processes start from the same starting point each time the experiment is run, making the results more reliable and comparable.

[0133] Step 3: Load the task information, including the environment configuration details, state space, action space, and so on. First, print the name of the task being reset to understand the currently loaded environment. Then, open the pre-set JSON file containing all task information. Read the file and parse the relevant information for the current task, such as the environment configuration, description, and examples.

[0134] Step iv: Initialize the large language model and prepare to generate guidance strategies.

[0135] Specifically, first initialize the large language model instance and determine whether it is online. If it is online, initialize the large language model and set the relevant model and URL. Preset relevant prompt words according to the target task in advance, and use the prompt words to limit and guide the generation of the large language model. For example:

[0136] [Mission Phase]: Outer Breakthrough (Radar Station Suppression)

[0137] [Environmental situation]:

[0138] Time: 2024-08-15 03:15 (night)

[0139] Geographic location: 118.72°E, 24.45°N, sea state level 3

[0140] Blue side defense status: high radar scanning intensity (L band, cycle 12 seconds), dense mine density (8 mines per square kilometer in the A3 area)

[0141] Status: 2 mounts remaining, 68% energy remaining

[0142] [Commander's Order]: The highest priority goal is to paralyze the radar station, and the secondary goal is to preserve combat power.

[0143] [Output requirements]: 1. Path planning (heading sequence + speed strategy); 2. Strike plan

[0144] Then, initialize the large language model and set the system prompt by sending a request to the specified interface. First, input the appropriate prompt to the large language model, and input a set of K strategy options related to the current task, consisting of a strategy option set Π: {π k :S→P(A)}, where k∈{1, 2, ..., K} represents the option index. When presented with an environment state s, the large language model is requested to guide the next action, initially selecting a high-level option π from the policy option set π k , prompted by the text description c(s) of the environment state s. Subsequently, an execution strategy α~π is generated based on the selected strategy option k (s), as guidance provided by the large language model.

[0145] Next, a query is sent to the large language model to retrieve the action plan. Specifically, a request text with prompts is sent to the large language model's URL and the returned JSON response is parsed to extract the plan. If the text contains new observations, the large language model is queried multiple times to generate a plan. The generated action list and policy probabilities are stored in the skill dictionary for later retrieval as action guidance.

[0146] Step v: Create a teacher model and generate a teacher model policy. This model guides the UAV's behavior based on the current environment state and the action plan generated by the large language model, helping the UAV learn how to perform tasks in complex environments. This process gradually reduces the UAV's reliance on the teacher model policy and ultimately allows it to independently perform tasks in the environment. The teacher model initialization process is performed simultaneously with the initialization of the large language model, interacting with each other.

[0147] Specifically, a teacher model is created, which receives the task name, action space, and state space of the UAV. It also determines whether the network is currently offline. An instance is created, and the task name and prefix information are passed to it. The instance then receives the behavior list and policy probabilities from the skill dictionary. Based on the behavior list and the current environment state, an action is generated, such as freely exploring the outer layer or moving to a specific location.

[0148] Step vi: Initialize the student model with the policy, and use the initialized student model to collect policy experience data. For details, see step S2, which will not be described here.

[0149] It's important to note that the experience buffer must be initialized. This includes defining a buffer to accept the MDP discount factor (used to calculate the decay of future rewards), the GAE discount factor (used to adjust the advantage function estimate to reduce variance and improve learning stability), and a data storage device. Configuring the experience buffer primarily involves configuring specific internal data structures, such as initializing lists or tensors for storing environment states, actions, rewards, next environment states, and completion flags. These data structures will be used to store the agent's experience data later.

[0150] Step vii: Perform iterative training in the simulation environment, recording the training time for each iteration. In each iteration, the experience buffer is first cleared, and then sufficient training trajectory data is collected. If the amount of data in the experience buffer is insufficient for a valid training session, further data collection is continued until sufficient data is available for a training session. The collected data is then used to update the student model's policy. The student model learns its policy by minimizing a pre-set second loss function.

[0151] It is understandable that the current student model can be saved every few iterations and the performance of the student model can be evaluated, including running multiple trajectories and recording the average reward, trajectory length, and success rate. Specifically, first check whether a seed list for evaluation is provided. If not provided, a default seed list will be generated to ensure that each evaluation has a different random seed, thereby reducing the randomness of the evaluation results. Then initialize the list for storing the evaluation results, including the total reward, trajectory length, and success rate. Then run one or more trajectories and collect performance data of the student model to evaluate the performance of the student model. After completing the evaluation of all environment seeds, calculate the average reward, average trajectory length, and success rate, which can be used to analyze the performance of the student model, or as a basis for further training and adjustment, to help understand the performance of the student model on specific tasks, and provide guidance for future improvements.

[0152] For example, the pseudo code of the strategy learning algorithm of the student model of this embodiment is as follows:

[0153]

[0154]

[0155] The multi-level task decision-making method for unmanned vehicles that combines a large model with reinforcement learning in the embodiment of the present invention uses a teacher-student approach. By integrating the adversarial rule guidance of a large language model with the dynamic decision-making capabilities of reinforcement learning, it brings multiple beneficial effects and significantly improves the execution capability of unmanned vehicles in complex adversarial environments. Its core advantages are reflected in the following aspects:

[0156] In the task of breaking through the multi-layer defense chain, traditional reinforcement learning methods rely on a single task goal and static environment assumptions, making it difficult to cope with the high dynamics and task coupling of the adversarial environment. However, the present invention builds a knowledge base through a large language model, breaks down the task goals into multi-level subtasks (such as electronic suppression, signal decryption, dynamic strikes), and designs a hierarchical reward mechanism to enable the unmanned boat to gradually accumulate local successful experiences during exploration. This hierarchical decision-making logic not only greatly shortens the training convergence time, but also can more accurately allocate resources in practical applications (such as prioritizing the energy consumption of electronic warfare modules), avoiding ineffective actions caused by global goal ambiguity.

[0157] In the face of the strong dynamic characteristics of the adversarial environment, by introducing a dynamic adversarial learning framework, the decision model is adjusted online through real-time perception of changes in the adversary's defense strategy (such as radar band switching, encryption protocol upgrades). Traditional methods often have problems with strategy rigidity due to the lack of environmental evolution modeling capabilities - for example, when the adversary enables a new communication jamming technology, the unmanned boat still mechanically executes the preset jamming plan, but exposes its own position. The present invention analyzes historical data through a large language model, predicts the adversary's behavior pattern, and combines reinforcement learning to quickly generate adaptive strategies. For example, in the scenario of communication protocol mutation, the decoding logic can be reconstructed based on semantic understanding, rather than relying on a fixed decryption process, thereby significantly improving the success rate of penetration.

[0158] In terms of compliance and trusted decision-making, the traditional black-box decision-making model is difficult to meet the needs of ethical review and confrontation coordination. The present invention generates explainable confrontation logs through a large language model (such as "detour path A was selected because the minefield density threshold exceeded the limit"), making the decision-making process transparent and facilitating the intervention and verification of commanders. At the same time, the solution supports the "human-in-the-loop" collaborative mode, allowing commanders to adjust the strategy weights (such as "prioritize concealment") in real time through natural language instructions to ensure the consistency of artificial intelligence decisions and confrontation intentions. This design retains the efficiency advantage of autonomous decision-making while avoiding ethical risks (such as accidental injury to civilian targets).

[0159] The technical framework of this invention also increases generalizability and has cross-scenario transfer capabilities. By adjusting the knowledge base of the large language model and the environment parameters of reinforcement learning, it can be quickly adapted to other diverse tasks, providing a scalable technical foundation for the future. Compared with traditional methods, this invention achieves comprehensive breakthroughs in dynamic adaptability, task collaboration, and decision explainability.

[0160] It should be noted that, in this document, relational terms such as first and second are used solely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Furthermore, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that an article or device comprising a list of elements includes not only those elements but also other elements not explicitly listed. Without further limitation, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the article or device comprising the element. Terms such as "connected" or "connected" are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. References to orientations or positional relationships, such as "upper," "lower," "left," and "right," are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate description and simplify the description of the present invention. They do not indicate or imply that the device or element referred to must have, be constructed, or operate in a specific orientation, and are therefore not to be construed as limiting the present invention.

[0161] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0162] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A multi-level task decision-making method for unmanned boats combining large models and reinforcement learning, characterized by: include: Step 1: Obtain the environmental observation value of the unmanned boat in its current state; Step 2: The environmental observation value and the current state of the unmanned boat are used as the environmental state of the unmanned boat, and input into the strategy model to obtain the execution strategy of the unmanned boat; wherein, the strategy model is obtained based on a large language model and reinforcement learning training.

2. The unmanned boat multi-level task decision-making method combining a large model and reinforcement learning according to claim 1 is characterized in that: The environmental observation values ​​include basic environmental observation values ​​and target observation values.

3. The unmanned boat multi-level task decision-making method combining large models and reinforcement learning according to claim 1 is characterized in that: The training process of the policy model includes: S1: The interaction process between the unmanned boat and the environment is abstracted into a sequential decision problem of a Markov decision process, expressed as<S,A,τ,R,γ> , where S is the state space, A is the action space, τ is the transition probability function, R is the single-step reward obtained from the environment, and γ is the discount factor; S2: Collect policy experience data using the student model after policy initialization; S3: Optimizing the strategy network of the student model using the strategy experience data to obtain a preliminarily trained student model; S4: Build and initialize a teacher model based on a large language model; S5: Utilize the teacher model to perform strategy network optimization on the preliminarily trained student model to obtain a trained student model, and use the trained student model as the strategy model.

4. The unmanned vehicle multi-level task decision-making method combining a large model and reinforcement learning according to claim 3 is characterized in that: The S2 includes: S21: If there is a pre-trained student model, input the environmental state of the unmanned vehicle into the pre-trained student model to obtain the action and action reward of the unmanned vehicle, and store the environmental state of the unmanned vehicle and its corresponding action and action reward as the strategy experience data in the experience buffer; S22: If there is no pre-trained student model, a policy neural network Actor module and a value neural network Critic module are constructed as the student model based on the proximal policy optimization algorithm, the environmental state of the unmanned boat is input into the policy neural network Actor module and the value neural network Critic module, the action and action reward of the unmanned boat are obtained, and the environmental state, action, action reward and environmental state after the action is executed of the unmanned boat are stored in the experience buffer as the policy experience data.

5. The unmanned vehicle multi-level task decision-making method combining a large model and reinforcement learning according to claim 4 is characterized in that: When there is a pre-trained student model, S3 includes: S311: extracting a preset amount of experience data from the experience buffer; S312: Calculating a first loss value of the extracted experience data according to a preset first loss function; S313: Performing strategy network optimization on the pre-trained student model according to the first loss value to obtain the preliminarily trained student model.

6. The unmanned vehicle multi-level task decision-making method combining a large model and reinforcement learning according to claim 5 is characterized in that: The first loss function is expressed as: L total =βL policy +(1-β)L value ; Among them, L policy =-E[log(π θ (a t |s t ))A t ]+λ(π θ (a t |s t )) α -1) 2 ; L value =E[(V new (s t )-V old (s t )) 2 ]; Where, L total is the first loss function, L policy is the policy loss function, L value is the value loss function, β is the weight parameter, s t represents the environmental state of the unmanned boat at time t, a t Represents the environmental state s t The corresponding action, π θ (a t |s t ) indicates that in the environmental state s t Next, perform action a t The probability of A t is the advantage function, λ is the regularization parameter, α is a constant greater than 1, E represents the expectation, V new (s t ) represents the new value function for the environment state s t The estimated value of V old (s t ) represents the old value function for the environment state s t estimated value.

7. The unmanned vehicle multi-level task decision-making method combining a large model and reinforcement learning according to claim 4 is characterized in that: When there is no pre-trained student model, S3 includes: S321: extracting a preset amount of experience data from the experience buffer; S322: Input the extracted experience data into the value neural network critic module for training, and update the network parameters of the value neural network critic module according to the temporal difference error; S323: Optimizing the policy network of the policy neural network Actor module according to the estimated advantage function obtained by the value neural network Critic module to obtain the preliminarily trained student model.

8. The unmanned boat multi-level task decision-making method combining large models and reinforcement learning according to claim 7 is characterized in that: The timing differential error is expressed as: The estimated advantage function is expressed as: The objective of the policy network optimization is expressed as: Where, δ t represents the timing difference error, r t represents the reward parameter, γ represents the discount factor, s t represents the environmental state of the unmanned boat at time t, s t+1 represents the environmental state of the unmanned boat at time t+1, In the strategy π θ The environment is in state s t+1 The value function of In the strategy π θ The environment is in state s t The value function of To estimate the advantage function, k represents the number of steps backtracking in the calculation of the estimated advantage function, σ t+l represents the time series difference error at time t+l, β represents the trade-off parameter, represents the optimization goal of the policy network, is the network parameter of the policy neural network Actor module, clip is the truncation function, θ′ represents the network parameter of the value neural network Critic module, Indicates that the policy parameters are The strategy ratio when ∈ represents the truncation parameter, Represents an estimate of expectation.

9. The unmanned vehicle multi-level task decision-making method combining a large model and reinforcement learning according to claim 4 is characterized in that: The S4 includes: Building a teacher model including a translator and a pre-trained large language model, and setting a set of strategy options in the teacher model; The translator is used to generate a text description according to the environmental status of the unmanned boat, and the pre-trained large language model is used to output the execution strategy of the unmanned boat according to the text description, user intention instructions and the strategy option set.

10. The unmanned boat multi-level task decision-making method combining large models and reinforcement learning according to claim 9 is characterized in that: The S5 includes: The environmental state of the unmanned boat is input into the teacher model and the preliminarily trained student model respectively, and the policy network optimization of the preliminarily trained student model is achieved by minimizing a preset second loss function, where the second loss function is expressed as: Where, is the second loss function, is the loss function used to encode environmental feedback in reinforcement learning algorithms, represents the expectation of the environment state, H(π T (·|s)||π θ (·|s)) represents the regularization term for the difference between the execution strategies generated by the teacher model and the initially trained student model, s represents the environment state, represents the degradation parameter, θ represents the policy network parameter of the student model, π T (·|S) represents the execution strategy generated by the teacher model, π θ (·|s) represents the execution strategy generated by the student model.

Images