Reinforcement learning decision-making method fusing multi-granularity semantic prompt of large language model

Abstract

The invention belongs to the technical field of artificial intelligence and machine learning, and particularly relates to a reinforcement learning decision-making method fusing large language model multi-granularity semantic prompts, which comprises the following steps of: extracting feature vectors from an environment state, calling a large model to generate macroscopic, mesoscopic and microscopic prompts and uniformly coding; in the interaction process, prompt update is adaptively triggered according to a fixed period, state mutation or abnormal behaviors, strategy features are obtained through cross attention fusion states and prompts during decision making, strategy candidate actions and prompt candidate actions are combined, a final action is selected according to the prompt confidence coefficient and the fusion probability, and a decision is made. A composite reward is constructed according to task completion, exploration, prompt response, behavior dispersion and penalty terms, and a strategy network is updated through multi-step data iteration. The real-time performance and the matching degree of prompt guidance can be improved in a high-dimensional sparse reward and dynamic environment, convergence is accelerated, the exploration efficiency and the generalization stability are improved, and the strategy drift risk is reduced.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and machine learning technology, and specifically relates to a reinforcement learning decision-making method that integrates multi-granular semantic prompts from a large language model. Background Technology

[0002] Reinforcement learning (RL), an important branch of artificial intelligence, continuously optimizes decision-making strategies through the interaction between intelligent agents and their environment. It has been widely applied in complex tasks such as robot control, path planning, resource scheduling, and real-time strategic games. In these tasks, agents typically need to make long-term decisions in environments that are not fully known or are dynamically changing to maximize cumulative rewards. However, when the state space is high-dimensional, reward signals are sparse, and the environment changes frequently, traditional reinforcement learning methods often face problems such as slow training convergence speed, low exploration efficiency, poor behavioral stability, and insufficient policy generalization ability, making it difficult to meet the efficiency and reliability requirements of practical engineering scenarios.

[0003] In recent years, Large Language Models (LLMs), with their advantages in semantic understanding, knowledge representation, and reasoning, have been increasingly incorporated into reinforcement learning decision-making processes. They are used to generate task planning, strategic cues, and action guidance, providing high-level semantic information and domain knowledge from the outside for policy learning. By providing textual cues to the agent during the training or inference phases, it is hoped that the agent's exploration direction can be improved, policy convergence accelerated, and the understanding and adaptability to complex tasks enhanced.

[0004] However, existing RL-LLM (a joint framework of all reinforcement learning agents (RL) and large language model (LLM)) fusion methods still have several limitations. On the one hand, the prompt structure mostly remains at two layers: "macro-policy" and "micro-action." One type of prompt focuses on high-level task objectives and global policies, while the other directly provides specific action suggestions, lacking a meso-level semantic planning layer between the two. For example, there is a lack of explicit modeling and expression of subtask decomposition, stage intentions, and key intermediate states, resulting in insufficient granularity of prompts and difficulty in achieving effective connection and coordination between global strategy and local operations.

[0005] On the other hand, existing methods often use fixed periods or preset trigger conditions for updating prompts, which usually cannot respond and adjust in a timely manner to sudden changes in environmental conditions or abnormal deviations in agent behavior. When the environment is highly dynamic or task phases switch frequently, the prompt content is prone to lag or mismatch with the environmental state and policy requirements, reducing the prompt's ability to guide the decision-making process in real time.

[0006] Furthermore, existing solutions for integrating prompts with policy networks often employ simple feature concatenation, chaining, or directly introducing text encoding results as additional inputs into the policy network, lacking deep alignment and interaction modeling mechanisms based on state features. The correlation between prompt semantics and environmental states and historical trajectories is not fully explored, making it difficult for prompts to differentiate and selectively function under different states, thus limiting the effective guidance of semantic information for policy optimization.

[0007] In complex task scenarios involving high-dimensional state spaces, sparse rewards, and dynamically changing environments, the aforementioned problems are further compounded, making it difficult for existing RL-LLM fusion methods to simultaneously balance convergence speed, exploration efficiency, and policy generalization ability. This can easily lead to path dependence, excessive reliance on initial cues or early experience, policy drift in the later stages of training, and significant instability in agent behavior across different environmental instances or task variants.

[0008] Therefore, there is a need for a reinforcement learning decision-making method that can systematically improve the granular structure design, generation and update mechanism, and integration method with policy networks for complex dynamic environments. This method should enable multi-granularity prompt collaboration, dynamic prompt adaptive update based on state changes, and deep alignment and interactive modeling of prompt semantics and policy features. This would improve policy convergence speed, enhance task adaptability and generalization ability, and improve the stability and interpretability of agent behavior in complex tasks. Summary of the Invention

[0009] To address the technical problems of slow convergence, unstable behavior, and poor generalization caused by the single structure of the prompts, insufficient dynamic update mechanism, and inadequate fusion of semantics and policies in existing technologies, this application provides a reinforcement learning decision-making method that integrates multi-granular semantic prompts from a large language model. By introducing adaptively updatable macro, meso, and micro multi-granular semantic prompts and deeply aligning and integrating them with policy features, and combining them with dynamic confidence trade-offs and composite reward constraints, reinforcement learning can be guided more effectively in real time in high-dimensional sparse reward dynamic environments. This accelerates convergence, improves exploration efficiency and generalization stability, and reduces the risks of policy drift and path dependence.

[0010] This application provides a reinforcement learning decision-making method that integrates multi-granularity semantic prompts from a large language model, the method comprising:

[0011] Step 1: Obtain environmental state information, extract features from the environmental state information, and obtain the state feature vector at the current moment;

[0012] Step 2: Based on the state feature vector, call the large language model to generate multi-granularity prompt information of macro-level prompts, meso-level prompts and micro-level prompts, and uniformly encode each granularity prompt to obtain the corresponding multi-granularity prompt vector;

[0013] Step 3: During the continuous interaction between the agent and the environment, based on the preset fixed period detection conditions, state change detection conditions and abnormal behavior detection conditions, determine whether the multi-granularity prompt needs to be updated. When any detection condition is met, the prompt is updated and Step 2 is re-executed.

[0014] Step 4: After completing the multi-granularity cue vector encoding, at each decision time, the current state feature vector is fused with the embedding vectors of macro-cues, meso-cues and micro-cues through a cross-attention mechanism to obtain a policy fusion feature adapted to the current environmental state.

[0015] Step 5: Input the fused features into the policy network to obtain the action probability distribution and determine the policy network candidate actions. At the same time, determine the prompt candidate actions based on the current multi-granularity prompts. Based on the prompt confidence and fusion probability, select the final action to be executed between the prompt candidate actions and the policy network candidate actions.

[0016] Step 6: Based on the feedback generated in the environment by the final action, calculate the task completion reward, exploration reward, cue response reward, behavior distraction reward, and behavior penalty, and combine the rewards to obtain the composite reward function at the current moment;

[0017] Step 7: Based on the state feature vectors at multiple time steps, the corresponding final execution actions, the composite reward value, and the state feature vector at the next time step, iteratively update the parameters of the policy network so that the policy network can complete the optimization under the guidance of the composite reward.

[0018] In a preferred implementation, step 1 further includes:

[0019] Step 1.1: During the interaction between the intelligent agent and the external environment, environmental status data from the visual sensor, position sensor, task management module and environmental interface are collected in real time;

[0020] Step 1.2: Perform noise reduction, normalization, and data format unification on the collected raw environmental data;

[0021] Step 1.3: Input the preprocessed environmental state data into the feature encoding module to extract the local spatial features, temporal features and global context features of the environmental state data. Then, fuse the multimodal features output by the feature encoding module to obtain a set of low-dimensional dense state feature vectors.

[0022] Low-dimensional dense state feature vector for:

[0023]

[0024] In the formula: Represents a state-coded network; The state representation vector.

[0025] In a preferred implementation, step 2 further includes:

[0026] Step 2.1: Convert the current state feature vector obtained in Step 1 into a single vector. Summary of Historical Trajectory By concatenating the data, a context state representation at time t is formed. The state representation is then input into a large language model, which dynamically generates multi-granular language prompts given the context.

[0027] Step 2.2: Perform semantic encoding on the macro-level, meso-level, and micro-level cues obtained in Step 2.1, mapping each granularity of cues to a unified vector space to obtain macro-level cues vector, meso-level cues vector, and micro-level cues vector.

[0028] In the preferred implementation, further, in step 2.1, the three granularity prompts output by the large language model at time t constitute a prompt set. :

[0029]

[0030] In the formula: , These indicate macroscopic, mesoscopic, and microscopic granularity levels, respectively.

[0031] In step 2.2, the prompt text for each layer is... Input semantic encoder The resulting embedding vector is:

[0032]

[0033] In the formula: Represents the time t at time t. The embedding vector of the layer hint; This indicates the set of granularity for the prompt; This represents the environmental state feature vector of the system at time step t; This represents a summary of trajectory information within the k steps from time tk to t; This represents the semantic fusion function of the language model at time step t; This means mapping the natural language output of a large language model to a compact, low-dimensional vector.

[0034] In a preferred implementation, step 3 further includes:

[0035] Step 3.1: At each time step, perform fixed period detection, state change detection and abnormal behavior detection for the current time step. When any one of the fixed period condition, state change condition or abnormal behavior condition is met, generate a prompt update trigger signal.

[0036] Step 3.2: When the prompt update trigger signal exists, the context state representation at time t is reconstructed based on the current state feature vector and historical trajectory summary at that time step. The reconstructed context state representation is input into the large language model to obtain new macro prompts, new meso prompts, and new micro prompts. A new multi-granularity prompt set is formed from the new macro prompts, new meso prompts, and new micro prompts.

[0037] In the preferred implementation, further, in step 3.1, the condition for triggering the update of the notification during fixed-period detection is:

[0038]

[0039] In the formula: t represents the time step index corresponding to the current decision moment; This indicates the preset fixed period length parameter; This represents the modulo operator; This indicates a periodic trigger condition if and only if t can be triggered. When the integer division occurs, the fixed-period scheduling unit triggers an update prompt.

[0040] The conditions for triggering the state mutation detection update are:

[0041]

[0042] In the formula: Indicates the number of variables at the current moment; Indicates the number of variables at the previous time step; The preset threshold for change;

[0043] The conditions for triggering the update of abnormal behavior detection prompts are:

[0044]

[0045] In the formula: t represents the time step index corresponding to the current decision moment; Indicates the length of the trajectory window used for abnormal behavior detection; This represents the position information of the agent at the i-th time step; This represents the displacement distance of the agent between two adjacent time steps i-1 and i; Indicates that recently The average distance traveled by the agent within each time step; This represents the minimum movement threshold.

[0046] In the preferred implementation, further, in step 3.2, the granularity hint set... for:

[0047]

[0048] In the formula: Indicates the target area or location coordinates; Indicates the direction of movement; Indicates the execution step size.

[0049] In a preferred implementation, step 4 further includes:

[0050] Step 4.1: Embed the macro-level hints generated in Step 2 Mesoscopic hints embedded and micro-cue embedding Concatenate them in order, align them within the same matrix, and you will get the hint embedding matrix;

[0051] Hint: Embedded Matrix for:

[0052]

[0053] In the formula: t represents the decision moment or time step, corresponding to the t-th observation and decision in the reinforcement learning interaction process; This indicates that a splicing operation is performed in the row direction; This represents the embedding vector of the macroscopic feedback at time t; This represents the embedding vector of the mesoscopic suggestion at time t; This represents the embedding vector of the micro-cues at time t; , , These respectively indicate the macroscopic, mesoscopic, and microscopic granularity categories; This indicates the hint embedding matrix, where 3 corresponds to three types of hints: macroscopic, mesoscopic, and microscopic, each occupying one row. Indicates the dimension of a single prompt embedding;

[0054] Step 4.2: Using the current state feature vector as query vector The prompt will be embedded in the matrix. As key vector K and value vector V, fused features are calculated through a cross-attention mechanism;

[0055]

[0056] In the formula: t represents the time step index; This represents the state feature vector at time t; The hint embedding matrix represents time t; Representation matrix transpose; This represents the unified feature dimension of the query vector and the key vector; This represents the scaled dot product similarity score between the query vector and the three suggestion key vectors; This represents an exponential mapping function that is normalized by row.

[0057] In the preferred implementation, further, in step 5, the final action selection rule... for:

[0058]

[0059] In the formula: This represents the final action performed at time t; This indicates the candidate actions for the prompt obtained by mapping the current prompt information; This represents the candidate actions given by the reinforcement learning policy network; This indicates the confidence level of the current message; This represents a random number that is uniformly distributed in the interval (0,1); This represents the fusion probability parameter; Indicates when the condition If the conditions are not met simultaneously, the system selects candidate actions from the policy network. As the final action.

[0060] In the preferred implementation, further, in step 6, the composite reward function at the current time... for:

[0061]

[0062]

[0063] In the formula: This indicates the reward items for completing the task; Indicates exploration rewards; This indicates a reward for responding to the prompt; This indicates that the reward for the behavior is distributed. Indicates a penalty item; This indicates the number of resources successfully collected in the current round or statistical period; This indicates the weighting coefficient for the task completion reward; This represents the set of states that are accessed for the first time within the current statistical period; The cardinality of the new state set upon first visit; Indicates the exploration reward coefficient; This indicates the number of times the agent's actions are consistent with the suggested prompts within the current statistical period; This indicates the reward weighting coefficient for the prompt response; , This represents the spatial coordinates of the agent at the i-th time step; , Represents the spatial coordinates of the agent at time step j; n represents the total number of time steps within the current statistical window; Indicates the distributed reward coefficient for the behavior; This indicates the number of actions that did not produce valid output within a certain time period; This represents the penalty coefficient.

[0064] The beneficial effects of this application are:

[0065] First, the reinforcement learning decision-making method of this application, which integrates multi-granularity semantic prompts based on a large language model, addresses the problems of existing RL-LLM fusion methods, such as single prompt granularity structure, lagging prompt updates, insufficient alignment and interaction between prompt semantics and policy features, and slow convergence, inefficient exploration, and policy drift in high-dimensional sparse reward dynamic environments. This method introduces macro-, meso-, and micro-level semantic prompts with unified encoding based on environmental state feature extraction, and achieves adaptive prompt updates through a multi-condition triggering mechanism involving fixed periods, state mutations, and abnormal behaviors. At each decision moment, cross-attention is used to selectively fuse state features with multi-granularity prompts, and during the action selection phase, prompt confidence and policy probability are comprehensively considered to dynamically balance prompt actions and policy actions. Simultaneously, a composite reward consisting of task completion, exploration, prompt response, behavioral dispersion, and penalty terms is constructed to jointly constrain policy learning. This effectively improves the real-time performance and matching degree of prompt guidance in complex dynamic scenarios, enhances the synergy between global strategy and local operations, promotes policy convergence and exploration efficiency, improves policy generalization ability and behavioral stability, and reduces path dependence and policy drift risks.

[0066] Second, in the preferred implementation, this application collects multi-source environmental state data from visual sensors, position sensors, task management modules, and environmental interfaces in real time in step 1, and sequentially performs denoising, normalization, and format unification processing. Then, it inputs the data into the feature encoding module to simultaneously extract local spatial features, temporal dynamic features, and global contextual features and fuse them. This can compress high-dimensional, heterogeneous, and noisy original observations into low-dimensional dense state feature vectors. This reduces state representation redundancy and training computation overhead while ensuring the integrity of key decision information, improves the robustness and consistency of state representation, and weakens the interference of sensor noise and data scale differences on policy learning. In turn, it provides a more stable and generalizable input representation for subsequent large language model prompt generation, cross-attention fusion, and policy network decision-making, promoting overall training convergence and decision stability improvement.

[0067] Third, in the preferred implementation, this application generates macro, medium and micro three-layer prompts by combining the current state features and historical trajectory summaries into a context input LLM, and aligns the semantic encoding to the same vector space, thereby improving the completeness, stability and stage adaptability of the prompts in a continuous temporal context, and enhancing the interpretable semantic constraints of the strategy.

[0068] Fourth, in the preferred implementation, this application sets three types of triggering conditions—period, state change, and abnormal behavior—in step 3 to adaptively update the prompts. This maintains stability and low overhead when things are stable, and regenerates them in a timely manner when things change or become abnormal to avoid semantic lag and correct policy deviations, so that the prompts are continuously synchronized with environmental dynamics and behavioral performance.

[0069] Fifth, in the preferred implementation, this application performs cross-attention fusion of the multi-granularity cue embedding matrix and the state query vector in step 4, and dynamically allocates weights according to similarity to achieve selective absorption and alignment, thereby reducing cue redundancy and noise amplification, avoiding misleading and strengthening the cue-state coupling expression.

[0070] Sixth, in the preferred implementation, this application introduces an action selection mechanism based on cue confidence and fusion probability parameters in step 5: when the cue information is credible and meets the random sampling conditions, the cue candidate action is executed first; otherwise, the policy network candidate action is executed back. On the one hand, high-confidence cue provides directional guidance in the early training stage or reward sparsity stage, reducing blind exploration and accelerating the accumulation of effective experience; on the other hand, when the cue may be mismatched or uncertain, the back-off mechanism avoids the continuous misleading of the policy by erroneous cue, reducing the risk of path dependence and policy drift.

[0071] Seventh, in the preferred implementation, this application constructs a composite reward system for task completion, exploration, prompt response, behavior dispersion, and penalty items through step 6 and weights and regulates it. This not only improves the accumulation of rewards and state coverage under sparse rewards, but also strengthens the consistent execution of high-confidence prompts and punishes invalid actions, thereby reducing stagnation and repetition, and improving convergence and behavior stability. Attached Figure Description

[0072] Figure 1 This is a schematic diagram of the overall process of a reinforcement learning decision-making method that integrates a large language model with multi-granular semantic prompts according to an embodiment of the present invention.

[0073] Figure 2 This is a schematic diagram of the workflow of the environmental state perception module in step 1 of an embodiment of the present invention.

[0074] Figure 3 This is a schematic diagram of the workflow of the prompt generation module in step 2 of an embodiment of the present invention;

[0075] Figure 4 This is a schematic diagram illustrating the experimental results obtained using the method of this invention in one embodiment of the present invention, taking the StarCraft II resource gathering task CollectMineralShards as an application scenario. Detailed Implementation

[0076] To enable those skilled in the art to better understand the technical solutions of this application, the following will provide a more detailed description of this application in conjunction with the accompanying drawings and embodiments.

[0077] The directional terms such as above, below, left, right, front, and back used in this application are based on the positional relationships shown in the attached drawings. Different attached drawings may result in different positional relationships, therefore they should not be interpreted as limitations on the scope of protection.

[0078] In this application, the terms "installation," "connection," "interlocking," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, or a connection that allows communication between components. They can also refer to a direct connection or an indirect connection through an intermediate medium. They can refer to the internal connection of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0079] As per the instruction manual Figure 1 The present invention provides a reinforcement learning decision-making method that integrates multi-granularity semantic prompts from a large language model, the method comprising:

[0080] Step 1: Obtain environmental state information, extract features from the environmental state information, and obtain the state feature vector at the current moment.

[0081] Step 1 includes:

[0082] Step 1.1: During the interaction between the intelligent agent and the external environment, environmental status data from the visual sensor, position sensor, task management module and environmental interface are collected in real time.

[0083] Step 1.1 The environmental state perception module uses a state information acquisition unit to synchronously and continuously acquire and cache environmental state data from the vision sensor, position and attitude sensor, task management module, and environmental interface in real time. Specifically, the state information acquisition unit triggers an acquisition cycle at each interaction time step, subscribes to the output of each data source through a bus or communication interface, adds a unified format timestamp to the raw data from different sources, and aligns and sorts the data according to the timestamps to construct a temporal state data cache queue of length L, which supports subsequent temporal modeling and decision reasoning. The vision sensor acquires image frames or depth maps of the current scene; the position and attitude sensor acquires the two-dimensional and three-dimensional coordinates, velocity, and orientation information of the agent and key targets in the environment; the task management module outputs task management information such as the current task stage identifier, sub-task number, and constraints; and the environmental interface returns structured data such as resource quantity, friendly and enemy unit attributes, and environmental rule parameters.

[0084] In typical complex task scenarios, such as real-time strategy game environments like StarCraft II, the raw data collected in step 1.1 may specifically include: local image observations obtained by visual sensors (e.g., unit spatial distribution, terrain information, resource point locations and occupation status), global task parameters provided by the task management module (e.g., remaining time in the level, task stage objectives, task reward factors, failure penalty factors, etc.), and historical behavior trajectory information (including past operation instruction sequences, skill release records, unit scheduling commands and their corresponding environmental feedback results) jointly formed by the environment interface and the behavior execution module. These multimodal and heterogeneous data from different sources are uniformly organized into raw environmental state inputs at time step t, used to describe the agent's observable information about the environment and task context at the current moment. Combined with a historical state sequence of length L, this provides a complete state information foundation for subsequent modules to perform temporal dependency modeling, strategy optimization, and prompt generation.

[0085] Step 1.2: Perform noise reduction, normalization, and data format unification on the collected raw environmental data.

[0086] In step 1.2, the multi-source raw environmental state data obtained in step 1.1 is systematically preprocessed by the state preprocessing unit. The heterogeneous data from different sensors and functional modules are mapped to a consistent state representation space and converted into a dimension-compatible tensor form, providing standardized input for subsequent feature extraction, state fusion and policy learning.

[0087] Specifically, the state preprocessing unit first performs denoising operations on the various types of raw data acquired in step 1.1 to remove sensor noise and logically inconsistent redundant information. Geometric distortion correction, filtering denoising (such as Gaussian filtering, median filtering, etc.), and brightness / contrast cropping are performed on image frames to suppress imaging noise and compression artifacts. Combining environmental structural constraints and rule information, logical filtering or interpolation correction is applied to observation data that clearly violate physical constraints or environmental rules (such as unreasonable coordinate jumps, negative resource values, illegal state identifiers, etc.). For continuously changing numerical features over time, sliding window smoothing or exponentially weighted moving averages are used to smooth high-frequency jitter and occasional outliers, thereby improving the stability and interpretability of the state sequence in the time dimension.

[0088] Furthermore, based on denoising, features of different dimensions and orders of magnitude are normalized to eliminate the impact of scale differences on subsequent model training and inference. Geometric features such as spatial coordinates and size information are linearly scaled according to the environment boundary or map size to fall within the [0,1] or [-1,1] interval. Scalar features with large differences in order of magnitude, such as resource quantity, reward value, and loss value, are normalized using methods such as logarithmic compression, z-score standardization, or interval scaling. For control quantities or statistics that vary over time and have a wide numerical range, piecewise normalization or adaptive distribution normalization can be used to improve the consistency of features in different scenarios.

[0089] After denoising and normalization, the multimodal data undergoes structural and format unification to meet the input dimension and data type requirements of subsequent network structures. Image data is uniformly adjusted to a preset fixed resolution and number of channels (e.g., uniformly in H×W×C tensor form), and cropped or padded as necessary to align spatial dimensions. Structured features such as task management parameters, global environment parameters, and unit attributes are homogenized, and missing fields are filled in using default values ​​or masking mechanisms, and rearranged into fixed-length feature vectors. Discrete symbolic features (such as unit type, faction identifier, task stage identifier, action category, etc.) are converted into numerical representations that can be input into the model along with continuous features using one-hot encoding or embedding vector mapping. The processed visual features, spatial features, task features, and environmental parameters are concatenated or grouped according to predefined dimensions and uniformly encapsulated into a well-structured, dimensionally compatible multi-channel tensor representation.

[0090] Through the above-mentioned denoising, normalization and format unification preprocessing, step 1.2 normalizes the multi-source heterogeneous state data from visual sensors, position and attitude sensors, task management modules and environmental interfaces into a unified state representation space, and organizes it into a tensor format that adapts to the subsequent network structure, providing a stable, clean and structurally consistent state information foundation for subsequent modules to perform temporal dependency modeling, strategy optimization and prompt generation.

[0091] Step 1.3: Input the preprocessed environmental state data into the feature encoding module to extract the local spatial features, temporal features and global context features of the environmental state data. Then, fuse the multimodal features output by the feature encoding module to obtain a set of low-dimensional dense state feature vectors.

[0092] In step 1.3, the feature encoding module performs modal encoding and unified fusion on the normalized environmental state data output in step 1.2. The aim is to simultaneously extract the local spatial features, time series features, and global context features of the environmental state and map them to a unified state representation space, providing a unified input for the subsequent multi-granularity semantic prompt generation module and policy decision module.

[0093] In one embodiment, the feature encoding module may include several convolutional neural network layers and optional recurrent neural network layers. Specifically, for image-type inputs (such as scene image frames, local observation maps, etc.), a multi-layer convolutional structure based on 3×3 convolutional kernels is adopted, and the number of convolutional channels can be set to 32, 64, and 128 respectively. Each convolutional layer is followed by a non-linear activation function and downsampling (such as stride convolution or pooling) operations to progressively extract local texture features and global scene layout features at different scales, and to gradually compress them in the spatial dimension to form a compact high-dimensional visual feature representation. For structured vector inputs such as location coordinates, resource quantity, task labels, and unit attributes, a multi-layer fully connected network or a one-dimensional convolutional network can be used for non-linear mapping. Through several linear transformations with activation functions, the original low-dimensional or medium-dimensional features are mapped to high-dimensional abstract feature representations to enhance the model's ability to express complex relationships (such as the constraint relationship between resources, units, and tasks). For a time series consisting of L consecutive frames of states, after completing the spatial or structured encoding described above, the feature sequences of each time step can be input into a gated recurrent unit (GRU) or a long short-term memory network (LSTM). By modeling the dependencies between historical states through a recurrent neural network, long-term and short-term dynamic changes across time steps are captured, thereby forming a high-dimensional hidden state representation containing historical information at the current time t.

[0094] The visual features, structured features, and time-series hidden states output from each branch can be combined into a unified feature representation using methods such as feature concatenation, weighted summation, or attention-weighted fusion. Subsequently, the fused high-dimensional features are input into at least one fully connected mapping network, and compressed to a preset dimension d (e.g., d = 256) through linear transformation and nonlinear activation, resulting in a low-dimensional dense state feature vector for the current time step. :

[0095] (1)

[0096] In the formula: Represents a state-coded network; Let R represent the state representation vector, where R is the set of real numbers.

[0097] This state feature vector It also encodes local details, global layout, and historical dynamic information of the environment, which can be used as a shared input representation for subsequent multi-granularity semantic prompt generation and strategy decision-making modules.

[0098] In another specific embodiment, the feature encoding module can be implemented using a simplified convolutional neural network structure: the convolutional neural network includes two convolutional layers, with the first layer having 32 kernels and the second layer having 64 kernels, followed by a ReLU activation function after each convolutional layer; after completing the two convolutional and activation layers, the feature map is flattened, and the flattened features are mapped to a fixed-length dense vector through at least one fully connected layer, which is used as the state feature representation at the current time step. This embodiment has a more lightweight structure and is suitable for deployment scenarios with high real-time requirements or limited computing resources.

[0099] Step 2: Based on the state feature vector, call the large language model to generate multi-granularity prompt information of macro-level prompts, meso-level prompts, and micro-level prompts, and uniformly encode each granularity prompt to obtain the corresponding multi-granularity prompt vector.

[0100] Specifically, step 2 includes:

[0101] Step 2.1: Convert the current state feature vector obtained in Step 1 into a single vector. Summary of Historical Trajectory By concatenating the data, a context state representation at time t is formed. The state representation is then input into the Large Language Model (LLM), which dynamically generates multi-granular language prompts given the context.

[0102] Multi-granularity language prompts include macro-prompts, meso-prompts, and micro-prompts, denoted as follows: , , The three granularities of prompts output by the large language model at time t constitute the prompt set. :

[0103]

[0104] In the formula: , These indicate macroscopic, mesoscopic, and microscopic granularity levels, respectively.

[0105] Macroeconomic Tips Macro-level cues are used to characterize the global objectives and long-term strategic planning of a task, focusing on describing the overall action direction and resource allocation principles of an agent over a longer timescale. For example, in the StarCraft II environment, macro-level cues might involve abstract instructions such as "prioritize resource gathering," "switch between offensive and defensive strategies," and "prioritize expansion and operations," used to constrain the agent to maintain consistency and coherence in its strategic objectives at the global level. Meso-level cues... Used to characterize local environmental constraints and phased sub-task requirements, focusing on depicting local decision-making information at medium temporal and spatial scales. For example, meso-level cues can reflect changes in the distribution of local map resources, the quantity and type of enemy and friendly units within a certain time window, or the risk-benefit trade-offs in a specific area, providing guidance for agents' tactical adjustments and resource allocation within local areas. Micro-level cues This is used to generate finer-grained, real-time action commands that directly affect atomic actions at the execution level, such as the movement, attack, construction, and skill casting of individual units. Micro-level cues provide real-time, fine-grained operational guidance during high-frequency interactions, ensuring the precision and flexibility of strategy execution. This is achieved through a set of cues at the macro, meso, and micro levels. This ensures the hierarchy and diversity of prompts at both the time scale and abstract level: the macro level is responsible for the overall guidance of long-term goals and global strategies, the meso level is responsible for the dynamic regulation of local battlefields and phased tasks, and the micro level is responsible for the fine-grained execution control at the immediate action level, thereby significantly improving the matching degree and collaborative decision-making ability between language prompts and the dynamic evolution of the environment.

[0106] Step 2.2: Perform semantic encoding on the macro-level, meso-level, and micro-level cues obtained in Step 2.1, mapping each granularity of cues to a unified vector space to obtain macro-level cues vector, meso-level cues vector, and micro-level cues vector.

[0107] In step 2.2: Each layer of prompt text... Input semantic encoder This yields the hint embedding vector.

[0108] The hint embedding vector is:

[0109]

[0110] In the formula: Represents the time t at time t. The embedding vector of the layer prompt is a low-dimensional vector representation of the prompt text generated at time step t, which belongs to the granular level (macro, meso, micro) after semantic encoding; This indicates the set of granularity for prompting, where For macroscopic level hints, This serves as a reminder for the meso-level. This is a hint for the microscopic layer; This represents the environmental state feature vector of the system at time step t, output by the environmental state perception module. This represents a summary of trajectory information within the k steps from time tk to t; This represents the semantic fusion function of the language model at time step t, i.e., a callable large language model, with input as... and The output is a natural language prompt or a semantic hidden state; This means mapping the natural language output of a Large Language Model (LLM) to a compact, low-dimensional vector, used to compress the semantic prompts of the LLM output into continuous numerical vectors for direct use by the policy network.

[0111] Step 3: During the continuous interaction between the agent and the environment, based on the preset fixed period detection conditions, state change detection conditions and abnormal behavior detection conditions, determine whether the multi-granularity prompt needs to be updated. When any detection condition is met, the prompt is updated and Step 2 is re-executed.

[0112] Step 3 includes:

[0113] Step 3.1: At each time step, perform fixed period detection, state change detection, and abnormal behavior detection for the current time step. When any one of the fixed period condition, state change condition, or abnormal behavior condition is met, generate a prompt update trigger signal.

[0114] Step 3.1 designs a dynamic multi-granularity prompt generation mechanism based on environmental state triggering. The aim is to balance the response frequency of prompt generation with policy stability, achieving intelligent control by generating prompts when necessary and suppressing them when ineffective, thereby improving the efficiency of semantic injection and the robustness of policy gains. The prompt update triggering module is refined into three parallel detection sub-units: a fixed-period scheduling unit, a state mutation detection unit, and an abnormal behavior detection unit. At each time step t, the system receives current observation information and agent position data from the environmental state perception module, and sequentially processes them through the three units. When any unit meets the triggering condition, it determines that the prompt set needs to be refreshed and sends a "prompt update request" signal to subsequent prompt generation steps.

[0115] The fixed-period detection process of the fixed-period scheduling unit includes: the system maintains a global time step counter t, which increments by 1 every time it interacts with the environment, and the period length parameter is preset. It can be calibrated offline based on task duration, rate of environmental change, etc., with typical values ​​ranging from 20 to 200 steps. At time step t, the fixed-period scheduling unit calculates and triggers an update based on the following formula:

[0116]

[0117] In the formula: t represents the time step index corresponding to the current decision moment; This represents a preset fixed period length parameter, used to force a refresh of the prompt set within a fixed time interval. This ensures that even if key environmental quantities do not change significantly, the prompts can be updated according to a predetermined rhythm, thereby reducing the risk of semantic drift and decision shift caused by prompt expiration. This represents the modulo operator; This indicates a periodic trigger condition if and only if t can be triggered. When the integer division occurs, the fixed-period scheduling unit triggers an update prompt.

[0118] If equation (4) holds true, a unified update time benchmark is reached, triggering a prompt update; otherwise, it is not triggered by the periodic condition. Through equation (4), even if the key environmental variables change little within a certain period of time, the unit will force a refresh of the prompt set at fixed time intervals to avoid semantic lag and strategy drift caused by the long-term use of old semantics in prompts.

[0119] The state monitoring and mutation determination process of the state mutation detection unit includes: selecting several key state variables from the environmental state vector to form a set K, such as the quantity of resources. (Total mineral and gas content, etc.), enemy unit count, critical target health, base security, etc. The system then maintains the value of each critical variable from the previous time step, such as... This is implemented using a circular buffer or a simple register. Taking the number of resources as an example, it is calculated at time step t. And determine whether its absolute value exceeds the threshold. :

[0120]

[0121] In the formula: Indicates the amount of resources at the current moment; Indicates the amount of resources at the previous moment; This is a preset threshold for change.

[0122] The settings can be made based on experience or statistical characteristics, such as taking a certain percentage of the maximum resource capacity (e.g., 5%-10%), or adaptively setting it based on the historical sample variance. This mechanism ensures that the prompts can respond promptly to sudden changes in mission objectives or environmental conditions, such as a sudden decrease in resources or a sharp increase in the number of enemy units.

[0123] Furthermore, multivariate expansion: for each key quantity Independent thresholds can be configured for all of them. When any variable satisfies: When this occurs, it is considered a state change, triggering a prompt update.

[0124] This state change detection unit ensures that the prompts are updated in a timely manner when unexpected situations such as a sudden decrease in resources, a surge in enemy forces, or damage to key buildings occur, guiding the strategy network to reassess the situation and improving its robustness in responding to environmental changes.

[0125] The abnormal behavior detection unit's behavior trajectory monitoring and identification process includes: system maintenance length is A sliding window records the agent's recent... The position sequence of each time step is Calculate the Euclidean distance to the previous step for each step within the window:

[0126]

[0127] When the average movement distance is lower than the minimum movement threshold At that time, it is believed that the intelligent agent may get stuck in a stagnation or local inefficiency cycle:

[0128]

[0129] In the formula: t represents the time step index corresponding to the current decision moment, used to mark the t-th step of the agent's interaction with the environment; This indicates the length of the trajectory window used for abnormal behavior detection, i.e., the length of the most recent... The time scale for statistical analysis of agent movement within a time step. It is a positive integer; This represents the position information of the agent at the i-th time step, which can be a two-dimensional or three-dimensional coordinate vector, such as map coordinates (x, y, z). i ,y i ) or (x i ,y i ,z i ); This represents the displacement distance of the agent between two adjacent time steps i-1 and i; Indicates that recently The average distance moved by the agent within each time step is used to quantify its overall level of activity. This represents the minimum movement threshold. When the average movement distance is less than this threshold, the agent is considered to be in a state of stagnation, wandering in a small area, or repeating inefficient behaviors. The value can be preset based on the map size and normal unit movement speed, or determined experimentally.

[0130] If the condition in formula (7) is met, a prompt update is triggered. This abnormal behavior detection unit can automatically trigger new prompts when the agent moves back and forth in a small area for a long time, stays in place, or repeatedly performs the same invalid action, encouraging the strategy to break out of local optima or invalid exploration, and improving the flexibility and diversity of strategy execution.

[0131] Step 3.1 ensures basic timeliness of prompts at fixed intervals, agile response capability of prompts during sudden changes in state, and proactively breaks inefficient cycles when abnormal behavior occurs. This achieves a balance between prompt update frequency and strategy stability, providing a reliable triggering mechanism for subsequent multi-granularity prompt generation and fusion.

[0132] Step 3.2: When the prompt update trigger signal exists, the context state representation at time t is reconstructed based on the current state feature vector and historical trajectory summary at that time step. The reconstructed context state representation is input into the large language model to obtain new macro prompts, new meso prompts, and new micro prompts. A new multi-granularity prompt set is formed from the new macro prompts, new meso prompts, and new micro prompts.

[0133] When an update is triggered, the state digester and history compressor are first invoked to process the current environment state vector. and historical trajectory summary Organize the data to generate a status description text for prompting the agent. This text might include: the agent's current coordinates and orientation, the spatial distribution and remaining quantity of the target area or resource points, the number and approximate location of friendly and enemy units, the current task stage, and remaining time constraints. An example text format could be: "Our unit is currently located at coordinates (15,20). There is an uncollected resource point A in the northeast region, with 120 units of minerals remaining; collection efficiency has decreased by 10% in the past 50 steps. Please provide macro-level strategies, meso-level regional scheduling, and micro-level action suggestions, and output them in the format of coord, direction, step."

[0134] The aforementioned state description text is taken as input and fed into the Large Language Model (LLM) in the language model calling unit. In a preferred embodiment, by explicitly specifying the output format in the prompt, the LLM is required to provide instructions at three granularities: macro, meso, and micro. For example, it can be returned in a JSON-like structure: at the macro level, the global movement or exploration direction; at the meso level, the movement direction and duration of a specific region block or resource cluster; and at the micro level, the precise movement direction and execution steps of the current control unit.

[0135] The natural language or structured text output by the language model is processed by the parser in the prompting parsing and encoding unit, mapping key fields into triples (coord, direction, step). The coordinate parameter coord represents the coordinate information of the target region or target location, which can be discrete grid coordinates (x, y) or a region ID. For the macroscopic layer M... a The preferred coordinates for the coord are the center coordinates of a large region or quadrant; for the mesoscopic layer M... b coord represents a specific resource point or a medium-sized region block; for the micro-level M i `coord` indicates the target position accurate to the cell level. The direction parameter `direction` represents the direction of movement from the current coordinates to the target coordinates. In one embodiment, `direction` is a discrete direction marker selected from the set {N,NE,E,SE,S,SW,W,NW,STAY}, or it can be represented in angular form as a continuous value θ∈[0,2π]. The execution step size `step` represents the number of time steps for continuous movement or holding in the given direction. It is a positive integer and is constrained to a preset range [1,step]. max Within the macro-level, the step value is larger and is used to describe the global movement trend over a long period of time, while the step value is smaller and is used to describe the detailed action plan within a few steps.

[0136] The resulting triplet (coordt, directiont, stept) is written into the corresponding granularity of the cue set to form a new multi-granularity cue set. This set is then cached for subsequent decision calculations in the cue fusion module and policy network. Granularity cue set for:

[0137]

[0138] In the formula: Indicates the target area or location coordinates; Indicates the direction of movement; Indicates the execution step size.

[0139] By using the multi-mechanism joint triggering method of formula (8), the prompts can take into account periodic updates, responses to environmental changes and corrections of abnormal behaviors, thereby improving the dynamic adaptability and robustness of the decision-making process.

[0140] At time step t, the system uses the flags from the fixed-period scheduling unit, the state change detection unit, and the abnormal behavior detection unit to determine whether the prompt set needs to be refreshed. If a refresh is needed, an update command is sent to the prompt generation module, and the trigger cause type (periodic, state change, abnormal behavior) is recorded for subsequent strategy learning or parameter tuning.

[0141] Step 4: After completing the multi-granularity cue vector encoding, at each decision time, the current state feature vector is fused with the embedding vectors of macro-cues, meso-cues and micro-cues through a cross-attention mechanism to obtain a policy fusion feature adapted to the current environmental state.

[0142] Step 4 includes:

[0143] Step 4.1: Embed the macro-level hints generated in Step 2 Mesoscopic hints embedded and micro-cue embedding Concatenate them in order, align them within the same matrix, and you will get the hint embedding matrix.

[0144] At time t, three types of cue embedding vectors are obtained from step 2.2, including macroscopic cue embedding vectors. Mesoscopic hints embedded Micro-level hints embedded The three types of hints are concatenated along the row dimension in a fixed order of macro, meso, and micro to form a hint embedding matrix:

[0145]

[0146] In the formula: This indicates that a splicing operation is performed in the row direction; This indicates the dimension of a single prompt embedding.

[0147] Step 4.1 unifies the originally independent macro-level, meso-level, and micro-level cues into a cue embedding matrix with a fixed number of rows, providing structured input for subsequent cross-attention calculations.

[0148] Step 4.2: Using the current state feature vector as query vector The prompt will be embedded in the matrix. As key vector K and value vector V, fused features are calculated through a cross-attention mechanism.

[0149] At time t, the current state feature vector of the policy network input is taken as... In one embodiment, it is preferred to set This allows the query vector to have the same dimension as the key vector and value vector, where State feature vector Dimensions Embedding vector for a single cue , , The dimension; in another embodiment, it can also be transformed by linear transformation. and Projected onto a unified dimension .

[0150] In this embodiment, the state feature vector and the cue embedding matrix are directly used as inputs to the cross-attention, letting , ,in Let K be the query vector, V be the key vector, and V be the value vector, with all three vectors having the same dimension. According to the calculation formula for cross-attention, the fused features are obtained:

[0151]

[0152] The cross-attention mechanism (i.e., the Attention function) is defined as follows:

[0153]

[0154] In the formula: The query vector, in this invention, is represented by the state feature vector at the current moment. Composition; K represents the key vector matrix, which in this invention consists of the hint embedding matrix. Composition; V represents the value vector matrix, which in this invention is the same as the key vector matrix, and is also composed of the hint embedding matrix. constitute; This indicates the dimension of the query vector and the key vector, used for scaling the relevance score. This represents the similarity score matrix of the inner product between the query vector and the key vectors of each row; This represents an exponential mapping function that normalizes row by row, used to convert the scoring matrix into attention weights for each key vector.

[0155] Furthermore, substituting formula (11) into formula (10), we obtain the final fusion feature:

[0156]

[0157] In the formula: t represents the time step index; The state feature vector at time t has the preferred dimension. ; The cue embedding matrix at time t has the preferred dimension. , where 3 represents the three row vectors corresponding to the three types of prompts (macro, meso, and micro); Representation matrix The transpose of is used to... Perform dot product matching to generate a relevance score; This represents the unified feature dimension of the query vector and the key vector, used to scale the dot product value for stable training; This represents the scaled dot product similarity score between the query vector and the three suggestion key vectors; This represents the exponential mapping function normalized row-wise, yielding the attention weights. .

[0158] The calculation result is the fused feature vector obtained by weighted summation of attention weights and value vector matrix V. Through the cross-attention mechanism, the policy network can automatically learn the semantic interaction weights between state features and prompts at each level, generating policy fusion features suitable for the current environment state.

[0159] Step 5: Input the fused features into the policy network to obtain the action probability distribution and determine the policy network candidate actions. At the same time, determine the prompt candidate actions based on the current multi-granularity prompts. Based on the prompt confidence and fusion probability, select the final action to be executed between the prompt candidate actions and the policy network candidate actions.

[0160] Specifically, take the fused feature vector obtained in step 4. As input to the policy network, the fused state feature vector is fed into the policy network, and the neural network parameters are used for mapping and normalization to obtain the probability distribution function of the candidate actions:

[0161]

[0162] In the formula: This indicates that, given a fused feature vector Under the given conditions, each selectable action The probability distribution; The state-hint fusion feature vector obtained through the cross-attention mechanism is shown in step 4; Let represent the weight matrix of the output layer of the policy network, with dimension . ,in The number of discrete actions. To fuse feature vectors The dimension is the trainable parameter; The bias vector corresponding to the weight matrix is ​​a trainable parameter. This represents a function that performs exponential normalization along the action dimension, used to convert the scores obtained from linear transformation into probabilities for each action.

[0163] Based on the probability distribution, candidate actions for the policy network can be obtained by sampling or maximum value selection:

[0164]

[0165] In the formula: This represents the candidate actions given by the reinforcement learning policy network, i.e., based on the probability distribution. The selected action; This indicates that, given a fused feature vector Under the given conditions, each selectable action The probability distribution; Operators that select an action from a given probability distribution.

[0166] In a preferred embodiment, the sampling method can be multinomial sampling, or it can be combined with an ε-greedy strategy to maintain exploratory nature.

[0167] Furthermore, take the granularity hint set obtained in step 3. According to predefined action mapping rules, it is converted into a prompt candidate action. For example: Mapping to movement commands in eight directions or continuous angles will Mapped to the number of steps or repetitions of the same action, when multiple controllable units exist, it can be... Used to determine controlled units or target areas. In one embodiment, the generated prompt actions can be validated for legality, for example: checking whether the target coordinates are within the map's reachable area, checking whether the action matches the current unit type, and checking whether the number of steps executed is within the allowed range. If the validation fails, the prompt is directly marked as untrustworthy, and penalized in subsequent confidence calculations.

[0168] Furthermore, the credibility of the current round of prompts is assessed, and the prompt confidence level is calculated. This confidence level comprehensively reflects the reliability of the prompt in terms of semantic completeness, task matching, and executability. In a preferred embodiment, the prompt confidence level... It can be calculated in the following form:

[0169]

[0170] In the formula: This indicates the completeness score given based on the parser's parsing results of the prompt format, such as whether all required fields have been correctly filled; This indicates the matching score between the prompt action and the current task objective, such as whether the prompt direction points to the target area, and whether it is conducive to resource collection or tactical advancement; This indicates the legality score of an action, such as the degree of feasibility after collision detection or boundary crossing detection; , , Represents the weighting coefficients, satisfying + + .

[0171] The overall reliability of the prompt is quantified into a real number in the interval [0,1] by formula (15) and used for subsequent "dual-source fusion" decision-making.

[0172] Furthermore, in the action selection stage, the present invention uses a prompting system for candidate actions. With policy network candidate actions It employs a dual-source fusion mechanism, rather than relying solely on either source. A pre-set confidence threshold for the prompt is used. and fusion probability parameters Fusion probability parameters The probability of triggering a prompt is controlled, assuming the prompt is credible, to balance prompt utilization with autonomous strategies. In one embodiment, the probability fusion... It can be adaptively adjusted according to the training process or time step. For example, a smaller value can be taken in the early stage of training to encourage the policy network to explore on its own; and the value can be gradually increased after the training is stable to make full use of high-quality cues.

[0173] At time t, random numbers rand() are uniformly sampled from the interval (0,1), and the final action selection rule is determined according to the following formula. :

[0174]

[0175] In the formula: This represents the final action performed at time t; This indicates that based on the current prompt information The mapped candidate actions are suggested. This represents the candidate actions given by the reinforcement learning policy network, i.e., based on the probability distribution. The selected action; This represents the confidence level of the current prompt, with a value range of [0,1]. The larger the value, the more reliable the prompt. This indicates the confidence threshold. This represents a random number uniformly distributed in the interval (0,1), used to introduce random selection when the hint is reliable; This represents the fusion probability parameter, with a value range of (0,1], which is used to limit the upper limit of the probability of selecting a prompt candidate action when the prompt confidence meets the condition; Indicates when the condition When neither condition is simultaneously met, i.e., when the confidence level is insufficient or the fusion probability condition is not met, the system selects candidate actions from the policy network. As the final action.

[0176] When the confidence level of the suggestion is high, that is, when the confidence level of the suggestion is high. ＞ Confidence threshold suggestion At that time, and the fusion probability meets the set conditions. When the prompt information is used, the action candidates generated should be used first. In cases where the prompts are unreliable or the fusion probability is insufficient, i.e. or If both occur, then return to the output of the policy network. .

[0177] In this way, the present invention effectively avoids the problems of excessive intervention by prompts or excessive deviation of the policy network, so that action decisions can not only improve the rationality with the help of prompt information, but also maintain the exploratory and adaptive nature required by the reinforcement learning process, thereby improving the robustness and generalization ability of the system.

[0178] Step 6: Based on the feedback generated in the environment by the final action, calculate the task completion reward, exploration reward, prompt response reward, behavior dispersion reward, and behavior penalty, and combine the rewards to obtain the composite reward function at the current moment.

[0179] Specifically, step 6 constructs a composite reward function for the agent across multiple dimensions, including task completion, exploration, responsiveness to prompts, behavioral diversity, and suppression of abnormal behavior. This step includes reward items for task completion, exploration, responsiveness to prompts, behavioral diversity, and penalties, which are ultimately combined linearly with weighted averages to obtain the total reward.

[0180] At the task completion level, the agent's goal is to maximize task rewards, such as maximizing the amount of resources gathered or the number of objectives completed in a StarCraft II resource gathering task. To this end, task completion rewards are defined. as follows:

[0181]

[0182] In the formula: This indicates the amount of resources successfully collected in the current round or statistical period, such as the total amount of minerals collected. This represents the weighting coefficient of the task completion reward, used to adjust the proportion of task rewards in the total reward.

[0183] In practice, the newly collected data can be accumulated at each step, or calculated all at once at the end of a round. And based on this, This ensures that the agent continuously learns and optimizes towards achieving the task objective.

[0184] To enhance the agent's exploration capabilities in the environment and prevent it from prematurely converging to local optima, an exploration reward term is introduced. Let the set of new states visited for the first time be denoted as . Define exploration rewards for:

[0185]

[0186] In the formula: This represents the set of states that are accessed for the first time within the current statistical period, such as the first map grid or region cell reached. The cardinality of the set of new states visited for the first time, i.e., the number of new states visited; This represents the exploration reward coefficient, used to control the intensity of the impact of exploration gains on the total reward.

[0187] During project implementation, the system maintains a "set of visited states" and checks at each step whether the current state is a new state; if it is a new state, it is added to the new state list. It also increases exploration rewards to encourage agents to continuously expand their exploration areas.

[0188] At the cue response level, a cue response reward is introduced to measure the agent's compliance with cue information when performing actions. Cue Response Reward Defined as:

[0189]

[0190] In the formula: This indicates the number of times the agent's actions are consistent with the prompts and suggestions within the current statistical period, such as the number of steps taken to continuously follow the prompt's direction or action. This indicates the reward weighting coefficient for the prompt response.

[0191] During implementation, at each time step, the actual action performed by the agent is compared with the candidate action given by the prompt generation module. Whether it matches, and increments when it matches. This reward enhances the guiding role of prompts in decision-making and reduces the time required for the policy network to explore high-quality policies.

[0192] To prevent agents from repeatedly acting in localized areas and resulting in insufficient exploration space coverage, this invention introduces a behavior dispersion reward term, utilizing the agent's spatial location distribution to measure behavioral diversity. First, the location dispersion is defined as follows:

[0193]

[0194] And command: Dispersed rewards for behavior

[0195] In the formula: , The spatial coordinates of the agent at the i-th time step can be map grid coordinates; n represents the total number of time steps within the current statistical window, such as the number of recent steps or the number of steps in a round. This represents the average Manhattan distance between all pairs of locations within the window, used to measure the spatial dispersion of the agent's movement trajectory. This represents the distributed reward coefficient for the behavior.

[0196] By calculating the average distance between trajectory points, when the agent moves over a large area... Larger, corresponding This also increases, thereby encouraging more dispersed and diverse exploratory behaviors.

[0197] To prevent the agent from stagnating for extended periods, performing ineffective actions, or falling into infinite loops, a penalty term is introduced to negatively reward persistent ineffective actions. The penalty term is defined as follows:

[0198]

[0199] In the formula: This indicates the number of actions that do not produce effective output within a certain period of time, such as the number of steps of continuously "staying in place" or repeatedly performing actions that have no impact on the environment; This represents the penalty coefficient, which is positive. The negative sign in front of it indicates the deduction of the reward.

[0200] During implementation, the system determines whether a certain action is an "invalid action" based on preset rules (e.g., no change in position, no change in resources, and no combat). If it is determined to be invalid, then... This can be used to punish the behavior, prompting the agent to actively avoid inefficient behavior patterns.

[0201] Finally, the above reward items are combined in a linear weighted manner to obtain the composite reward for the current moment or the current round:

[0202]

[0203] In the formula: , , , , The weight parameters are given by formulas (17)-(22). - A balance can be achieved between task benefits, exploratory nature, cue compliance, behavioral diversity, and inhibition of aberrant behavior through empirical settings or parameter tuning during training.

[0204] Through the composite reward function constructed in step 6 above, this invention not only considers the completion of the task objective itself, but also introduces multi-dimensional indicators such as exploratory nature, prompt responsiveness, behavioral diversity, and abnormal behavior punishment, so that the agent can achieve synergistic improvement in learning efficiency, behavioral stability and policy diversity during the training process, thereby improving the overall learning performance and policy generalization ability.

[0205] Step 7: Based on the state feature vectors at multiple time steps, the corresponding final execution actions, the composite reward value, and the state feature vector at the next time step, iteratively update the parameters of the policy network so that the policy network can complete the optimization under the guidance of the composite reward.

[0206] Step 7: The composite reward function constructed in Step 6 Guided by this approach, the policy network is iteratively trained and its parameters optimized to improve the agent's long-term learning performance in complex environments. Specifically, in each training iteration, the current policy network interacts with the environment for several rounds, resulting in a trajectory set consisting of states, actions, rewards, and the next state.

[0207]

[0208] In the formula: This represents the sample dataset (trajectory set) obtained by the agent interacting with the environment in one or more training iterations; t represents the time step index, used to identify the t-th moment of the agent's interaction with the environment, t=1,…,T, where T is the total number of steps of the sampled trajectory in this iteration; It represents the environmental state observed at time step t, such as a encoded state vector including resource information, unit location, etc. Indicates the state The following are the actions output and actually executed by the policy network; This represents the composite reward value obtained at time step t, which is obtained by a linear combination of the reward items defined in step 6. Indicates the execution of an action It then interacts with the environment and transitions to the next state.

[0209] Based on the sampled trajectory, the dominance function at each time step is calculated using the temporal difference or generalized dominance estimation (GAE) method. :

[0210]

[0211] In the formula: This represents the estimated advantage function at time step t, used to measure the performance of a state. Next action How well or poorly it performs relative to the current average level of the strategy; This represents the accumulated discount reward starting from time step t and proceeding along the trajectory. This indicates that the value network represents the state. The given state value estimate reflects the state value under the current policy. The expectation of long-term returns upon departure.

[0212] In the advantage function Based on this, the Proximal Policy Optimization (PPO) algorithm is used to update the policy network parameters. A cut-off objective function is constructed. :

[0213]

[0214] In the formula: This represents the objective function used by the Proximal Policy Optimization (PPO) algorithm; a larger value indicates a better current policy update direction. This indicates the expected value or average value over the sampling time step t, which is generally calculated and averaged over a mini-batch of samples. The policy probability ratio at time step t is defined as the ratio of the probability of the new policy and the old policy being in the same state and action pair. This represents the estimated value of the dominance function at time step t; Indicates to Perform a truncation (trimming) operation, that is, when Less than Time to take ,when Greater than Time to take Otherwise keep constant; This represents the shearing hyperparameter used to limit the magnitude of a single policy update. Typical values ​​are 0.1-0.3 to prevent excessive policy changes from causing training instability. It is a preset shearing coefficient. This indicates taking the smaller value of the two expressions within the parentheses, used to constrain the objective function whether the advantage is positive or negative.

[0215] In the implementation, the Adam optimizer is used to update the parameters of the policy network and the value network using gradient descent. The learning rate, momentum coefficient, and other hyperparameters of Adam are set before training or adaptively adjusted during training. Through multiple iterations, the policy network gradually approaches the optimal policy under the meaning of compound reward.

[0216] During training, an adaptive adjustment mechanism is introduced to dynamically adjust several key hyperparameters based on observed composite reward levels, cue response counts, and the number of exploration states. These hyperparameters include, but are not limited to: 1. Cue confidence threshold. With fusion probability When the overall quality of the prompts is high, the prompt weight should be appropriately increased; when the prompts do not contribute significantly to the returns, the prompt weight should be decreased to enhance the autonomous exploration ratio of the strategy network; 2. Weight parameters such as exploration reward coefficient and penalty coefficient. - The system fine-tunes its learning rates based on changes in task completion rate and exploration coverage during training. The learning rates of the policy network and value network are automatically adjusted to decrease or increase when training oscillations or slow convergence are detected. Through these adaptive adjustments, the system can continuously balance the relationship between task rewards, exploration depth, and cue utilization during training, preventing learning from getting stuck in local optima or experiencing unstable convergence due to improper weight settings for any one dimension.

[0217] Through the training and optimization process constructed in step 7, under the unified guidance of the composite reward function, the parameters of the policy network can be continuously updated as the agent interacts with the environment. Combining the stable update characteristics of PPO and the adaptive gradient optimization characteristics of Adam, as well as the dynamic adjustment of hyperparameters such as cue weights and exploration weights, the learning efficiency, behavioral stability and policy diversity of the agent in complex environments are improved simultaneously.

[0218] Example

[0219] This embodiment employs a reinforcement learning decision-making method that integrates multi-granularity semantic prompts from a large language model. Building upon steps 1-7 above, the StarCraft II resource acquisition task CollectMineralShards is used as the application scenario, and implementation and verification are completed on the hardware and software platforms shown in Tables 1 and 2. The hardware platform includes: an Intel Core i5-12400F processor, 16GB of RAM, an NVIDIA GeForce RTX 4060 graphics card, and a Windows 10 x64 operating system. The software environment includes: Python 3.9, PyTorch 2.0, StarCraft II v4.10, and PySC2 runtime environment. The large language model is called via the DeepSeek-Chat API to generate multi-granularity semantic prompts, thereby guiding the training and decision-making of the reinforcement learning strategy.

[0220] Table 1

[0221]

[0222] Table 2

[0223]

[0224] On this platform, the method includes the following steps:

[0225] Step 1: Collect environmental status information and extract feature vectors.

[0226] As per the instruction manual Figure 2 During the interaction between the intelligent agent and the environment, the environmental state perception module collects raw state information from the StarCraftII environment in real time, including unit coordinates, resource point distribution, friendly unit attributes (health, load), enemy reconnaissance information, etc.

[0227] The collected raw states are first normalized and standardized in the state preprocessing unit. The spatial coordinates are scaled to the [0,1] interval, the resource quantity is reduced by logarithmic compression to reduce the numerical span, and the time series is smoothed using a sliding window to eliminate high-frequency noise.

[0228] Subsequently, the state feature encoding unit extracts spatial and temporal features through a three-layer convolutional neural network and a one-layer GRU network, and finally outputs a 256-dimensional state feature vector s through a fully connected layer. t This serves as the unified input for subsequent prompt generation and policy decision-making. Experiments show that, compared to directly using the original state, introducing this step 1 increases the agent's cumulative reward after 1 million training steps from 2293.07 to 5374.47.

[0229] Step 2: Generate and uniformly encode multi-granularity prompts using a large language model.

[0230] Based on the environmental state feature vector obtained in step 1 and the trajectory information of the most recent steps, the prompt generation module constructs contextual description text and outputs multi-granular semantic prompts through a large language model.

[0231] As per the instruction manual Figure 3 Specifically, the status digester in the prompt building unit transcribes key information such as the current resource quantity, the distribution of explored areas, and the number and location of units into structured text. The history compressor in the prompt building unit compresses the trajectory of the most recent K steps and extracts key events such as "decreased acquisition efficiency" and "enemy unit appearance," which together form the prompt input text.

[0232] The template constraint unit fills the above text into three types of templates: macro, meso, and micro, to obtain hierarchical prompt requests. The language model invocation unit sends the templated input to the LLM, which then obtains macro-strategy suggestions, regional scheduling suggestions, and specific unit action suggestions.

[0233] The prompt parsing and encoding unit performs rule parsing on the natural language output by LLM, extracts structured fields such as region ID, coordinates, and action set, and encodes each type of prompt into a 512-dimensional prompt vector through the Transformer encoder, so that macro, meso, and micro prompts are mapped to a unified vector space, which facilitates subsequent fusion and computation.

[0234] Step 3: Update prompts based on fixed periods, sudden changes in state, or abnormal behavior.

[0235] During the operation of the intelligent agent, the system continuously determines whether to regenerate prompts based on the current state characteristics and behavioral trajectory. The fixed-period detection process involves setting the refresh step size. =120, when the current time step t satisfies At this time, a forced refresh of the prompt will be performed to ensure that the prompt does not lag for an extended period. The state change detection process involves monitoring key state variables (such as resource quantities). ),when If a drastic change is detected in the mission objective or environmental situation, a new prompt will be generated. The abnormal behavior detection process is as follows: calculate the most recent... The average distance traveled by the intelligent agent when If the agent is determined to be stuck in a stagnant or inefficient loop, an update prompt will be triggered.

[0236] When any trigger condition is met, the system inputs a new state summary into the LLM and regenerates the multi-granularity cue tuple. Meanwhile, a hint caching strategy is adopted: if the current state summary is highly similar to the previous one, the cache hint is directly reused to reduce unnecessary LLM calls and strategy jitter.

[0237] Step 4: Use a cross-attention mechanism to fuse cues and state features.

[0238] The prompt fusion module receives the state feature vector output by the environment state perception module, as well as the prompt vectors of various granularities generated in steps 2 / 3. First, in the feature alignment unit, both are adjusted to the same dimension. Then, in the fusion calculation unit, the state feature vector is used as a query, and the macro, meso, and micro prompt vectors are sequentially concatenated into a prompt embedding matrix. A cross-attention mechanism is used to calculate the correlation between the state and the prompts at each layer, resulting in a fused feature representation. The weight adjustment unit dynamically adjusts the weight of cue information in attention based on the training phase: increasing the cue weight in the early stages of training to guide the rapid formation of the strategy; and gradually decreasing the cue weight in the later stages of training to enhance the agent's autonomous decision-making and generalization capabilities. The final output is a fused feature vector. As input to the policy network.

[0239] Step 5: Generate the final action based on the confidence level and fusion probability provided.

[0240] The action generation module receives the fused feature vector. The probability distribution of each candidate action is calculated using a policy network. And based on this distribution, candidate actions for the policy network are obtained. At the same time, the system will provide the current triplet based on the prompt. Constructing suggestion candidate actions The system evaluates the prompts in terms of semantic completeness, task matching, and action legitimacy, and calculates the prompt confidence score. .

[0241] A dual-source fusion mechanism is adopted in the action selection phase: when (Threshold default is 0.6), and the random number satisfies... When prompted, select a candidate action. As the final action, otherwise, select a candidate action from the policy network. As the final action, the fusion probability β decays exponentially with the training process. , For the initial fusion weights, The attenuation coefficient is... A sliding count is used to indicate the number of times a suggestion is adopted. This design ensures that suggestions have a greater impact in the early stages of training, gradually giving way to the autonomous policies of the policy network in the later stages.

[0242] Step 6: Construct a composite reward function based on multidimensional rewards.

[0243] The reward adjustment module calculates five types of rewards based on environmental feedback and prompts after each action is performed. (Task completion reward) A positive reward (+10 to +50) is given upon completing resource gathering or reaching the target point, reinforcing goal-oriented behavior; exploration rewards are also provided. Rewards (+1 to +5) are given when entering unexplored areas or discovering new resources, increasing exploration coverage; prompts also provide rewards. Rewards (+2 to +10) are given when the actual action closely matches the prompt, encouraging the proper use of prompts; rewards are distributed throughout the behavior. Rewards (+1 to +5) are given when multiple agents are evenly distributed and avoid concentrating in the same area, thus improving overall efficiency; behavioral penalties are also included. When invalid actions, wasted resources, or violations of constraints (such as collisions) occur, negative rewards (−10 to 0) are applied to constrain abnormal behavior.

[0244] The weight allocation unit sets adjustable weights for the aforementioned rewards. , , , , The reward fusion units are combined as follows:

[0245]

[0246] Obtain the composite reward value at the current moment, which will be used for subsequent strategy optimization.

[0247] Step 7: Train and optimize the policy network parameters under the guidance of the composite reward function.

[0248] Building upon the steps outlined above, the policy optimization module iteratively updates the policy network parameters through continuous interaction between the agent and the environment: it continuously collects data sequences consisting of state features, final actions, compound rewards, and the next state; in each training round, it calculates the policy gradient signal based on the compound reward, constructs a shearing objective function using the proximal policy optimization (PPO) algorithm to constrain the variation between the old and new policies, preventing policy oscillations caused by over-updates; it uses the Adam optimizer to update the policy network parameters using gradients, with an initial learning rate set to 0.001; and it monitors metrics such as cumulative reward, cue response rate, and exploration coverage in real time. When the reward improvement is not significant over multiple rounds, it automatically adjusts the learning rate and cue weights to reduce ineffective exploration or increase cue confidence, thereby maintaining the stability and efficiency of the training process.

[0249] As per the instruction manual Figure 4Using the StarCraft II resource gathering mission as a test scenario, a comparative experiment was conducted using the intelligent agent trained by the method of this invention. Figure 4 (a) shows the state at the initial stage of the mission, where multiple resource points are scattered across the ground. Figure 4 Initially, the agents are concentrated in the center of the map. After several interactions, such as... Figure 4 As shown in (b), guided by the multi-granularity prompts and cross-attention fusion mechanism generated by the large language model, each agent is automatically assigned to the vicinity of resource clusters in different directions, enabling rapid expansion of the task area. As some resources are collected and the location of available resources changes, such as... Figure 4 As shown in (c), the system automatically triggers a prompt update after detecting a sudden change in state and an abnormal stagnation trend. The agent then adjusts its movement direction and division of labor, avoiding mined-out areas and turning to uncovered resource points. Later in the task, as... Figure 4 As shown in (d), the agent continues to actively search for and disperse the remaining resource points, without lingering in local areas for a long time or repeating ineffective actions.

[0250] Based on the experimental phenomena described above, it can be seen that the intelligent agent using the method of this invention can more effectively cover the entire task area and improve resource collection efficiency during task execution. At the same time, when the environment changes significantly or the task conditions are complex, the system can automatically adjust its strategy based on multi-granularity prompts and composite rewards, avoiding long-term ineffective operations or rigid actions, and exhibiting better adaptability and stability.

[0251] Through the reward regulation module of this invention, the intelligent agent can achieve a balance between goal completion, exploration and innovation, prompting execution, teamwork, and error constraints.

[0252] The reinforcement learning decision-making method of the present invention integrates multi-granular semantic prompts of a large language model. By introducing adaptively updatable macro, meso, and micro multi-granular semantic prompts and merging them with policy features in depth, and combining them with dynamic confidence trade-offs and composite reward constraints, reinforcement learning can be guided more in real time in a high-dimensional sparse reward dynamic environment. This accelerates convergence, improves exploration efficiency and generalization stability, and significantly reduces policy drift and path dependence risks.

[0253] The above descriptions are merely embodiments of this application, and common knowledge regarding specific structures and characteristics in the solutions is not described in detail here. It will be apparent to those skilled in the art that this application is not limited to the details of the above exemplary embodiments, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Therefore, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A reinforcement learning decision-making method integrating multi-granularity semantic prompts from a large language model, characterized in that, The method includes: Step 1: Obtain environmental state information, extract features from the environmental state information, and obtain the state feature vector at the current moment; Step 2: Based on the state feature vector, call the large language model to generate multi-granularity prompt information of macro-level prompts, meso-level prompts and micro-level prompts, and uniformly encode each granularity prompt to obtain the corresponding multi-granularity prompt vector; Step 3: During the continuous interaction between the agent and the environment, based on the preset fixed period detection conditions, state change detection conditions and abnormal behavior detection conditions, determine whether the multi-granularity prompt needs to be updated. When any detection condition is met, the prompt is updated and Step 2 is re-executed. Step 4: After completing the multi-granularity cue vector encoding, at each decision time, the current state feature vector is fused with the embedding vectors of macro-cues, meso-cues and micro-cues through a cross-attention mechanism to obtain a policy fusion feature adapted to the current environmental state. Step 5: Input the fused features into the policy network to obtain the action probability distribution and determine the policy network candidate actions. At the same time, determine the prompt candidate actions based on the current multi-granularity prompts. Based on the prompt confidence and fusion probability, select the final action to be executed between the prompt candidate actions and the policy network candidate actions. Step 6: Based on the feedback generated in the environment by the final action, calculate the task completion reward, exploration reward, cue response reward, behavior distraction reward, and behavior penalty, and combine the rewards to obtain the composite reward function at the current moment; Step 7: Based on the state feature vectors at multiple time steps, the corresponding final execution actions, the composite reward value, and the state feature vector at the next time step, iteratively update the parameters of the policy network so that the policy network can complete the optimization under the guidance of the composite reward.

2. The reinforcement learning decision-making method for integrating multi-granularity semantic prompts from a large language model as described in claim 1, characterized in that, Step 1 includes: Step 1.1: During the interaction between the intelligent agent and the external environment, environmental status data from the visual sensor, position sensor, task management module and environmental interface are collected in real time; Step 1.2: Perform noise reduction, normalization, and data format unification on the collected raw environmental data; Step 1.3: Input the preprocessed environmental state data into the feature encoding module to extract the local spatial features, temporal features and global context features of the environmental state data. Then, fuse the multimodal features output by the feature encoding module to obtain a set of low-dimensional dense state feature vectors. Low-dimensional dense state feature vector for: ； In the formula: Represents a state-coded network; The state representation vector.

3. The reinforcement learning decision-making method for integrating multi-granularity semantic prompts from a large language model as described in claim 1, characterized in that, Step 2 includes: Step 2.1: Convert the current state feature vector obtained in Step 1 into a single vector. Summary of Historical Trajectory By concatenating the data, a context state representation at time t is formed. The state representation is then input into a large language model, which dynamically generates multi-granular language prompts given the context. Step 2.2: Perform semantic encoding on the macro-level, meso-level, and micro-level cues obtained in Step 2.1, mapping each granularity of cues to a unified vector space to obtain macro-level cues vector, meso-level cues vector, and micro-level cues vector.

4. The reinforcement learning decision-making method for integrating multi-granularity semantic prompts from a large language model according to claim 3, characterized in that, In step 2.1, the three granularity cues output by the large language model at time t constitute the cue set. : ； In the formula: , These indicate macroscopic, mesoscopic, and microscopic granularity levels, respectively. In step 2.2, the prompt text for each layer is... Input semantic encoder The resulting embedding vector is: ； In the formula: Represents the time t at time t. The embedding vector of the layer hint; This indicates the set of granularity for the prompt; This represents the environmental state feature vector of the system at time step t; This represents a summary of trajectory information within the k steps from time tk to t; This represents the semantic fusion function of the language model at time step t; This means mapping the natural language output of a large language model to a compact, low-dimensional vector.

5. The reinforcement learning decision-making method for integrating multi-granularity semantic prompts from a large language model according to claim 1, characterized in that, Step 3 includes: Step 3.1: At each time step, perform fixed period detection, state change detection and abnormal behavior detection for the current time step. When any one of the fixed period condition, state change condition or abnormal behavior condition is met, generate a prompt update trigger signal. Step 3.2: When the prompt update trigger signal exists, the context state representation at time t is reconstructed based on the current state feature vector and historical trajectory summary at that time step. The reconstructed context state representation is input into the large language model to obtain new macro prompts, new meso prompts, and new micro prompts. A new multi-granularity prompt set is formed from the new macro prompts, new meso prompts, and new micro prompts.

6. The reinforcement learning decision-making method for integrating multi-granularity semantic prompts from a large language model according to claim 5, characterized in that, In step 3.1, the condition for triggering the update prompt during fixed-period detection is as follows: ； In the formula: t represents the time step index corresponding to the current decision moment; This indicates the preset fixed period length parameter; This represents the modulo operator; This indicates a periodic trigger condition if and only if t can be triggered. When the integer division occurs, the fixed-period scheduling unit triggers an update prompt. The conditions for triggering the state mutation detection update are: ； In the formula: Indicates the number of variables at the current moment; Indicates the number of variables at the previous time step; The preset threshold for change; The conditions for triggering the update of abnormal behavior detection prompts are: ； In the formula: t represents the time step index corresponding to the current decision moment; Indicates the length of the trajectory window used for abnormal behavior detection; This represents the position information of the agent at the i-th time step; This represents the displacement distance of the agent between two adjacent time steps i-1 and i; Indicates that recently The average distance traveled by the agent within each time step; This represents the minimum movement threshold.

7. The reinforcement learning decision-making method for integrating multi-granularity semantic prompts from a large language model according to claim 5, characterized in that, In step 3.2, the granularity hint set for: ； In the formula: Indicates the target area or location coordinates; Indicates the direction of movement; Indicates the execution step size.

8. The reinforcement learning decision-making method for integrating multi-granularity semantic prompts from a large language model according to claim 1, characterized in that, Step 4 includes: Step 4.1: Embed the macro-level hints generated in Step 2 Mesoscopic hints embedded and micro-cue embedding Concatenate them in order, align them within the same matrix, and you will get the hint embedding matrix; Hint: Embedded Matrix for: ； In the formula: t represents the decision time or time step, corresponding to the t-th observation and decision in the reinforcement learning interaction process; This indicates that a splicing operation is performed in the row direction; This represents the embedding vector of the macroscopic feedback at time t; This represents the embedding vector of the mesoscopic suggestion at time t; This represents the embedding vector of the micro-cues at time t; , , These respectively indicate the macroscopic, mesoscopic, and microscopic granularity categories; This indicates the hint embedding matrix, where 3 corresponds to three types of hints: macroscopic, mesoscopic, and microscopic, each occupying one row. Indicates the dimension of a single prompt embedding; Step 4.2: Using the current state feature vector as query vector The prompt will be embedded in the matrix. As key vector K and value vector V, fused features are calculated through a cross-attention mechanism; ； In the formula: t represents the time step index; This represents the state feature vector at time t; The hint embedding matrix represents time t; Representation matrix transpose; This represents the unified feature dimension of the query vector and the key vector; This represents the scaled dot product similarity score between the query vector and the three suggestion key vectors; This represents an exponential mapping function that is normalized by row.

9. The reinforcement learning decision-making method for integrating multi-granularity semantic prompts from a large language model according to claim 1, characterized in that, In step 5, the final action selection rule is determined. for: ； In the formula: This represents the final action performed at time t; This indicates the candidate actions for the prompt obtained by mapping the current prompt information; This represents the candidate actions given by the reinforcement learning policy network; This indicates the confidence level of the current message; This represents a random number that is uniformly distributed in the interval (0,1); This represents the fusion probability parameter; Indicates when the condition If the conditions are not met simultaneously, the system selects candidate actions from the policy network. As the final action.

10. The reinforcement learning decision-making method for integrating multi-granularity semantic prompts from a large language model according to claim 1, characterized in that, In step 6, the composite reward function at the current time step. for: ； ； In the formula: This indicates the reward items for completing the task; Indicates exploration rewards; This indicates a reward for responding to the prompt; This indicates that the reward for the behavior is distributed. Indicates a penalty item; This indicates the number of resources successfully collected in the current round or statistical period; This indicates the weighting coefficient for the task completion reward; This represents the set of states that are accessed for the first time within the current statistical period; The cardinality of the new state set upon first visit; Indicates the exploration reward coefficient; This indicates the number of times the agent's actions are consistent with the suggested prompts within the current statistical period; This indicates the reward weighting coefficient for the prompt response; , This represents the spatial coordinates of the agent at the i-th time step; , Represents the spatial coordinates of the agent at time step j; n represents the total number of time steps within the current statistical window; Indicates the distributed reward coefficient for the behavior; This indicates the number of actions that did not produce valid output within a certain time period; This represents the penalty coefficient.

Images