Hierarchical strategy learning system for multi-agent adversarial optimization

Abstract

The application discloses a layered strategy learning system for multi-agent adversarial optimization, and particularly relates to the field of simulation confrontation, and is used for solving the problem that it is difficult to automatically generate a reusable structured scheme from massive trajectory data in multi-round confrontation simulation; a multi-agent confrontation simulation platform containing an environment model and an entity set is constructed, state, observation, action, reward and constraint information in each round of confrontation process are uniformly collected and written into a trajectory buffer, time segmentation and cooperative group abstraction based on yield change and spatial relationship are performed on the trajectory data, an event sequence with stage structure is formed, the event sequence is coded as scene features and matched with template features in a scheme template library, a mapping relationship of events to time slot positions, role slot positions and resource slot positions is determined by using an evaluation network and a rule description vector, structured scheme data is generated and converted into an intermediate script instruction sequence which can be parsed and executed by the simulation platform.

Description

Technical Field

[0001] This invention relates to the field of simulated adversarial systems, and more specifically, to a hierarchical policy learning system for multi-agent adversarial optimization. Background Technology

[0002] Currently, decision evaluation platforms based on multi-agent adversarial simulation are widely used in complex scenarios such as traffic dispatching, emergency response, unmanned system swarm confrontation, and virtual security drills. These platforms typically construct virtual environments containing multiple types of entities, resource nodes, and constraint rules, introducing multi-agent reinforcement learning algorithms with centralized training and distributed execution. This allows each agent to automatically learn cooperative and game-theoretic strategies during multiple rounds of adversarial processes. Existing technologies can effectively output entity action sequences, trajectory data, and staged rewards over continuous time scales to verify the merits of a particular strategy combination in a given scenario. However, in real business processes, platform users often do not directly use the action sequences themselves, but rather rely on structured schemes or planning scripts to drive subsequent simulation replays, scheduled execution, or cross-system interactions. For example, it is necessary to fix collaborative groups, key time nodes, key locations, and resource instruction sets in the form of schemes, and to store and version them for a long time. This has led to a common manual translation process in engineering practice: on the one hand, technicians need to manually sort out the collaborative relationships and key events between entities from a large number of time-series trajectories generated by multiple rounds of adversarial simulations, abstracting several stages and action combinations; on the other hand, business personnel then write structured solutions or executable scripts according to the format rules of their respective systems based on these manually sorted results, for subsequent system calls or cross-validation. As the scale of the scenario increases, the types of entities increase, and the constraints become increasingly complex, this method, which relies heavily on manual analysis and experience-based induction, is time-consuming, prone to missing implicit constraints, and makes it difficult to reproduce the fine-grained behavioral logic in adversarial simulations in a timely manner. This makes it difficult to consolidate, reuse, and compare the adversarial strategies obtained from multi-agent reinforcement learning under a unified solution form, thus restricting the application depth of virtual adversarial platforms in high-frequency simulations and automated decision support.

[0003] Therefore, a prominent technical problem has emerged in the aforementioned virtual multi-entity adversarial simulation environment: under the premise of using multi-agent reinforcement learning to obtain multi-round adversarial interaction trajectories and global state data, there is a lack of an automated conversion mechanism for a unified structured scheme representation. It is impossible to reliably abstract, organize, and encode high-dimensional temporal trajectories into scheme data and intermediate scripts that can be directly parsed and executed by the simulation platform or external system while ensuring the complete semantic expression of adversarial behavior. As a result, the efficiency and maintainability of the existing system are significantly limited when it is necessary to quickly generate multiple comparable schemes, support rolling simulations and scheme library construction, and integrate with external scheduling systems.

[0004] To address the aforementioned problems, a technical solution is provided. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a hierarchical policy learning system for multi-agent adversarial optimization. This system constructs a multi-agent adversarial simulation platform that includes an environment model and entity sets. It uniformly collects and writes the state, observation, action, reward, and constraint information during each round of adversarial process into a trajectory buffer. The trajectory data undergoes time segmentation and collaborative group abstraction based on changes in payouts and spatial relationships, forming an event sequence with a stage structure. This event sequence is encoded as scene features and matched with template features in a scheme template library. An evaluation network and rule description vectors are used to determine the mapping relationship between events and time slots, role slots, and resource slots. Structured scheme data is generated and converted into an intermediate script instruction sequence that can be parsed and executed by the simulation platform, thus solving the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] S1: Construct an environment model and entity set, use multi-agent reinforcement learning to obtain state, action and constraint information of multiple adversarial time steps, and store it in the trajectory buffer;

[0008] S2: Based on the trajectory buffer, divide the time interval according to the changes in reward, distance or constraint events, construct an entity relationship graph within the time interval to obtain collaborative groups, and generate event sequences by combining the time intervals and collaborative groups;

[0009] S3: Obtain scene feature vectors using event sequence encoding, match the scene feature vectors with templates in the solution template library to select target templates, and determine the mapping results from events to time slots, role slots, and resource slots by combining the evaluation network with the rule description vectors;

[0010] S4: Generate structured scheme data based on the target template and its event-to-slot mapping results, perform consistency checks on the structured scheme data according to the rule description vector, and convert the checked structured scheme into an intermediate script instruction sequence sorted by time.

[0011] Furthermore, step S1 includes simultaneously collecting global state data, local observation data of each entity, action commands of each entity, instant reward data of each entity, constraint event markers, and resource usage information for each time step during each round of adversarial simulation. The above data are then combined into recording units according to the adversarial round index and time step index and sequentially stored in the trajectory buffer.

[0012] Furthermore, step S2 includes calculating, based on the records of each time step in the trajectory buffer, a reward sequence reflecting the change in overall gains, an average distance sequence reflecting the degree of aggregation in the entity space, and a constraint event count sequence reflecting the rule triggering situation according to the adversarial rounds, and automatically determining the start and end times of the stage based on the judgment result that the change amplitude of adjacent time steps in at least one type of sequence exceeds a preset threshold, thereby obtaining a set of time intervals with clear start and end times.

[0013] Furthermore, step S2 also includes, within each time interval, using each entity in the entity set as a node, constructing an entity relationship graph based on the spatial proximity of entities within the time interval and the consistency of behavior categories, obtaining a set of collaborative groups by dividing the entity relationship graph into connected components and determining edge density, and generating event records on the combination of each time interval and each collaborative group. The event records contain at least time interval information, collaborative group identifier, dominant behavior category, and constraint event category, thereby forming an event sequence arranged in chronological order.

[0014] Furthermore, step S3 includes encoding each event in the event sequence into a fixed-length event feature vector according to the time interval length, its relative position in the whole round of confrontation, the size of the cooperative group, the dominant behavior category, and the constraint event category information, and sequentially encoding the event feature vectors according to the chronological order of the events to obtain a scene feature vector uniquely corresponding to each round of confrontation. The scene feature vector is used to match the template feature vectors in the scheme template library.

[0015] Furthermore, the solution template library includes multiple preset templates. Each template includes a template feature vector that matches the scene feature vector, a set of time slots describing the key time structure, a set of role slots describing roles and collaborative relationships, and a set of resource slots describing resource configuration behavior. Within the template, formal constraint information such as order constraints, resource capacity constraints, and role compatibility constraints are given in advance for the time slots, role slots, and resource slots.

[0016] Furthermore, step S3 also includes calculating the similarity between the scene feature vector and each template feature vector, selecting templates whose similarity meets the preset conditions to form a candidate template set, generating multiple candidate mappings from events to time slots, role slots, and resource slots for each template in the candidate template set, inputting the scene feature vector, template feature vector, candidate mapping structure features, and rule description vector into the evaluation network to obtain a score value used to distinguish the merits of different candidate mappings, and selecting the candidate mapping with the highest score as the mapping result from events to slots.

[0017] Furthermore, step S4 includes, after selecting the target template, according to the event-to-time slot mapping result, merging the start time and end time of multiple events mapped to the same time slot into the time interval information in the time field set, and maintaining the reference relationship with the event sequence; at the same time, according to the event-to-role slot mapping result and the collaboration group set, filling the entity identifier and its type information corresponding to each role slot into the role field set, and recording the association between the role and the time field.

[0018] Furthermore, step S4 also includes extracting the resource type, resource quantity, resource source object, and resource target object corresponding to each resource slot based on the event-to-resource slot mapping result and the resource usage record in the trajectory buffer, organizing them into resource instruction entries in the resource instruction field set, merging resource instructions with continuous time and consistent parameters, retaining resource instructions with overlapping time but different target objects, and recording the reference information of the corresponding time field and role field in each resource instruction entry.

[0019] Furthermore, step S4 also includes performing a consistency check on the time field set, role field set, and resource instruction field set in the structured scheme data with the rule description vector. After confirming that there are no contradictory relationships between the fields that violate the time order constraint, resource capacity constraint, and role compatibility constraint, the structured scheme data is sequentially converted into a script instruction sequence ordered by time. Each script instruction includes an instruction type, target identifier, parameter set, and time trigger condition, and is attached with a corresponding event record and a traceability mark for the time interval.

[0020] The technical effects and advantages of the hierarchical policy learning system for multi-agent adversarial optimization proposed in this invention are as follows:

[0021] This solution unifies and structures the data throughout the entire virtual multi-entity adversarial simulation process. It abstracts the previously scattered state, action, and constraint information, originally presented as time steps, into stage intervals, collaborative groups, and semantically clear event sequences. These are then mapped onto predefined scheme templates, ultimately forming well-structured scheme data and executable scripts. This achieves a seamless link from adversarial behavior to scheme structure and execution scripts within a single technical framework, allowing the simulation results to be stored in a stable format and repeatedly used, facilitating high-frequency simulations and unified management.

[0022] This scheme introduces collaborative groups and stage intervals as intermediate representations at the event layer. Combining rule description information and an evaluation network, it selects the optimal mapping between events and time slots, role slots, and resource slots. This ensures that the generated structured scheme has clear boundaries and consistent semantics across the three dimensions of time organization, role allocation, and resource configuration. By performing consistency checks on the scheme data and transforming it into a time-ordered sequence of script instructions, a unified execution entry point and a consistent scheme expression method can be maintained across different simulation platforms, reducing redundant modeling and manual conversion.

[0023] This solution enables automated generation and traceable management of solutions from data in virtual multi-entity adversarial scenarios. It preserves the collaborative behavior patterns generated during single or multiple simulations in the form of standardized solutions for long-term use. This can be used for rapid replay and comparison of solution differences in different rounds, and also facilitates the construction of an extensible solution library to support subsequent analysis, evaluation, and optimization decisions, providing a stable foundation for solution generation and application for complex adversarial businesses. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the hierarchical policy learning system for multi-agent adversarial optimization according to the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1: Figure 1 This invention presents a hierarchical policy learning system for multi-agent adversarial optimization, comprising:

[0027] S1: Construct an environment model and entity set, use multi-agent reinforcement learning to obtain state, action and constraint information of multiple adversarial time steps, and store it in the trajectory buffer;

[0028] S2: Based on the trajectory buffer, divide the time interval according to the changes in reward, distance or constraint events, construct an entity relationship graph within the time interval to obtain collaborative groups, and generate event sequences by combining the time intervals and collaborative groups;

[0029] S3: Obtain scene feature vectors using event sequence encoding, match the scene feature vectors with templates in the solution template library to select target templates, and determine the mapping results from events to time slots, role slots, and resource slots by combining the evaluation network with the rule description vectors;

[0030] S4: Generate structured scheme data based on the target template and its event-to-slot mapping results, perform consistency checks on the structured scheme data according to the rule description vector, and convert the checked structured scheme into an intermediate script instruction sequence sorted by time.

[0031] The technical solution proposed in this invention revolves around the solution generation requirements in virtual multi-entity adversarial simulation. The overall logic consists of four levels: data acquisition and organization, stage and collaborative abstraction, templated structure alignment, and solution and script generation, forming a complete technical chain from adversarial behavior to structured solutions.

[0032] Step S1 involves uniformly collecting the global state, local observations of each entity, behavioral commands, immediate gains, resource consumption, and constraint triggering status for each round of adversarial simulation and at each moment. This data is then stored in a trajectory data container according to the adversarial rounds and time sequence. Simultaneously, during the data collection phase, derived relationship features such as inter-entity distances and resource node loads are calculated, allowing subsequent abstract processing to directly obtain the required information based on a single traversal. In this way, the scattered data generated during the simulation is organized into a clearly structured and indexable set of trajectories, laying a stable foundation for subsequent processing.

[0033] Step S2 involves automatically segmenting the timeline based on trajectory data, considering changes in revenue, spatial aggregation, and constraint events, creating a set of stages with clearly defined start and end times. Within each stage, entities are used as nodes, and a collaborative relationship graph is constructed based on spatial proximity and behavioral category consistency. Collaborative groups are then extracted through connectivity and edge density analysis. Dominant behaviors and constraint triggering types are then statistically analyzed using combinations of stages and collaborative groups to generate a time-ordered event sequence. This process elevates the fine-grained trajectory at the continuous time step level to a mid-level event representation with stage structure, collaborative structure, and semantic labels, facilitating subsequent structured processing according to the proposed scheme.

[0034] Step S3: For each round of adversarial event sequence, an encoding network is used to map individual events and their time positions, group sizes, behavior categories, and constraint types into fixed-length vectors. Based on this, a scene vector describing the overall characteristics of the entire round of adversarial action is obtained through sequential encoding. A template library containing multiple scheme structures is pre-built, with each template describing time structure, role structure, and resource structure. A set of candidate templates is selected by calculating the similarity between the scene vector and each template vector. Within the candidate template range, several candidate correspondences between events and template slots are generated according to time order and role settings. An evaluation network is introduced, taking scene features, template features, mapping structure features, and rule description vectors as input, to score each candidate mapping. The mapping results from the selected events, which are compatible with the constraints and have the best overall quality, to time slots, role slots, and resource slots are chosen. This ensures that the event patterns automatically generated during the adversarial process are formally unified with the predefined scheme structures.

[0035] Step S4 involves constructing a structured scheme data object based on the selected template and mapping results. This object is then populated with key time intervals, the correspondence between roles and collaborative groups, and resource instruction entries. Consistency and rule compliance checks are performed across the time, entity, and resource dimensions. After constructing the structured scheme, the time, role, and resource fields are converted into a time-sorted sequence of script instructions. Each instruction records its type, target object, parameter set, and triggering conditions, while retaining a traceability marker from the original event. Through this process, the implicit collaborative behaviors and resource utilization processes in multi-entity adversarial simulations are solidified into a savable, replayable, and integrable structured scheme and execution script. This facilitates high-frequency simulations, scheme library construction, and cross-platform invocation, significantly improving the overall effectiveness of virtual adversarial simulations in scheme generation and application.

[0036] In virtual multi-entity adversarial simulation scenarios, multi-agent reinforcement learning generates state, observation, and action data that evolve continuously over time steps. If a unified, indexable information set containing derivational relationships is not formed in the initial stage, the accuracy and repeatability of subsequent stage division, cooperative group extraction, and event abstraction cannot be guaranteed. Therefore, it is necessary to first construct an environment model, entity set, and trajectory buffer through step S1 to provide a stable data foundation for subsequent steps.

[0037] Detailed implementation of step S1:

[0038] In step S1, the environment model is first constructed. Environmental Model The virtual space scope, time step, resource node set, and constraint rule set are clearly defined. Each constraint rule limits resource limits, prohibited spatial areas, or prohibited combinations of behaviors. (Based on an environment model) Define entity collection Each entity Each entity corresponds to an independent acting agent and has entity type, motion capability parameters, resource capability parameters, and constraint attribute markers. Predefine the local observation vector space and the action vector space. The local observation vector is used to describe the information that an entity can obtain at a certain time step, and the action vector is used to describe the decision output of the entity at a time step.

[0039] Construct a set of time steps , where integer This indicates the number of time steps in a single adversarial simulation. For any given time step... Define the global state vector Global state vector It is composed of a set of global environment state vectors and a set of entity state vectors: the global environment state vectors contain the global resource occupancy and global constraint states; the entity state vector set... Each vector in Describe entity At time step The location, velocity, remaining resources, and internal working status are defined separately. For ease of later reference, a separate position vector is defined. Describe entity Coordinates in virtual space.

[0040] In a multi-agent reinforcement learning framework, for each entity Define local observation vector With action vectors Local observation vector From the training process at time step Action vectors are generated based on the visible range and the sensing model. Based on the entity policy network Calculated and in the environmental model Mid-drive entity Complete one action. For each time step... Instant reward values ​​are also generated. Here, in a single round of simulation, the instant reward values ​​of all entities are summed to obtain the total reward scalar. ,in To counter round indexing.

[0041] To store all data from multiple simulation rounds, a trajectory buffer is constructed. Track buffer A two-dimensional index structure is used, the first dimension... Represents the adversarial round index, the second dimension. This represents the time step index. Each record unit. Includes the following: global state vector Local observation vector set Action vector set The set of instantaneous reward scalars for each entity, and the total reward scalar. And the constraint event tag set. The constraint event tag set is recorded at time step. All constraint rules triggered above are used to reflect situations such as resource overruns, entry into restricted areas, or other rule triggering.

[0042] Fill the trajectory buffer During the process, some derived features are calculated on the fly and stored in the recording unit to reduce subsequent repetitive calculations. This applies to any different entity. With entity At time step Spatial relationships, introducing distance scalar :

[0043]

[0044] in Represents the Euclidean norm. Distance scalar. This describes the instantaneous spatial proximity between entities. In the resource dimension, a resource occupancy vector is constructed, recording the usage and remaining capacity of each resource node at each time step. These derived features, along with the original state, observations, and actions, are recorded together. In this process, complete recording units of information are formed. Trajectory buffer. After the index is built, index each round. Perform data integrity checks and time alignment. Data integrity checks confirm that each time step record unit contains a global state vector, a set of observation vectors, and a set of action vectors. For any missing content caused by external factors, placeholder vectors are used to fill in the gaps to maintain dimensional consistency. Time alignment ensures that all rounds use the same set of time steps. This is to prevent timeline misalignment during subsequent vertical comparisons and horizontal aggregations.

[0045] Through step S1, the environmental model Entity set Time step set With trajectory buffer Establish a unified data foundation and trajectory buffer. The data not only includes the state, observation, action, and reward information for each time step, but also derived features such as inter-entity distance and resource consumption, providing complete input for time segmentation, relationship graph construction, and collaborative group extraction in step S2, so that subsequent abstraction does not depend on additional data sources and maintains label consistency.

[0046] Track buffer The virtual multi-entity adversarial process is recorded as a fine-grained time-step sequence, but structured scheme generation focuses more on phased behavioral patterns and inter-entity collaborative relationships. Without a reasonable method to extract phase intervals, collaborative groups, and event sequences from continuous time steps, template matching and scheme generation cannot accurately express when, which entities, and how they collaborate to produce key behaviors. Therefore, step S2 needs to be performed in the trajectory buffer. Based on this, time segmentation, relationship diagram construction, and event abstraction are completed.

[0047] Detailed implementation of step S2:

[0048] In step S2, index each adversarial round. Process the corresponding trajectories sequentially First, construct the time step determination function for phase division. The total reward scalar is already known. The distance between entities and the number of constraint events require the construction of stage boundary determination indicators.

[0049] Define the average distance scalar :

[0050]

[0051] in For the number of entities, The entity calculated in step S1 With entity At time step The distance. Average distance scalar. Indicates round At time step The average spatial density of all entity pairs. For the constraint event label set, the cardinality is calculated to obtain the constraint event count. This is used to reflect the frequency of rule triggering.

[0052] Construct a time piecewise function based on the above quantities.

[0053]

[0054] Where the threshold and These are pre-defined based on the reward magnitude and spatial scale, respectively. (Function) Returning 1 indicates a time step. With time step There are stage boundaries. For all calculate Collect to satisfy Time step index As a boundary time step. Combining the start time step 0 and the end time step. For rounds Construct several time intervals ,in This is the interval number.

[0055] After obtaining the set of time intervals, index each round. With interval number Construct an entity relationship graph. The graph uses nodes to represent entities and edges to represent the strength of collaborative behavior between entities. For the time interval... For any different entities Calculate the scalar of synergy strength :

[0056]

[0057] in This is an indicator function; it takes the value 1 if the condition is true and 0 if the condition is false. This is a distance threshold used to determine spatial proximity. Action vector Corresponding behavior category number. Collaboration strength scalar. It is composed of the sum of spatial proximity counts and behavioral category consistency counts, and is used to measure the entity. With entity The degree of coordination within the time interval.

[0058] Constructing an adjacency matrix based on cooperative strength :

[0059]

[0060] Where the threshold This represents the minimum frequency of effective cooperative behaviors. It is achieved through an adjacency matrix. An undirected graph is formed within the time interval. All connected components are then obtained using either depth-first search or breadth-first search. The set of nodes in each connected component is used as an initial cooperative group. For each initial cooperative group... Calculate edge density :

[0061]

[0062] in The number of edges within the group. The number of nodes in the group. When the edge density... Below the preset density threshold If necessary, further graph partitioning (e.g., using spectral clustering) is performed on the connected component to obtain more compact subgroups, and the set of nodes in each subgroup is used as the final cooperative group; otherwise, the initial cooperative group is directly used as the final cooperative group. Through the above process, in each time interval... Internally formed collaborative group set .

[0063] After the set of collaborative groups is generated, abstract events are generated for each time interval and the combination of collaborative groups. Round indexing. Time interval numbering With collaborative group identifier From the trajectory buffer Extracting groups within a time interval The system records entity actions, counts the frequency of different behavior categories within the group within a given time interval, and selects the most frequent behavior category number as the dominant behavior category label; simultaneously, it counts the set of constraint event categories related to group members. The system utilizes the start time of the time interval. End time of time interval Collaborative group entity set The event record unit is constructed by labeling the dominant behavior category and the set of constraint event categories. For rounds Event record units generated by combining all time intervals and groups are sorted in ascending order of the start time of the time interval to form a round. event sequence .

[0064] Through step S2, from the trajectory buffer Starting with fine-grained time-step data, using time-piece function Divide the phase intervals and assess the strength of cooperation. Adjacency matrix and edge density Extracting the set of collaborative groups And generate event sequences based on the combination of time intervals and collaborative groups. These events and group tags form a mid-level semantic representation in terms of time, entity, and behavior category, providing a structured foundation for scene feature encoding and template matching in step S3.

[0065] Event Sequence With collaborative group sets Trajectory information has already been compressed at the stage and collaboration level, but structured solution generation requires mapping these time-ordered events to a limited number of solution templates. Within each template, the mapping between events and time slots, and between role slots and resource slots, needs to be established to generate executable solutions. Without a clear encoding method, template matching method, and mapping quality evaluation mechanism, conflicts or inconsistencies with constraint rules can easily arise in the event-to-template mapping. Therefore, step S3 needs to complete event and scene encoding, solution template matching, and mapping optimization.

[0066] Detailed implementation of step S3:

[0067] In step S3, the index for each round is first... event sequence Perform event-level coding. Each event... Includes round index Time interval numbering Collaborative group identifier Start time of time interval End time of time interval Collaborative group entity set The system includes the dominant behavior category number and the set of constraint event categories. Basic numerical characteristics are constructed for each event, including the length of the time interval. Normalization start time Collaborative group size And a vector formed by embedding or multi-hot encoding the set of dominant behavior categories and constraint event categories. These features are then concatenated into a one-dimensional vector. .

[0068] Define event encoding function :

[0069]

[0070] in and Given a trainable parameter matrix and a bias vector, For element-wise nonlinear functions, such as ReLU. Event encoding function. Map each event as a fixed-length vector. This facilitates subsequent sequential processing while preserving key semantic information.

[0071] indexes for the same round event feature sequence Perform scene-level coding and define scene coding functions. :

[0072]

[0073] in This represents the scene feature vector. The scene encoding function can be implemented using a simple loop structure:

[0074]

[0075] in , and For trainable parameters, It is a real weight matrix from the hidden vector at the previous time step to the current hidden vector, with the number of rows and columns equal to the dimension of the scene feature vector; It is a real weight matrix from the current event feature vector to the current hidden vector, with the number of rows equal to the dimension of the scene feature vector and the number of columns equal to the dimension of the event feature vector; This is a real-valued bias vector applied to the current hidden vector, with a length equal to the dimension of the scene feature vector; , and The gradient is obtained during the training phase using sample data and gradient update algorithms, and remains fixed during the runtime phase. It is a hyperbolic tangent function. Through cumulative processing of event sequences, the scene feature vector... Expressing event sequence, duration patterns, and behavior category transfer patterns.

[0076] After the scene feature vectors are constructed, the solution template library is introduced. Each template It corresponds to a predefined scheme structure, containing template feature vectors. Time slot set Character slot collection With resource slot set It also includes a set of internal constraints for the template, describing the slot order, resource capacity limits, and role compatibility conditions. Template feature vector. It is obtained by the template designer based on the number of slots inside the template, the expected number of stages, the combination of typical behavior categories, and the constraint type coding.

[0077] To select candidate templates, a similarity function is introduced. :

[0078]

[0079] in It is the Euclidean norm. Similarity scalar. A similarity threshold is set between -1 and 1, with higher values ​​indicating a closer resemblance between the scene features and the template features. Construct a candidate template index set :

[0080]

[0081] If the set is empty, several templates with the highest similarity can be selected as fallback candidates.

[0082] In the candidate template index set Once determined, index each round. With each template index Construct a candidate mapping set of events to slots Template time slot set Sort according to the internal time sequence of the template. Event sequence By start time Sort. First, construct the basic mapping. Scan the event sequence sequentially, for each event Based on the event's start time and duration, the event is assigned to a specific time slot that satisfies the template's time interval order. This ensures that the time sequence cannot be reversed. Then, several candidate mappings are constructed near the base mapping. New mapping schemes are generated by adjusting the time slots of a small number of events, changing the character slot selection, and changing the resource slot matching method. Each mapping scheme is denoted as... For each candidate mapping, immediately check the temporal order constraints, resource capacity constraints, and role compatibility constraints in the template's internal constraint set. Mappings that violate any hard constraints are directly eliminated, ensuring the set... Only structurally feasible mapping schemes are retained.

[0083] To evaluate the quality of different mapping schemes, a rule-based description vector is introduced. From environmental models The system collects rule entries such as resource limits, spatial prohibition zones, and behavioral combination prohibition conditions, and encodes them into vectors in a fixed format. Each round of indexing... Template Index With candidate mapping elements Construct the evaluation input vector The input vector contains scene feature vectors. Template feature vector Mapping structure encoding vector With the rule description vector Mapping structure encoding vector This can include metrics such as time slot coverage ratio, character slot coverage ratio, resource slot usage, and rule violation count. These sub-vectors are concatenated in a fixed order to form... .

[0084] Define the evaluation network:

[0085]

[0086] in , , , , , For trainable parameters, each represents a parameter array in the evaluation network, namely, the real weight matrix from the input vector to the first hidden layer, the real weight matrix from the first hidden layer to the second hidden layer, the real weight vector from the second hidden layer to the score output, and the real bias vectors corresponding to the two hidden layers and the output layer. The dimensions are uniquely determined by the input feature dimension and the two hidden layer dimensions, and the specific values ​​are determined through the conventional neural network training process. For non-linear functions, such as ReLU. Output a score scalar. The mapping quality is comprehensively reflected in three aspects: the degree of rule satisfaction, the degree of event semantic preservation, and the degree of template structure utilization. For each template index... ,from Selecting a scoring scalar The mapping that is the largest and not less than the quality threshold is used as the template. The corresponding optimal mapping, and then in all Select the highest scorer from the optimal mapping to determine the round. The final template index and event-to-slot mapping results.

[0087] Through step S3, the event sequence Event-level encoding and scene-level encoding are used to construct scene feature vectors. In the solution template library Using similarity functions Construct a candidate template set and through candidate mapping set Rule description vector With evaluation network The optimal template index and the correspondence between events and slots are output, thereby forming a computable connection between virtual multi-entity adversarial behavior and abstract scheme templates, providing a complete mapping basis for generating structured scheme data objects and intermediate scripts in step S4.

[0088] The event-to-slot mapping results and the final template index have provided an abstract structural framework for virtual multi-entity adversarial behavior. However, this abstract structure still needs to be transformed into a structured scheme and script description that can be stored long-term, visualized, replayed, and parsed and executed by the simulation platform. Without a unified scheme data object and intermediate script statement definition, the correspondence between template slots and real events will be difficult to integrate with the actual execution environment. Step S4 aims to complete the generation of the structured scheme and intermediate scripts while maintaining the integrity of the three types of information: time, role, and resources.

[0089] Detailed implementation of step S4:

[0090] In step S4, indexing for each round Using the final template index and event-to-slot mapping results output in step S3, in the scheme template library Selected template Read template Predefined time slot set Character slot collection With resource slot set This initializes the structured schema data object. The structured schema data object defines top-level fields including the round index. Template Index The middle layer fields are divided into a set of time fields, a set of role fields, and a set of resource instruction fields; the bottom layer fields are placeholders for specific parameters corresponding to each slot instance.

[0091] The time field set is used to store the correspondence between template time slots and actual time intervals. The time slot set... For each slot, the subset of events mapped to that slot is found based on the event-to-slot mapping result. Each event subset contains several events. Each event has a start time of the time interval. End time of the time interval For a subset of events corresponding to the same time slot, the minimum start time of all events is taken as the actual start time of the slot, and the maximum end time of all events is taken as the actual end time of the slot. Simultaneously, the time order of each event within the slot is recorded for restoring fine-grained order during subsequent script execution. Each element in the time field set records the time slot identifier, the actual time interval, and the event list index set, forming a reference relationship with the stage interval number.

[0092] The role field collection is used to store the mapping between template role slots and collaborative group entities and task information. (The last part, "for the role slot collection," appears to be an unrelated fragment and is omitted from the translation.) For each role slot, retrieve all events mapped to that role slot based on the event-to-slot mapping result, read the collaboration group identifiers from these events, and then retrieve the collaboration group set. Retrieving entity collections The role field contains information related to the entity type. It records the role slot identifier, a list of entity IDs, entity type statistics, a summary of the dominant behavior category, and a set of time slot identifiers associated with the role. Cross-referencing is established between the role field and the time field, allowing tracking of the role's activity range within each time slot.

[0093] The resource instruction field set is used to store the mapping between template resource slots and specific resource usage actions. For the resource slot set... For each resource slot, select the set of events mapped to that resource slot based on the event-to-slot mapping result, and for each event, add it to the trajectory buffer. The system retrieves records involving resource usage within a specified time interval, extracting parameters such as resource type, resource quantity, resource source entity or resource node, and target entity or target region number. These parameters are then encapsulated into resource instruction entries. Resource actions with consecutive time intervals and identical parameters within the same resource slot are merged into a single resource instruction entry. Resource actions with overlapping time intervals but different targets are retained as multiple instruction entries, with references to the role and time fields recorded in the resource field, indicating the role involved and the time range of the resource action.

[0094] After the structured data objects are populated, consistency checks and normalization are performed. First, the consistency of entity allocation between the time field set and the role field set is checked. For example, whether the same entity is assigned to a mutually exclusive role slot in the same time slot. If a conflict is found, it is handled according to template constraints and rule description vectors. Adjust role assignments or mark them as requiring manual review. Next, check the resource instruction field set and rule description vector. Ensure consistency between capacity and regions. If total resource usage exceeds the agreed limit or the affected region intersects with a prohibited region, adjust resource instructions or add a flag. Recheck all ID references within the structured data object to ensure there are no missing entity IDs, time slot IDs, or event indexes. After validation, standardize field naming and sorting to ensure the structured data object conforms to the preset format.

[0095] After the structured data objects are prepared, an intermediate script sequence is constructed for direct execution by the simulation platform. The intermediate script is represented by an ordered set of script statements, each defined as a quadruple:

[0096]

[0097] in Indicates the type of script statement, such as a role assignment instruction or a resource usage instruction; The target entity ID or role slot ID; This is a parameter vector, containing resource quantity, behavior category number, target area number, etc. This is a time-triggered condition, including the associated time slot number or specific start and end times. During script generation, the time field, role field, and resource instruction field are read from the structured scheme data object. The content is converted into a set of script statements, and an event tracing marker is appended to each script statement, indicating the event record unit index and stage interval number associated with that instruction. All script statements are arranged in chronological order to form a script sequence, which can be indexed by the simulation platform at the corresponding round. The above parsing and execution enables automatic replay of virtual multi-entity confrontation scenarios.

[0098] In step S4, based on the final template index and event-to-slot mapping results, the time field set, role field set, and resource instruction field set are constructed and populated to complete the generation of the structured scheme data object, and the rule description vector is used. After completing consistency verification and standardization, the scheme field content is finally translated into an intermediate script statement sequence containing four elements: type, objective, parameters and time conditions. This realizes the closed-loop transformation of virtual multi-entity adversarial behavior from abstract template mapping results to executable schemes and replayable scripts, which facilitates unified calling and verification in different simulation platforms and scheduling environments.

[0099] Specifically, the above are merely preferred embodiments of this application and are not intended to limit this application.

[0100] The threshold values ​​for each pre-defined block of speech in this invention can be pre-calibrated through offline simulation testing, or set to fixed values ​​according to on-site operating procedures.

[0101] In the description of this specification, references to terms such as "an embodiment," "example," and "specific example" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0102] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A hierarchical policy learning system for multi-agent adversarial optimization, characterized in that, Including the following steps: S1: Construct an environment model and entity set, use multi-agent reinforcement learning to obtain state, action and constraint information of multiple adversarial time steps, and store it in the trajectory buffer; S2: Based on the trajectory buffer, divide the time interval according to the changes in reward, distance or constraint events, construct an entity relationship graph within the time interval to obtain collaborative groups, and generate event sequences by combining the time intervals and collaborative groups; S3: Obtain scene feature vectors using event sequence encoding, match the scene feature vectors with templates in the solution template library to select target templates, and determine the mapping results from events to time slots, role slots, and resource slots by combining the evaluation network with the rule description vectors; S4: Generate structured scheme data based on the target template and its event-to-slot mapping results, perform consistency checks on the structured scheme data according to the rule description vector, and convert the checked structured scheme into an intermediate script instruction sequence sorted by time.

2. The hierarchical policy learning system for multi-agent adversarial optimization according to claim 1, characterized in that: Step S1 includes collecting global state data, local observation data of each entity, action commands of each entity, instant reward data of each entity, constraint event markers, and resource usage information at each time step during each round of adversarial simulation. The above data are then combined into recording units according to the adversarial round index and time step index and stored sequentially in the trajectory buffer.

3. The hierarchical policy learning system for multi-agent adversarial optimization according to claim 2, characterized in that: Step S2 includes: based on the records of each time step in the trajectory buffer, calculating the reward sequence reflecting the overall profit change, the average distance sequence reflecting the degree of aggregation of entity space, and the constraint event count sequence reflecting the rule triggering situation according to the adversarial rounds; automatically determining the start and end time of the stage based on the judgment result that the change amplitude of adjacent time steps in at least one type of sequence exceeds a preset threshold, and obtaining a set of time intervals with clear start and end times.

4. The hierarchical policy learning system for multi-agent adversarial optimization according to claim 3, characterized in that: Step S2 further includes, within each time interval, using each entity in the entity set as a node, constructing an entity relationship graph based on the spatial proximity of entities within the time interval and the consistency of behavior categories, obtaining a set of cooperative groups by dividing the entity relationship graph into connected components and determining edge density, and generating event records on the combination of each time interval and each cooperative group. The event records contain at least time interval information, cooperative group identifier, dominant behavior category, and constraint event category, thereby forming an event sequence arranged in chronological order.

5. The hierarchical policy learning system for multi-agent adversarial optimization according to claim 4, characterized in that: Step S3 includes encoding each event in the event sequence into a fixed-length event feature vector according to the time interval length, relative position in the whole round of confrontation, collaborative group size, dominant behavior category, and constraint event category information, and sequentially encoding the event feature vector according to the chronological order of the events to obtain a scene feature vector uniquely corresponding to each round of confrontation. The scene feature vector is used to match the template feature vector in the scheme template library.

6. The hierarchical policy learning system for multi-agent adversarial optimization according to claim 5, characterized in that: The solution template library includes multiple preset templates. Each template includes a template feature vector that matches the scene feature vector, a set of time slots describing the key time structure, a set of role slots describing roles and collaborative relationships, and a set of resource slots describing resource configuration behavior. Within the template, formal constraint information such as order constraints, resource capacity constraints, and role compatibility constraints are given in advance for the time slots, role slots, and resource slots.

7. The hierarchical policy learning system for multi-agent adversarial optimization according to claim 6, characterized in that: Step S3 further includes calculating the similarity between the scene feature vector and each template feature vector, selecting templates whose similarity meets the preset conditions to form a candidate template set, generating multiple candidate mappings from events to time slots, role slots and resource slots for each template in the candidate template set, inputting the scene feature vector, template feature vector, candidate mapping structure features and rule description vector into the evaluation network to obtain a score value used to distinguish the merits of different candidate mappings, and selecting the candidate mapping with the highest score as the mapping result from events to slots.

8. The hierarchical policy learning system for multi-agent adversarial optimization according to claim 7, characterized in that: Step S4 includes, after selecting the target template, according to the event-to-time slot mapping result, merging the start time and end time of multiple events mapped to the same time slot into the time interval information in the time field set, and maintaining the reference relationship with the event sequence; at the same time, according to the event-to-role slot mapping result and the collaboration group set, filling the entity identifier and its type information corresponding to each role slot into the role field set, and recording the association between the role and the time field.

9. The hierarchical policy learning system for multi-agent adversarial optimization according to claim 8, characterized in that: Step S4 also includes extracting the resource type, resource quantity, resource source object, and resource target object corresponding to each resource slot based on the event-to-resource slot mapping result and the resource usage record in the trajectory buffer, organizing them into resource instruction entries in the resource instruction field set, merging resource instructions with continuous time and consistent parameters, retaining resource instructions with overlapping time but different target objects, and recording the corresponding time field reference information and role field reference information in each resource instruction entry.

10. The hierarchical policy learning system for multi-agent adversarial optimization according to claim 9, characterized in that: Step S4 also includes performing a consistency check on the time field set, role field set, and resource instruction field set in the structured scheme data with the rule description vector. After confirming that there are no contradictory relationships between the fields that violate the time order constraint, resource capacity constraint, and role compatibility constraint, the structured scheme data is sequentially converted into a script instruction sequence ordered by time. Each script instruction includes an instruction type, target identifier, parameter set, and time trigger condition, and is attached with a corresponding event record and a traceability mark for the time interval.

Images