Diversified interactive agent construction system based on offline learning

Abstract

The invention relates to the technical field of agent construction, and discloses a diversified interactive agent construction system based on offline learning. The system comprises a data preprocessing and analysis module, a behavior normal form extraction module, an interaction sequence enhancement module, an offline strategy deduction module and an agent configuration generation module. The system carries out time sequence alignment and cleaning on original interaction logs, a behavior instruction stream and an environment feedback stream are separated, and an interaction normal form template library is generated through behavior mode distillation. And performing pattern matching and labeling on the interaction sequence based on the library to generate an enhanced sequence with a normal form label. And training an offline strategy deduction network by using the sequence, so that a diversified construction strategy set can be directly output for a new scene, and finally, an executable agent configuration file is generated. The system can automatically extract a high-level semantic behavior normal form from historical data and generate multiple differential strategies at a time, so that automatic and diversified construction of agent behaviors is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent construction technology, specifically to a diversified interactive intelligent agent construction system based on offline learning. Background Technology

[0002] When building interactive intelligent agents, relying on automatically learning effective behavioral patterns from massive offline interaction logs is a common approach. Existing technologies typically employ traditional data mining and statistical learning methods to cluster or analyze behavioral sequences to identify frequently occurring patterns. The patterns extracted by these methods are mostly low-level, statistically significant sequence fragments, lacking high-level semantic abstraction of the relationship between behavioral intent and contextual state, making it difficult to form structured knowledge that can directly drive the agent's decision-making. Meanwhile, existing methods based on offline reinforcement learning or imitation learning usually aim to train a single policy model designed to maximize cumulative reward or fit expert behavior. When faced with new interaction scenarios, these models often only output a behavioral policy considered "optimal" or "closest," resulting in a single and deterministic output.

[0003] This single-policy output mode is insufficient for application scenarios requiring agents to possess diverse behavioral capabilities. Generating a set of agent policies containing multiple feasible and different behavioral logics directly from offline logs, rather than a single policy, is a problem that current technologies have not effectively solved. Furthermore, how to automatically summarize and extract the complex behavioral patterns implicit in the original logs into interpretable and reusable standardized paradigm templates, and then use these templates to semantically enhance interaction sequences to support the generation of diverse policies, is also a gap in existing solutions. Summary of the Invention

[0004] The purpose of this invention is to provide a system for constructing diverse interactive intelligent agents based on offline learning, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a system for constructing diverse interactive intelligent agents based on offline learning, the system comprising: The data preprocessing and parsing module is configured to receive the raw interaction log data stream stored offline, perform cross-log source time alignment and cleaning processing on the raw interaction log data stream to obtain a standardized interaction event sequence, and separate the agent behavior instruction stream and the environment state feedback stream from the standardized interaction event sequence. The behavior paradigm extraction module is configured to input the agent's behavior instruction stream and the environmental state feedback stream into the behavior pattern distillation pipeline to generate an interaction paradigm template library containing multiple behavior paradigms. The interaction sequence enhancement module is configured to perform pattern matching and annotation processing on the standardized interaction event sequence based on the interaction paradigm template library, and generate an enhanced interaction sequence with paradigm tags. The offline policy inference module is configured to train the offline policy inference network using the enhanced interaction sequence with paradigm labels to generate a policy inference network model, and input the new interaction event sequence into the policy inference network model to output a policy set for diverse agents. The agent configuration generation module is configured to generate executable agent configuration files based on the diverse agent construction strategy set.

[0006] Preferably, the step of performing cross-log-source time-series alignment and cleaning processing on the original interaction log data stream to obtain a standardized interaction event sequence includes: Raw log fragments are extracted from different log sources. Each raw log fragment contains a timestamp, an agent identifier, and raw event description text. Based on the timestamps and agent identifiers, the original log fragments from different log sources are cross-correlated and sorted to form an initial event chain with agents as the dimension. Noise words are filtered and domain terminology is standardized and replaced for each original event description text in the initial event chain to obtain a cleaned event description. Dependency parsing was performed on the description of the purification event to identify the core behavioral actions, the target objects, and environmental modifiers. The identified core behaviors, targets, and environmental modifiers are reorganized according to a preset event structure template to generate structured event units. All structured event units are connected in chronological order to form the standardized interactive event sequence.

[0007] Preferably, separating the agent behavior instruction stream and the environment state feedback stream from the standardized interaction event sequence includes: For each structured event unit in the standardized interaction event sequence, the subject type is determined to distinguish between active behavior units initiated by the agent and state change units triggered by the environment. Extract all active behavior units according to their timestamps to form a preliminary sequence of behavior instructions; Extract all state change units according to their timestamps to form a preliminary state feedback sequence; The initial action instruction sequence is checked for instruction integrity, and the broken instruction chain caused by log loss is detected and filled to generate a continuous stream of agent action instructions. The initial state feedback sequence is checked for state continuity, and abnormal jumps in state values ​​over time are smoothed to generate a continuous environmental state feedback stream.

[0008] Preferably, the step of inputting the agent's behavior instruction stream and the environmental state feedback stream into the behavior pattern distillation pipeline to generate an interaction paradigm template library containing multiple behavior paradigms includes: The agent's behavioral instruction stream and the environment's state feedback stream are synchronously sliced ​​in the time dimension to obtain multiple aligned time segments; Within each aligned time segment, the agent's behavioral instructions are paired with the subsequent environmental state feedback to form an instruction-feedback pair; Cluster analysis is performed on all instruction feedback pairs, and the instruction feedback pairs are classified into different candidate pattern clusters based on the semantic similarity of the behavioral instructions and the similarity of the change patterns of the state feedback. Select typical instruction feedback pairs from each candidate pattern cluster, and abstract the semantic framework of their behavioral instructions and the changing patterns of their state feedback to generate an initial paradigm template. The initial paradigm templates are evaluated for generalization ability. Initial paradigm templates with similar semantic frameworks and overlapping change patterns are merged. Initial paradigm templates with too few instruction feedback pairs are eliminated. Finally, the interaction paradigm template library is formed. Each interaction paradigm template in the interaction paradigm template library contains an abstract behavior instruction template and an expected state feedback pattern.

[0009] Preferably, the step of performing pattern matching and annotation processing on the standardized interaction event sequence based on the interaction paradigm template library to generate an enhanced interaction sequence with paradigm tags includes: The semantic matching degree of each structured event unit in the standardized interactive event sequence is calculated with each abstract behavior instruction template in the interactive paradigm template library; Assign a paradigm tag corresponding to the abstract behavior instruction template with the highest semantic matching degree to each structured event unit, and use it as its behavior paradigm tag; Based on the behavioral paradigm labels, retrieve the corresponding expected state feedback patterns from the interaction paradigm template library; The actual environmental state changes immediately following the current structured event unit in the standardized interactive event sequence are compared with the retrieved expected state feedback pattern, and the conformity is calculated. The behavioral paradigm labels and the conformity level are used as annotation information and attached to the corresponding structured event units to form the enhanced interaction sequence with paradigm labels.

[0010] Preferably, the step of training the offline policy inference network using the enhanced interaction sequence with paradigm labels to generate a policy inference network model includes: Continuous segments are extracted from the enhanced interaction sequence with paradigm labels as training samples. Each training sample contains an initial event, a series of intermediate events, and an ending event. The event sequence in the training samples is converted into a multi-channel input tensor consisting of an event vector, a paradigm label vector, and a conformity vector. An offline policy inference network with multi-layered recurrence and attention mechanisms is constructed, comprising an event encoding layer, a pattern memory layer, and a policy decoding layer. The offline policy inference network is trained using the multi-channel input tensor, enabling the event encoding layer to learn to extract features from event vectors, the pattern memory layer to learn to activate different behavioral pattern memory units based on the paradigm label vector and conformity vector, and the policy decoding layer to learn to predict the next optimal behavioral paradigm and its corresponding operation instruction based on the encoded features and activated pattern memory units. When the prediction accuracy of the offline policy inference network on the validation set reaches a preset threshold, the trained policy inference network model is obtained.

[0011] Preferably, the step of inputting the new sequence of interactive events into the policy inference network model and outputting a diverse set of agent-constructed policies includes: The same standardization and paradigm labeling process is applied to the new sequence of interactive events to obtain a new enhanced sequence of interactive events with paradigm labels. The current segment of the new enhanced interaction sequence with paradigm labels is input into the event encoding layer of the policy inference network model to obtain the current state encoding; The current state encoding is input into the pattern memory layer of the policy inference network model, and the paradigm label vector and conformity vector of the current segment are input at the same time to activate multiple related behavioral pattern memory units. In the policy decoding layer of the policy inference network model, based on the current state encoding and all activated behavior pattern memory units, multiple candidate subsequent behavior paradigms and corresponding operation instructions are generated, and each candidate subsequent behavior paradigm is accompanied by a diversity score. Based on the diversity score, a preset number of candidate subsequent behavior paradigms and their operation instructions are selected from high to low to form a strategy option. For multiple decision points in the new interactive event sequence, the steps from inputting the current segment to generating policy options are repeated to generate a corresponding policy option for each decision point. The set of policy options for all decision points constitutes the diverse agent construction policy set.

[0012] Preferably, the step of generating an executable agent configuration file based on the diverse agent construction strategy set includes: Analyze each policy option in the diverse intelligent agent construction policy set, and extract the operation instruction sequence and the corresponding behavior paradigm identifier; Based on the behavior paradigm identifier, retrieve the corresponding abstract behavior instruction template and expected state feedback mode from the interaction paradigm template library; The specific operation instruction sequence in the strategy options is merged with the invoked abstract behavior instruction template to generate a specific, parameterized behavior execution script; Associate each behavior execution script with its corresponding expected state feedback pattern, which serves as the expected judgment condition for whether the behavior execution script is successful or not; All action execution scripts and their expected judgment conditions are arranged in the order of decision points of the policy options and encapsulated into an executable agent configuration file that conforms to the target platform specifications.

[0013] Preferably, after generating an executable agent configuration file, the system further executes a policy verification and optimization loop, including: The executable agent configuration file is loaded into the simulation test environment to drive the simulated agent to run; Record all simulated interaction events triggered by the simulated agent during operation to form a simulated interaction log; Extract key performance indicator sequences from the simulated interaction logs, compare the key performance indicator sequences with the preset target performance curve, and calculate the strategy execution deviation. If the deviation of the strategy execution exceeds the tolerance threshold, a correction data packet is generated based on the simulated interaction log; The corrected data packet is used as a new training sample and injected into the enhanced interaction sequence with paradigm labels to trigger incremental training of the policy inference network model and update the parameters of the policy inference network model.

[0014] Preferably, the system further includes a dynamic maintenance mechanism for an interaction paradigm template library, including: Continuously monitor newly generated standardized interaction event sequences; The newly generated standardized sequence of interactive events is matched with existing templates in the interactive paradigm template library to identify abnormal interactive fragments that cannot be effectively matched by existing templates. Clustering and abstracting the abnormal interaction fragments to generate candidate new paradigm templates; Evaluate the frequency of occurrence and potential utility value of the candidate new paradigm templates in historical interaction data; The candidate new paradigm templates that pass the evaluation will be incorporated into the interaction paradigm template library and marked as new templates for use in subsequent pattern matching and annotation processing.

[0015] Compared with the prior art, the beneficial effects of the present invention are: The behavior paradigm extraction module uses a behavior pattern distillation pipeline to comprehensively analyze the separated agent behavior command flow and environmental state feedback flow. This process can identify the temporal causal and logical relationships between commands and state feedback, automatically clustering and solidifying combinations of behavioral units with clear semantics, forming a structured interaction paradigm template library. This endows the original low-level interaction events with high-level behavioral semantic labels, transforming the unstructured log stream into a semantic sequence composed of standardized paradigms, providing a foundation for deep understanding and combinatorial innovation of strategies. The system obtains not simple statistical frequency patterns, but interpretable building blocks that can be directly mapped to the agent's decision-making logic.

[0016] The offline policy inference module trains a dedicated offline policy inference network, which uses augmented interaction sequences with paradigm labels as training samples. Its model design goal differs from traditional offline reinforcement learning policy networks; it is trained to infer multiple complete policy paths that differ in behavioral paradigm combinations, temporal arrangements, or objective emphases in response to new input interaction event sequences. This network directly outputs a diverse set of agent-constructed policies, where each policy is based on a learned paradigm library but differs in its specific application logic. This achieves a direct mapping from offline data to a multi-policy blueprint, ensuring behavioral diversity in the final agent ensemble during the policy generation phase and avoiding repeated training or tedious manual adjustments required to achieve diversity. Attached Figure Description

[0017] Figure 1 This is a sequence diagram of the system for constructing diverse interactive intelligent agents based on offline learning as described in this invention. Figure 2 A flowchart for cross-log source time-series alignment and cleaning processing; Figure 3 A flowchart generated for the behavioral pattern distillation and interaction paradigm template library; Figure 4 A scatter plot of state attribute matching rate and conformity. Figure 5 A two-column comparison chart of behavioral paradigms. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 This invention provides a system for constructing diverse interactive intelligent agents based on offline learning. The system includes: a data preprocessing and parsing module that receives raw interaction log data streams stored offline, performs cross-login time-series alignment and cleaning on the raw interaction log data streams to obtain standardized interaction event sequences, and separates the agent behavior command stream and environment state feedback stream from the standardized interaction event sequences. A behavior paradigm extraction module inputs the agent behavior command stream and environment state feedback stream into a behavior pattern distillation pipeline to generate an interaction paradigm template library containing multiple behavior paradigms. An interaction sequence enhancement module performs pattern matching and annotation processing on the standardized interaction event sequences based on the interaction paradigm template library to generate enhanced interaction sequences with paradigm labels. An offline policy inference module uses the enhanced interaction sequences with paradigm labels to train an offline policy inference network to generate a policy inference network model, and inputs new interaction event sequences into the policy inference network model to output a set of diverse intelligent agent construction strategies. An agent configuration generation module generates executable agent configuration files based on the set of diverse intelligent agent construction strategies.

[0020] Example 1: See Figure 2The data preprocessing and parsing module performs cross-log-source time-series alignment and cleaning on the raw interaction log data stream to obtain a standardized interaction event sequence. Raw log fragments are extracted from different log sources; each fragment contains a timestamp, agent identifier, and raw event description text. Based on the timestamp and agent identifier, the raw log fragments from different sources are cross-linked and sorted to form an initial event chain oriented towards the agent. Noise word filtering and domain terminology standardization replacement are performed on each raw event description text in the initial event chain to obtain a cleaned event description. Dependency parsing is performed on the cleaned event description to identify core actions, target objects, and environmental modifiers. The identified core actions, target objects, and environmental modifiers are reorganized according to a preset event structure template to generate structured event units. All structured event units are connected in chronological order to form a standardized interaction event sequence. Agent behavior command streams and environmental state feedback streams are separated from the standardized interaction event sequence. The subject type of each structured event unit in the standardized interaction event sequence is determined, distinguishing between active behavior units initiated by the agent and state change units triggered by the environment. All active action units are extracted according to their timestamps to form a preliminary action instruction sequence. All state change units are extracted according to their timestamps to form a preliminary state feedback sequence. The preliminary action instruction sequence is checked for instruction integrity, and breaks in the instruction chain caused by log loss are detected and filled to generate a continuous agent action instruction flow. The preliminary state feedback sequence is checked for state continuity, and abnormal jumps in state values ​​over time are smoothed to generate a continuous environmental state feedback flow.

[0021] In practical implementation, the data preprocessing and parsing module of a diversified interactive intelligent agent construction system based on offline learning receives the raw interactive log data stream stored offline. The raw interactive log data stream contains log fragments from different log sources. In the example scenario, the two log sources are a server log source and a client log source. The server log source records environmental state change events, and the client log source records intelligent agent operation command events. The format of the raw log fragments from the server log source is "Time stamp: 2023-10-01T08:00:01, Intelligent agent identifier: Agent_01, Original event description text: Environmental state updated to high danger level". The format of the raw log fragments from the client log source is "Time stamp: 2023-10-01T08:00:00, Intelligent agent identifier: Agent_01, Original event description text: Intelligent agent executes movement command". "Command to move left". The data preprocessing and parsing module extracts raw log fragments from different log sources. Each raw log fragment contains a timestamp, agent identifier, and raw event description text. Based on the timestamp and agent identifier, the raw log fragments from different log sources are cross-correlated and sorted to form an initial event chain with the agent as the dimension. In specific implementation, cross-correlation is completed by matching the same agent identifier with similar timestamps, and sorting is arranged in ascending order of timestamp. An example of the initial event chain is [Timestamp: 2023-10-01T08:00:00, Agent identifier: Agent_01, Raw event description text: Agent executes movement command to move left; Timestamp: 2023-10-01T08:00:01, Agent identifier: Agent_01, Raw event description text: Environment status updated to high danger level].

[0022] In some embodiments, an agent behavior instruction stream and an environment state feedback stream are separated from a standardized interaction event sequence. For each structured event unit in the standardized interaction event sequence, a subject type determination is performed to distinguish between active behavior units initiated by the agent and state change units triggered by the environment. The subject type determination is based on the subject field in the structured event unit; if the subject field is "agent," it is determined to be an active behavior unit; if the subject field is "environment," it is determined to be a state change unit. All active behavior units are extracted according to their timestamps to form a preliminary behavior instruction sequence. An example of a preliminary behavior instruction sequence is [Timestamp: T1, Structured event unit: Subject: Agent; Behavior: Move; Object: None; Modifier: Left]. All state change units are extracted according to their timestamps to form a preliminary state feedback sequence. An example of a preliminary state feedback sequence is [Timestamp: T2, Structured event unit: Subject: Environment; Behavior: State Change; Object: Environment; Modifier: High Danger Level]. The preliminary behavior instruction sequence is checked for instruction integrity to detect and fill in any breaks in the instruction chain caused by lost logs, generating a continuous agent behavior instruction stream.

[0023] Example 2: See Figure 3 The behavior paradigm extraction module inputs the agent's behavior command flow and the environment's state feedback flow into the behavior pattern distillation pipeline, generating an interaction paradigm template library containing multiple behavior paradigms. The agent's behavior command flow and the environment's state feedback flow are synchronously sliced ​​along the time dimension, resulting in multiple aligned time segments. Within each aligned time segment, the agent's behavior command is paired with the immediately following environment state feedback, forming a command-feedback pair. Cluster analysis is performed on all command-feedback pairs, grouping them into different candidate pattern clusters based on the semantic similarity of the behavior command and the similarity of the state feedback change patterns. Typical command-feedback pairs are selected from each candidate pattern cluster, and their semantic framework of the behavior command and the change patterns of the state feedback are abstracted to generate initial paradigm templates. The generalization ability of the initial paradigm templates is evaluated, merging initial paradigm templates with similar semantic frameworks and overlapping change patterns, and eliminating initial paradigm templates with too few command-feedback pairs, ultimately forming the interaction paradigm template library. Each interaction paradigm template in the library contains an abstract behavior command template and an expected state feedback pattern.

[0024] Within each aligned time segment, agent action instructions are paired with immediately following environmental state feedback to form instruction-feedback pairs. In specific implementations, "immediately following" is defined as the environmental state feedback's timestamp being after the agent action instruction's timestamp and within the same time window length ω. Examples of instruction-feedback pairs include (Instruction: Move_Left, Feedback: Danger_High) and (Instruction: Attack_Target A, Feedback: Target A_Health Points Decreased). Cluster analysis is performed on all instruction-feedback pairs, grouping them into different candidate pattern clusters based on the semantic similarity of the action instructions and the change pattern similarity of the state feedback. In some embodiments, the semantic similarity of the action instructions is obtained by calculating the cosine distance between the instruction text and the pre-trained word vector space, and the change pattern similarity of the state feedback is obtained by comparing the distance between the vector representation of the state feedback and the predefined change pattern prototype. The instruction-feedback pair clustering analysis uses a clustering algorithm based on a comprehensive similarity metric. In specific implementations, the comprehensive similarity metric uses the formula:

[0025] in: Indicates instruction feedback With instruction feedback Overall similarity between characters and These are preset weighting coefficients, characters Indicates instruction feedback Action instructions in With instruction feedback Action instructions in Semantic similarity between characters Indicates instruction feedback Status feedback in With instruction feedback Status feedback in Clustering algorithms assess the similarity of change patterns between clusters based on comprehensive similarity. Commands that exceed the threshold θ are grouped into the same candidate pattern cluster.

[0026] Typical instruction feedback pairs are selected from each candidate pattern cluster, and their semantic framework of behavioral instructions and the changing patterns of state feedback are abstracted to generate initial paradigm templates. In specific implementations, the selection of typical instruction feedback pairs is based on the comprehensive similarity centrality of the instruction feedback pairs in their respective candidate pattern clusters. The instruction feedback pair with the highest centrality is selected as the typical instruction feedback pair. The abstract semantic framework of behavioral instructions is to replace the parameters in the specific instructions with variables, and the abstract changing patterns of state feedback are to extract the core state variables in the feedback and their changing directions. The generalization ability of the initial paradigm templates is evaluated, and initial paradigm templates with similar semantic frameworks and overlapping changing patterns are merged, while initial paradigm templates with too few covered instruction feedback pairs are eliminated, ultimately forming an interaction paradigm template library. In some embodiments, semantic framework similarity is determined by calculating the edit distance of the abstract behavioral instruction templates, and overlapping changing patterns are determined by comparing the consistency of state variables and changing directions in the expected state feedback patterns. The generalization ability is evaluated based on the number of historical instruction feedback pairs covered by the initial paradigm templates. Initial paradigm templates with a coverage number lower than a threshold η are eliminated. Each interaction paradigm template in the interaction paradigm template library contains an abstract behavioral instruction template and an expected state feedback pattern.

[0027] It is understandable that the behavior paradigm extraction module distills reusable behavior patterns from historical interaction data through the above steps. The interaction paradigm template library, as an abstract representation library of behavior patterns, provides a matching basis for the subsequent interaction sequence enhancement module. It is also understandable that the output of the behavior pattern distillation pipeline is a structured interaction paradigm template library, rather than the original instruction feedback pair sequence.

[0028] Example 3: The interaction sequence enhancement module performs pattern matching and annotation on standardized interaction event sequences based on the interaction paradigm template library, generating enhanced interaction sequences with paradigm tags. The semantic matching degree of each structured event unit in the standardized interaction event sequence is calculated against each abstract behavior instruction template in the interaction paradigm template library. A paradigm tag corresponding to the abstract behavior instruction template with the highest semantic matching degree is assigned to each structured event unit as its behavior paradigm tag. Based on the behavior paradigm tag, the corresponding expected state feedback pattern is retrieved from the interaction paradigm template library. The actual environmental state change immediately following the current structured event unit in the standardized interaction event sequence is compared with the retrieved expected state feedback pattern, and the conformity is calculated. The behavior paradigm tag and conformity are used as annotation information and attached to the corresponding structured event unit to form an enhanced interaction sequence with paradigm tags.

[0029] In its implementation, the interaction sequence enhancement module performs pattern matching and annotation on standardized interaction event sequences based on the interaction paradigm template library. The standardized interaction event sequences are generated by the data preprocessing and parsing module. Examples of standardized interaction event sequences include multiple structured event units, such as "Subject: Agent; Action: Move; Object: None; Modifier: Left Direction" and "Subject: Agent; Action: Attack; Object: Target B; Modifier: Use Ranged Weapon". The interaction paradigm template library is generated by the behavior paradigm extraction module. Examples of the interaction paradigm template library include template X: {Abstract Behavior Command Template: "Move_<Direction>", Expected State Feedback Pattern: "Agent_Position: Update"}, and template Y: {Abstract Behavior Command Template: "Attack_<Target>_<Method>", Expected State Feedback Pattern: "<Target>_Health: Decrease"}. The interaction sequence enhancement module calculates the semantic matching degree between each structured event unit in the standardized interaction event sequence and each abstract behavior command template in the interaction paradigm template library.

[0030] In some embodiments, semantic matching degree calculation involves a comparison between the vectorized representation of a structured event unit and the vectorized representation of an abstract behavior instruction template. The vectorized representation of the structured event unit is obtained by concatenating the fields "subject," "behavior," "object," and "modifier" and inputting them into a text encoder. The vectorized representation of the abstract behavior instruction template is obtained by inputting the template string (such as "move_<direction>") into the same text encoder. In specific implementations, semantic matching degree calculation uses the following formula:

[0031] in: This represents the semantic matching degree between the k-th structured event unit and the j-th abstract behavior instruction template in the interaction paradigm template library, where the characters represent the semantic matching degree. This represents the cosine similarity calculation function, for characters. The vectorized representation of the k-th structured event unit, character Let X represent the vectorized representation of the j-th abstract behavior instruction template in the interaction paradigm template library, and the semantic matching degree between the structured event unit "Subject: Agent; Behavior: Move; Object: None; Modifier: Direction Left" and the template X "Move_<Direction>". The calculated high numerical value represents the semantic match degree with template Y "attack_<target>_<method>". The calculation yielded a low value.

[0032] Each structured event unit is assigned a paradigm tag corresponding to the abstract behavior instruction template with the highest semantic matching degree, which serves as its behavior paradigm tag. In specific implementation, the assignment process involves selecting the template with the highest semantic matching degree. The largest template index j is used, and the unique identifier of that template is appended to the structured event unit as a behavior paradigm label. For the structured event unit in the example above, since... Higher than The behavior paradigm label "template X_identifier" is assigned. Based on the behavior paradigm label, the corresponding expected state feedback pattern is retrieved from the interaction paradigm template library. In specific implementation, the interaction paradigm template library is queried through the behavior paradigm label "template X_identifier" to retrieve the corresponding expected state feedback pattern "agent_position:update". The retrieval operation is completed based on the mapping relationship between the label and the template.

[0033] The actual environmental state change immediately following the current structured event unit in the standardized interaction event sequence is compared with the retrieved expected state feedback pattern to calculate the conformity score. In specific implementation, the actual environmental state change is a structured event unit extracted from the standardized interaction event sequence that occurs immediately after the current structured event unit in time and whose subject is "environment". An example of the actual environmental state change is the structured event unit "Subject: Environment; Action: State Change; Object: Position; Modifier: Coordinates (X1, Y1)". The expected state feedback pattern "Agent_Position: Update" includes the state variable "Position" and the change direction "Update". The conformity score is calculated by comparing the consistency between the state variables and change direction extracted from the actual environmental state change and the state variables and change direction defined in the expected state feedback pattern. The conformity score can be a binary value or a continuous value based on the degree of attribute matching. In the example scenario, the actual changed state variable "Position" is consistent with the state variable "Position" in the expected state feedback pattern, and the actual change direction "Coordinate Change" implies the "Update" direction. Therefore, the conformity score is high.

[0034] In some embodiments, compliance is calculated using the following formula:

[0035] in: This represents the conformity score calculated for the k-th structured event unit, where the character... This represents the set of state attributes extracted from the actual environmental state changes following the k-th structured event unit, and the character set is... This represents the set of expected state attributes parsed from the retrieved expected state feedback patterns, characters. Indicates the number of elements in the set, character This represents the intersection operation of sets. In the example above, Includes {state variable: position, direction of change: coordinate change}. Given a set containing {state variable: position, change direction: update}, the intersection operation result {state variable: position} has 1 element. The number of elements is 2, therefore the satisfaction degree is 2. The value is calculated to be 0.5.

[0036] The behavioral paradigm label and conformity score are used as annotation information and attached to the corresponding structured event unit to form an enhanced interaction sequence with paradigm labels. In specific implementation, the attachment operation is to write the behavioral paradigm label and conformity score as new fields into the data structure of the structured event unit. An example of an enhanced interaction sequence with paradigm labels is [Timestamp: T1, Structured event unit: Subject: Agent; Behavior: Move; Object: None; Modifier: Left direction; Behavior paradigm label: Template X_Identifier; Conformity score: 0.8]. Optionally, the enhanced interaction sequence with paradigm labels is stored in the form of a time series database or an array. It can be understood that the enhanced interaction sequence with paradigm labels not only contains the original interaction event, but also injects behavioral pattern information extracted from historical data. The behavioral paradigm label identifies the abstract behavior category to which the current event belongs, and the conformity score reflects the degree of consistency between the actual consequence of the current event and the expected consequence of the historical behavior pattern. It can be understood that the output of the interaction sequence enhancement module is an enhanced interaction sequence with paradigm labels, which will serve as the direct input data for the subsequent offline policy inference network training.

[0037] See Figure 4 This is a scatter plot of state attribute matching rate and consistency, belonging to the data analysis chart category. Variable Relationships: From the distribution, there is no strong linear correlation between state attribute matching rate and consistency (consistency fluctuates significantly as the matching rate increases), indicating that attribute overlap is not the sole factor determining consistency. Paradigm Label Characteristics: Data points of different colors (paradigm labels) are scattered, with no single paradigm cluster corresponding to high / low consistency, reflecting that the matching effects of different paradigms are not significantly different. Outliers: The graph contains a few points with low matching rates but high consistency (or vice versa), which may correspond to special interaction scenarios. This type of chart is typically used for performance evaluation of interactive systems (such as analyzing the matching effect of agent behavior and environmental feedback), helping to optimize paradigm template libraries or policy inference models.

[0038] Example 4: The offline policy inference module uses augmented interaction sequences with paradigm labels to train an offline policy inference network to generate a policy inference network model. Continuous segments are extracted from the augmented interaction sequences with paradigm labels as training samples. Each training sample contains an initial event, a series of intermediate events, and an ending event. The event sequences in the training samples are converted into a multi-channel input tensor consisting of event vectors, paradigm label vectors, and conformity vectors. An offline policy inference network with multi-layer recurrent and attention mechanisms is constructed, including an event encoding layer, a pattern memory layer, and a policy decoding layer. The offline policy inference network is trained using the multi-channel input tensor, enabling the event encoding layer to learn to extract features from the event vectors, the pattern memory layer to learn to activate different behavioral pattern memory units based on the paradigm label vector and conformity vector, and the policy decoding layer to learn to predict the next optimal behavioral paradigm and its corresponding operation instruction based on the encoded features and the activated pattern memory units.

[0039] When the prediction accuracy of the offline policy inference network on the validation set reaches a preset threshold, the trained policy inference network model is obtained. A new sequence of interaction events is input into the policy inference network model, which outputs a policy set constructed by diverse agents. The same standardization and paradigm labeling process is applied to the new sequence of interaction events to obtain a new enhanced interaction sequence with paradigm labels. The current segment of the new enhanced interaction sequence with paradigm labels is input into the event encoding layer of the policy inference network model to obtain the current state encoding. The current state encoding is input into the pattern memory layer of the policy inference network model, along with the paradigm label vector and conformity vector of the current segment, activating multiple related behavioral pattern memory units. In the policy decoding layer of the policy inference network model, based on the current state encoding and all activated behavioral pattern memory units, multiple candidate subsequent behavioral paradigms and corresponding operation instructions are generated, each candidate subsequent behavioral paradigm accompanied by a diversity score. A preset number of candidate subsequent behavioral paradigms and their operation instructions are selected from high to low based on the diversity scores to form a policy option. For multiple decision points in a new sequence of interactive events, the steps from inputting the current segment to generating policy options are repeated, generating a corresponding policy option for each decision point. The set of policy options for all decision points constitutes a diverse agent-constructed policy set.

[0040] In practical implementation, the offline policy inference module uses augmented interaction sequences with paradigm labels to train the offline policy inference network to generate a policy inference network model. Continuous segments are extracted from the augmented interaction sequences with paradigm labels as training samples. Each training sample contains an initial event, a series of intermediate events, and an ending event. In the example scenario, the augmented interaction sequence segment with paradigm labels contains five consecutive structured event units, labeled as events E1, E2, E3, E4, and E5, respectively. Event E1 is set as the initial event, events E2, E3, and E4 are set as intermediate events, and event E5 is set as the ending event. The construction of the training samples aims to learn the policy mapping relationship from the initial event and intermediate event sequences to the ending event.

[0041] The event sequences in the training samples are converted into a multi-channel input tensor consisting of event vectors, paradigm label vectors, and conformity vectors. In practice, the event vector is obtained by jointly encoding the "subject", "behavior", "object", and "modifier" fields in the structured event unit into a fixed-length numerical vector. The paradigm label vector is obtained by one-hot encoding the behavior paradigm labels. The conformity vector is a one-dimensional vector composed of scalar conformity values. For a training sample containing multiple events, the event vector, paradigm label vector, and conformity vector corresponding to each event are stacked in the time dimension to form a multi-channel input tensor. See Table 1 for an example of the structure of the multi-channel input tensor.

[0042] Table 1: Multichannel Input Tensor Structure Table

[0043] An offline policy inference network with multi-layered recurrent and attention mechanisms is constructed. The offline policy inference network includes an event encoding layer, a pattern memory layer, and a policy decoding layer. The event encoding layer is composed of long short-term memory network units, which are responsible for sequentially processing the event vector sequence in the multi-channel input tensor and extracting time-related features. The pattern memory layer is a content-addressable memory matrix. Each row of the matrix stores a behavior paradigm memory unit. The key of the memory unit consists of the historical statistical features of the corresponding paradigm label vector and conformity vector. The policy decoding layer is a feedforward neural network. Its input is the final hidden state output by the event encoding layer and the weighted combination of the contents of the activated memory units in the pattern memory layer.

[0044] The offline policy inference network is trained using multi-channel input tensors. In practice, the training objective is to minimize the cross-entropy loss between the predicted behavior paradigm generated by the policy decoding layer and the behavior paradigm labels of real subsequent events in the training samples. The network parameters are adjusted using a backpropagation algorithm, which iteratively adjusts the parameters of the offline policy inference network based on the cross-entropy loss between the predicted behavior paradigm generated by the policy decoding layer and the behavior paradigm labels of real subsequent events in the training samples. In each training iteration, after forward propagation, the network obtains the predicted output, and the cross-entropy loss value between the predicted output and the true label is calculated. This loss value is used as a scalar to measure the current degree of prediction error of the network. Starting from this final loss value, the backpropagation algorithm calculates the gradient of the loss value with respect to each trainable parameter in the network from top to bottom, following the chain rule.

[0045] Specifically, the algorithm first calculates the gradient of the loss with respect to the connection weights and bias parameters of the policy decoding layer. These gradients indicate the direction and magnitude in which each parameter of the policy decoding layer should be adjusted to reduce the loss. The gradient information continues to propagate to the pattern memory layer, where the gradient of the loss with respect to the content of the memory units and the key-value parameters in the attention mechanism is calculated. This determines how to adjust the knowledge representation stored in the memory matrix and the attention method for retrieving knowledge. The gradients further propagate to the event encoding layer, where the gradient of the loss with respect to the gating weights and recurrent connection weights within the long short-term memory network units is calculated to adjust the extraction method of event sequence features. After obtaining the gradients of all parameters, the current value of each parameter is subtracted from the product of its gradient and a learning rate hyperparameter, thus updating that parameter. By repeatedly performing forward propagation, loss calculation, backpropagation gradient calculation, and parameter update on the entire training set, the parameters of the event encoding layer are adjusted to extract contextual features from the event vector sequence that can more effectively predict subsequent behavior patterns. The parameters of the pattern memory layer are adjusted to more accurately activate relevant behavior pattern memory units based on the input paradigm label vector and conformity vector. The parameters of the policy decoding layer are adjusted to more accurately predict the next optimal behavior paradigm and its operation instruction based on the encoded features and activated memory content. When the network's prediction accuracy on the validation set reaches a preset threshold, this iterative optimization process terminates. At this point, the network parameters have been adjusted to a relatively optimal state, resulting in the trained policy inference network model.

[0046] In some embodiments, a new sequence of interactive events is input into the policy inference network model, which outputs a set of diverse agent construction policies. The new sequence of interactive events is then subjected to the same standardization and paradigm labeling process to obtain a new enhanced interactive sequence with paradigm labels. The segment before the current decision point in the new enhanced interactive sequence with paradigm labels is used as input, and the current segment of the new enhanced interactive sequence with paradigm labels is input into the event encoding layer of the policy inference network model to obtain the current state encoding. The current state encoding is the hidden state vector output by the event encoding layer after processing all events in the current segment. The current state code is input into the pattern memory layer of the policy inference network model, along with the paradigm label vector and conformity vector of the last event in the current segment. This activates multiple related behavioral pattern memory units. In practice, the activation process is completed by calculating the similarity between the current state code and the key of each memory unit in the memory matrix, and selecting the top K memory units with the highest similarity. Each activated memory unit receives an attention weight. In the policy decoding layer of the policy inference network model, based on the current state code and the content of all activated behavioral pattern memory units, multiple candidate subsequent behavioral paradigms and corresponding operation instruction vectors are generated. Each candidate subsequent behavioral paradigm is accompanied by a diversity score.

[0047] Diversity scores are used to measure the degree of deviation between candidate subsequent behavioral paradigms and historically common behavioral patterns. It can be understood that candidate strategies with high diversity scores represent more exploratory behavioral choices. In practice, diversity scores are calculated using the following formula:

[0048] Where: characters The character represents the diversity score of the subsequent behavioral paradigm of the p-th candidate. This represents the original logistic output value generated by the policy decoding layer for the p-th candidate paradigm, in character format. This represents the total number of candidate paradigms generated by the policy decoding layer, in characters. The natural exponential function is used to select a preset number N of candidate subsequent behavioral paradigms and their operation instruction vectors from high to low based on the diversity score, thus forming a policy option.

[0049] For multiple decision points in a new sequence of interactive events, the steps from inputting the current segment to generating policy options are repeated, generating a corresponding policy option for each decision point. The set of policy options for all decision points constitutes a diversified agent construction policy set. Optionally, the diversified agent construction policy set is organized in a tree structure or a list structure, where each node represents a decision point, and N policy options for that decision point are attached to the node. In some embodiments, each policy option, in addition to containing candidate behavior paradigms and operation instruction vectors, also contains an execution success rate prediction value based on model confidence estimation.

[0050] Example 5: The agent configuration generation module generates an executable agent configuration file based on a diverse set of agent construction strategies. It parses each strategy option in the diverse agent construction strategy set, extracting the operation instruction sequence and corresponding behavior paradigm identifier. Based on the behavior paradigm identifier, it retrieves the corresponding abstract behavior instruction template and expected state feedback mode from the interaction paradigm template library. It merges the specific operation instruction sequence from the strategy option with the retrieved abstract behavior instruction template to generate specific, parameterized behavior execution scripts. It associates each behavior execution script with its corresponding expected state feedback mode, serving as the expected judgment condition for the success or failure of the behavior execution script. All behavior execution scripts and their expected judgment conditions are arranged according to the decision point order of the strategy options and encapsulated into an executable agent configuration file conforming to the target platform specifications. After generating the executable agent configuration file, the system also executes a strategy verification and optimization loop, loads the executable agent configuration file into the simulation test environment, and drives the simulated agent to run.

[0051] The system records all simulated interaction events triggered by the simulated agent during operation, forming a simulated interaction log. Key performance indicator (KPI) sequences are extracted from the simulated interaction log and compared with a preset target performance curve to calculate the policy execution deviation. If the policy execution deviation exceeds a tolerance threshold, a correction data package is generated based on the simulated interaction log. This correction data package is used as a new training sample and injected into an enhanced interaction sequence with paradigm labels, triggering incremental training of the policy inference network model and updating its parameters. The system also includes a dynamic maintenance mechanism for an interaction paradigm template library, continuously monitoring newly generated standardized interaction event sequences. Newly generated standardized interaction event sequences are matched with existing templates in the interaction paradigm template library to identify anomalous interaction segments that cannot be effectively matched by existing templates. These anomalous interaction segments are clustered and abstracted to generate candidate new paradigm templates. The frequency of occurrence and potential utility value of these candidate new paradigm templates in historical interaction data are evaluated. The evaluated candidate new paradigm templates are incorporated into the interaction paradigm template library and marked as new templates for subsequent pattern matching and annotation processing.

[0052] In practical implementation, the agent configuration generation module generates an executable agent configuration file based on the diverse agent construction strategy set output by the offline strategy inference module. It parses each strategy option in the diverse agent construction strategy set, extracting the operation instruction sequence and corresponding behavioral paradigm identifier. Based on the behavioral paradigm identifier, it retrieves the corresponding abstract behavioral instruction template and expected state feedback mode from the interaction paradigm template library maintained by the behavioral paradigm extraction module. In practice, it queries the interaction paradigm template library based on the behavioral paradigm identifier "composite exploration_paradigm_ID_01" to retrieve the associated abstract behavioral instruction template "move to <coordinates> and execute <probe action>" and the expected state feedback mode "<area>_visibility: increase; <agent>_energy: consumption_low". The retrieval operation is completed through a database query interface.

[0053] The specific operation instruction sequence in the strategy options is fused with the invoked abstract behavior instruction template to generate specific, parameterized behavior execution scripts. The fusion process instantiates the variables in the abstract behavior instruction template using the parameter values ​​in the specific operation instruction sequence. Each behavior execution script is associated with its corresponding expected state feedback pattern, serving as the expected judgment condition for the success or failure of the script's execution. In specific implementation, the associated operation converts the expected state feedback pattern "<region>_visibility: improved; <agent>_energy: consumption_low" into a verifiable logical assertion. An example logical assertion is "After execution, the visibility value around region (100, 200) should be greater than the value before execution, and the agent's energy value reduction should be less than the threshold ε". All behavior execution scripts and their expected judgment conditions are arranged according to the decision point order of the strategy options and encapsulated into an executable agent configuration file conforming to the target platform specifications. The executable agent configuration file uses a structured data format, encapsulating content including a decision point sequence, a list of optional behavior execution scripts under each decision point, and the expected judgment condition corresponding to each script.

[0054] In some embodiments, after generating an executable agent configuration file, the system also performs a policy verification and optimization loop, loads the executable agent configuration file into a simulation test environment, drives the simulated agent to run, the simulation test environment reproduces the core state logic of the real interaction environment, records all simulated interaction events triggered by the simulated agent during operation, and forms a simulation interaction log. The format of the simulation interaction log is consistent with the format of the original interaction log data stream.

[0055] Key performance indicator (KPI) sequences are extracted from simulated interaction logs. These KPI sequences are compared with a preset target performance curve to calculate the strategy execution deviation. In practice, KPIs can be task completion time, total resource consumption, or target achievement rate. The preset target performance curve defines the trajectory of the expected KPI over time. The strategy execution deviation is calculated using the following formula:

[0056] Where: characters Indicates the policy execution deviation, character Indicates the length of the key performance indicator sequence, characters This represents the l-th key performance indicator value extracted during the simulation run, character... This represents the l-th target index value corresponding to the preset target performance curve.

[0057] If the policy execution deviation exceeds the tolerance threshold, a correction data packet is generated based on the simulated interaction logs. This correction data packet contains information about the decision points that led to the underperformance, the actual triggered sequence of simulated interaction events, and the calculated policy execution deviation. This correction data packet is then used as a new training sample and injected into the enhanced interaction sequence with paradigm labels generated by the interaction sequence enhancement module. This triggers incremental training of the policy inference network model in the offline policy inference module, updating the model's parameters. In essence, the policy verification and optimization loop forms a closed loop, enabling the agent's policy construction to be evaluated and iteratively improved in a simulated environment. The system also includes a dynamic maintenance mechanism for an interaction paradigm template library, continuously monitoring newly generated standardized interaction event sequences processed by the data preprocessing and parsing modules.

[0058] The newly generated standardized interaction event sequences are matched with existing templates in the interaction paradigm template library to identify abnormal interaction fragments that cannot be effectively matched by existing templates. In specific implementations, matching is completed by calculating semantic matching degree. Interaction fragments with semantic matching degree below a threshold γ are identified as abnormal interaction fragments. Abnormal interaction fragments are clustered and abstracted to generate candidate new paradigm templates. Clustering is based on the similarity of structured event units in abnormal interaction fragments, and the abstraction process is consistent with the process of generating initial paradigm templates in the behavior paradigm extraction module. The frequency of occurrence and potential utility value of candidate new paradigm templates in historical interaction data are evaluated. In some embodiments, the frequency of occurrence is calculated by counting the number of event units in the historical standardized interaction event sequences that the candidate new paradigm template can cover, and the potential utility value is estimated by analyzing the final task benefits achieved by the interaction fragments applying the template. The evaluated candidate new paradigm templates are merged into the interaction paradigm template library and marked as new templates for subsequent pattern matching and annotation processing. Optionally, the merging operation will assign a version number to the new template and record its creation time and source.

[0059] See Figure 5 This is a two-bar chart comparing behavioral paradigms, used to show the difference between the "original occurrence count" and "configured usage count" of different behavioral paradigms. This type of chart is typically used for optimizing agent behavioral paradigms: by comparing the differences between "original occurrence" and "configured usage," the call weights of each paradigm in the agent's configuration file can be adjusted to make the agent's behavior more aligned with the needs of actual interaction scenarios. Currently, the agent configuration underutilizes some high-frequency paradigms (such as ID_02), and its configuration weight could be increased; while for paradigms with low original occurrence, such as ID_01, the rationale for its high configuration usage needs to be evaluated.

[0060] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0061] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A system for constructing diverse interactive intelligent agents based on offline learning, characterized in that, The system includes: The data preprocessing and parsing module is configured to receive the raw interaction log data stream stored offline, perform cross-log source time alignment and cleaning processing on the raw interaction log data stream to obtain a standardized interaction event sequence, and separate the agent behavior instruction stream and the environment state feedback stream from the standardized interaction event sequence. The behavior paradigm extraction module is configured to input the agent's behavior instruction stream and the environmental state feedback stream into the behavior pattern distillation pipeline to generate an interaction paradigm template library containing multiple behavior paradigms. The interaction sequence enhancement module is configured to perform pattern matching and annotation processing on the standardized interaction event sequence based on the interaction paradigm template library, and generate an enhanced interaction sequence with paradigm tags. The offline policy inference module is configured to train the offline policy inference network using the enhanced interaction sequence with paradigm labels to generate a policy inference network model, and input the new interaction event sequence into the policy inference network model to output a policy set for diverse agents. The agent configuration generation module is configured to generate executable agent configuration files based on the diverse agent construction strategy set.

2. The diversified interactive intelligent agent construction system based on offline learning according to claim 1, characterized in that, The process of performing cross-log-source time-series alignment and cleaning on the original interaction log data stream yields a standardized sequence of interaction events, including: Raw log fragments are extracted from different log sources. Each raw log fragment contains a timestamp, an agent identifier, and raw event description text. Based on the timestamps and agent identifiers, the original log fragments from different log sources are cross-correlated and sorted to form an initial event chain with agents as the dimension. Noise words are filtered and domain terminology is standardized and replaced for each original event description text in the initial event chain to obtain a cleaned event description. Dependency parsing was performed on the description of the purification event to identify the core behavioral actions, the target objects, and environmental modifiers. The identified core behaviors, targets, and environmental modifiers are reorganized according to a preset event structure template to generate structured event units. All structured event units are connected in chronological order to form the standardized interactive event sequence.

3. The diversified interactive intelligent agent construction system based on offline learning according to claim 2, characterized in that, The step of separating the agent behavior instruction stream and the environment state feedback stream from the standardized interaction event sequence includes: For each structured event unit in the standardized interaction event sequence, the subject type is determined to distinguish between active behavior units initiated by the agent and state change units triggered by the environment. Extract all active behavior units according to their timestamps to form a preliminary sequence of behavior instructions; Extract all state change units according to their timestamps to form a preliminary state feedback sequence; The initial action instruction sequence is checked for instruction integrity, and the broken instruction chain caused by log loss is detected and filled to generate a continuous stream of agent action instructions. The initial state feedback sequence is checked for state continuity, and abnormal jumps in state values ​​over time are smoothed to generate a continuous environmental state feedback stream.

4. The system for constructing diverse interactive intelligent agents based on offline learning according to claim 3, characterized in that, The process involves inputting the agent's behavioral command stream and the environmental state feedback stream into a behavioral pattern distillation pipeline to generate an interaction paradigm template library containing multiple behavioral paradigms, including: The agent's behavioral instruction stream and the environment's state feedback stream are synchronously sliced ​​in the time dimension to obtain multiple aligned time segments; Within each aligned time segment, the agent's behavioral instructions are paired with the subsequent environmental state feedback to form an instruction-feedback pair; Cluster analysis is performed on all instruction feedback pairs, and the instruction feedback pairs are classified into different candidate pattern clusters based on the semantic similarity of the behavioral instructions and the similarity of the change patterns of the state feedback. Select typical instruction feedback pairs from each candidate pattern cluster, and abstract the semantic framework of their behavioral instructions and the changing patterns of their state feedback to generate an initial paradigm template. The initial paradigm templates are evaluated for generalization ability. Initial paradigm templates with similar semantic frameworks and overlapping change patterns are merged. Initial paradigm templates with too few instruction feedback pairs are eliminated. Finally, the interaction paradigm template library is formed. Each interaction paradigm template in the interaction paradigm template library contains an abstract behavior instruction template and an expected state feedback pattern.

5. The diversified interactive intelligent agent construction system based on offline learning according to claim 4, characterized in that, The step of performing pattern matching and annotation on the standardized interaction event sequence based on the interaction paradigm template library to generate an enhanced interaction sequence with paradigm tags includes: The semantic matching degree of each structured event unit in the standardized interactive event sequence is calculated with each abstract behavior instruction template in the interactive paradigm template library; Assign a paradigm tag corresponding to the abstract behavior instruction template with the highest semantic matching degree to each structured event unit, and use it as its behavior paradigm tag; Based on the behavioral paradigm labels, retrieve the corresponding expected state feedback patterns from the interaction paradigm template library; The actual environmental state changes immediately following the current structured event unit in the standardized interactive event sequence are compared with the retrieved expected state feedback pattern, and the conformity is calculated. The behavioral paradigm labels and the conformity level are used as annotation information and attached to the corresponding structured event units to form the enhanced interaction sequence with paradigm labels.

6. The system for constructing diverse interactive intelligent agents based on offline learning according to claim 5, characterized in that, The step of training an offline policy inference network using the enhanced interaction sequences with paradigm labels to generate a policy inference network model includes: Continuous segments are extracted from the enhanced interaction sequence with paradigm labels as training samples. Each training sample contains an initial event, a series of intermediate events, and an ending event. The event sequence in the training samples is converted into a multi-channel input tensor consisting of an event vector, a paradigm label vector, and a conformity vector. An offline policy inference network with multi-layered recurrence and attention mechanisms is constructed, comprising an event encoding layer, a pattern memory layer, and a policy decoding layer. The offline policy inference network is trained using the multi-channel input tensor, enabling the event encoding layer to learn to extract features from event vectors, the pattern memory layer to learn to activate different behavioral pattern memory units based on the paradigm label vector and conformity vector, and the policy decoding layer to learn to predict the next optimal behavioral paradigm and its corresponding operation instruction based on the encoded features and activated pattern memory units. When the prediction accuracy of the offline policy inference network on the validation set reaches a preset threshold, the trained policy inference network model is obtained.

7. The system for constructing diverse interactive intelligent agents based on offline learning according to claim 6, characterized in that, The step of inputting the new sequence of interactive events into the policy inference network model and outputting a diverse set of agent-constructed policies includes: The same standardization and paradigm labeling process is applied to the new sequence of interactive events to obtain a new enhanced sequence of interactive events with paradigm labels. The current segment of the new enhanced interaction sequence with paradigm labels is input into the event encoding layer of the policy inference network model to obtain the current state encoding; The current state encoding is input into the pattern memory layer of the policy inference network model, and the paradigm label vector and conformity vector of the current segment are input at the same time to activate multiple related behavioral pattern memory units. In the policy decoding layer of the policy inference network model, based on the current state encoding and all activated behavior pattern memory units, multiple candidate subsequent behavior paradigms and corresponding operation instructions are generated, and each candidate subsequent behavior paradigm is accompanied by a diversity score. Based on the diversity score, a preset number of candidate subsequent behavior paradigms and their operation instructions are selected from high to low to form a strategy option. For multiple decision points in the new interactive event sequence, the steps from inputting the current segment to generating policy options are repeated to generate a corresponding policy option for each decision point. The set of policy options for all decision points constitutes the diverse agent construction policy set.

8. The system for constructing diverse interactive intelligent agents based on offline learning according to claim 7, characterized in that, The step of generating an executable agent configuration file based on the diverse agent construction strategy set includes: Analyze each policy option in the diverse intelligent agent construction policy set, and extract the operation instruction sequence and the corresponding behavior paradigm identifier; Based on the behavior paradigm identifier, retrieve the corresponding abstract behavior instruction template and expected state feedback mode from the interaction paradigm template library; The specific operation instruction sequence in the strategy options is merged with the invoked abstract behavior instruction template to generate a specific, parameterized behavior execution script; Associate each behavior execution script with its corresponding expected state feedback pattern, which serves as the expected judgment condition for whether the behavior execution script is successful or not; All action execution scripts and their expected judgment conditions are arranged in the order of decision points of the policy options and encapsulated into an executable agent configuration file that conforms to the target platform specifications.

9. The system for constructing diverse interactive intelligent agents based on offline learning according to claim 1, characterized in that, After generating an executable agent configuration file, the system also performs a policy verification and optimization loop, including: The executable agent configuration file is loaded into the simulation test environment to drive the simulated agent to run; Record all simulated interaction events triggered by the simulated agent during operation to form a simulated interaction log; Extract key performance indicator sequences from the simulated interaction logs, compare the key performance indicator sequences with the preset target performance curve, and calculate the strategy execution deviation. If the deviation of the strategy execution exceeds the tolerance threshold, a correction data packet is generated based on the simulated interaction log; The corrected data packet is used as a new training sample and injected into the enhanced interaction sequence with paradigm labels to trigger incremental training of the policy inference network model and update the parameters of the policy inference network model.

10. The system for constructing diverse interactive intelligent agents based on offline learning according to claim 1, characterized in that, The system also includes a dynamic maintenance mechanism for an interaction paradigm template library, including: Continuously monitor newly generated standardized interaction event sequences; The newly generated standardized sequence of interactive events is matched with existing templates in the interactive paradigm template library to identify abnormal interactive fragments that cannot be effectively matched by existing templates. Clustering and abstracting the abnormal interaction fragments to generate candidate new paradigm templates; Evaluate the frequency of occurrence and potential utility value of the candidate new paradigm templates in historical interaction data; The candidate new paradigm templates that pass the evaluation will be incorporated into the interaction paradigm template library and marked as new templates for use in subsequent pattern matching and annotation processing.

Images