A strategy optimization method and system based on trajectory structure analysis
By using trajectory structure analysis and fault mode recognition, a structured fault template is generated and patch synthesis is performed, which solves the problems of precision and stability in strategy optimization in the prior art and realizes the location, reusability and traceability of strategy optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SICHUAN NORMAL UNIV
- Filing Date
- 2026-04-18
- Publication Date
- 2026-05-26
Smart Images
Figure CN122086741A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence strategy optimization technology, and in particular to a strategy optimization method and system based on trajectory structure analysis. Background Technology
[0002] Currently, policy optimization frameworks based on large language models, such as Reflexion and ADRD, aim to achieve reinforcement learning through natural language feedback. The Reflexion framework employs a language reflection mechanism, generating natural language reflections through environmental or self-evaluation feedback and storing them in a memory bank to guide subsequent policy generation. However, this framework relies on LLM self-evaluation to generate feedback and directly update the policy, lacking a formal screening and verification mechanism, and cannot guarantee robust convergence of the policy towards the optimal direction. The ADRD framework is designed for interpretable autonomous driving scenarios, organizing environmental information through an information module. The planner in the agent module generates policy text, the encoder compiles it into executable code, and the aggregator provides optimization suggestions based on test results, forming a closed-loop iteration. However, this framework also relies on the subjective judgment of the language model to evaluate the policy effect, lacking the ability to perform refined analysis of the execution process. Furthermore, these methods have significant drawbacks, making it difficult to support robust optimization for complex tasks.
[0003] First, the policy selection mechanism is crude, and the feedback path lacks stability. Existing methods rely on simple language model output selection or self-evaluation, lacking formal convergence guarantees. For example, Reflexion updates the policy directly based on language feedback, which easily leads to repeated agent failures in complex tasks, resulting in low exploration efficiency and unstable policy evolution direction.
[0004] Secondly, strategy optimization lacks structured diagnostic capabilities, making it difficult to pinpoint the exact causes of failure. Existing technologies treat strategy execution as a black box, relying solely on task success or failure results or subjective verbal feedback, failing to accurately locate specific behavioral defects in the decision-making stage. Although ADRD generates procedural strategies, it also lacks structured analysis of the trajectory during optimization, resulting in vague improvement directions and low efficiency.
[0005] Third, the sampling and matching are not precise enough, and language feedback is easily affected by biases. The policies and feedback generated by the language model are easily affected by cue bias or environmental noise, causing policy oscillations or degradation. The subjective self-evaluation of Reflexion and the unconstrained code behavior of ADRD both reflect the problem of large feedback noise, making it difficult to ensure continuous policy improvement.
[0006] The aforementioned shortcomings and steps highlight the over-reliance of existing technologies on natural language feedback, while lacking the ability to refine and stabilize strategy optimization through trajectory structure analysis, procedural correction, and statistical verification mechanisms. Summary of the Invention
[0007] This invention provides a strategy optimization method and system based on trajectory structure analysis to address the shortcomings of existing language feedback-based strategy optimization methods in terms of the precision of strategy generation and updating, and the difficulty in ensuring continuous and stable policy improvement.
[0008] In view of the above technical problems, embodiments of the present invention provide a strategy optimization method based on trajectory structure analysis, including:
[0009] S1. Obtain the task environment description, construct structured prompts through the environment adaptation layer, generate procedural strategy code based on the structured prompts using a language model, perform syntax checks on the procedural strategy code, record genealogy information and evaluate sandbox games. If the verification fails, a retry mechanism is triggered. After all passes, proceed to adversarial execution and trajectory collection.
[0010] S2. The validated strategy is loaded from the local strategy library into the sandbox process of the simulation environment, and it engages in multiple rounds of competition with the opponent in the global strategy library. A differentiated opponent selection mechanism is adopted based on the game type. For symmetric games, the opponent is selected using Softmax temperature sampling based on Elo rating, and for asymmetric games, the strongest opponent is selected. The trajectory data is collected frame by frame by the log recorder and the win and loss results of each game are recorded. All data is persistently stored in the log directory in JSON format as the original trajectory data.
[0011] S3. Import the raw trajectory data into the structured processing module, using force, angle, boundary distance, and immediate reward as four-dimensional feature inputs; automatically segment the trajectory based on reward jumps or boundary distance abrupt changes; extract nine statistical features for each segment, calculate the weighted comprehensive anomaly score of the complete trajectory, and extract key decision nodes; generate a structured decision representation containing stage segment statistics, a list of key decision nodes, and global statistics, and persistently store it in the form of an in-memory data structure.
[0012] S4. Based on structured decision representation, scan all game records in the log directory, filter out losing games with a win rate below the threshold, and extract a 14-dimensional description vector from the trajectory of each game; after standardization, perform K-Means clustering analysis to identify high-frequency failure patterns; calculate the centroid Z-score deviation for each cluster to infer the triggering conditions and erroneous action types; abstract the failure patterns into structured fault templates and persist them.
[0013] S5. Convert the structured fault template and decision difference localization results into three-level structured correction instructions. Generate patch synthesis prompts based on the instructions and original code to guide the language model to make minimal local modifications and generate optimized strategy code. Save the optimized code to the local strategy library, calculate the number of changed lines through the unified difference algorithm, establish the mapping relationship between patches and fault templates, and form a traceable patch chain.
[0014] S6. Load the optimized strategy code into the sandbox process for adversarial testing, and select opponents based on the differences in game type; calculate the promotion criteria through statistical confidence intervals, include strategies that meet the promotion criteria into the global strategy library and perform population update operations, and enter the iterative optimization process for strategies that do not meet the criteria, generating an inheritance lineage chain to form a complete evolution record.
[0015] The present invention has the following beneficial effects:
[0016] Firstly, it achieves localizable and refined diagnosis of strategy optimization: Through the trajectory structured modeling module, the original adversarial process is transformed into a structured decision representation. Trajectory segments are automatically segmented based on reward jumps or boundary distance abrupt changes, nine statistical features are extracted, and a weighted comprehensive anomaly score is calculated. Key decision nodes are then extracted and their triggering reasons are labeled. This structured representation overcomes the limitation of existing technologies that treat strategy execution as a black box, supporting failure mode clustering analysis and success-failure situation difference localization. It achieves precise localization of strategy defects, providing a data foundation for subsequent targeted modifications.
[0017] Secondly, it achieves reusability and structured representation of failure modes: By introducing a failure mode template mechanism, a 14-dimensional descriptive vector is extracted from the statistical features of failure trajectories. After standardization, K-Means clustering analysis is performed to identify high-frequency failure scenarios, and Z-score deviation detection is used to infer triggering conditions and erroneous action types. Finally, the failure mode is abstracted into a structured failure template containing triggering conditions, erroneous action type annotations, cluster center feature vectors, and representative key nodes. This templated representation replaces the traditional unstable description method based on natural language feedback, making problem localization structural and reusable, and significantly improving the engineering controllability of strategy optimization.
[0018] Thirdly, it achieves traceability and local precision in strategy correction: This is achieved through a three-level structured correction instruction system and a patch-guided generation method based on program structure analysis. The correction instructions are divided into target, location, and constraint layers. An abstract syntax tree is used to parse and locate functions or conditional branches in the code, guiding the language model to make minimal modifications within the local code scope. The patch synthesis module quantifies the number of changed lines using a unified difference algorithm, extracts patch comments, and establishes a mapping relationship between each patch and its corresponding fault template and parent strategy. A genealogy tracing module records the complete optimization chain. This mechanism achieves a shift from "global regeneration" to "locally precise correction," ensuring the traceability of strategy evolution.
[0019] Fourth, it achieves statistical reliability and stability in strategy selection: A validation and selection mechanism based on statistical confidence intervals is employed, accumulating samples through adversarial training against multiple opponents in a sandbox process isolation environment. The system uses the Bootstrap resampling method or the Wilson score interval method to calculate the win rate confidence interval, using the lower bound of the confidence interval as the promotion criterion, effectively avoiding misjudgments caused by small sample fluctuations. Opponent selection employs a Softmax temperature sampling strategy based on Elo scores. The temperature parameter can adjust the balance between opponent diversity and adversarial intensity, ensuring stable strategy performance under different opponent levels and significantly improving the stability of strategy evolution.
[0020] Fifth, it achieves lifecycle management and dynamic optimization of the strategy population: by constructing a two-layer strategy pool and strategy lifecycle management structure. The global strategy library stores promotion strategies verified by confidence intervals, while the local strategy library stores candidate strategies that are being iteratively optimized. The two interact dynamically through a promotion mechanism and a Top-K backfilling mechanism. After a new strategy is promoted to the global library, the system automatically backfills the local library with the top-scoring strategies from the global library as a reference baseline for the next round of generation and optimization. The creation source, parent-child relationship, code change statistics, promotion events, and patch-fault template mapping of all strategies are recorded in the genealogy index file, forming a complete and queryable strategy evolution record, realizing continuous optimization and quality assurance of the strategy population.
[0021] In summary, through the synergistic effect of the aforementioned key technologies, this invention achieves a shift from "language reflection-driven optimization" to "structured trajectory diagnosis-driven optimization," making the programmatic strategy evolution process more refined, stable, and engineering-feasible. Attached Figure Description
[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 This is a flowchart of a strategy optimization method based on trajectory structure analysis in one embodiment of the present invention;
[0024] Figure 2 This is a structural diagram of a strategy optimization system based on trajectory structure analysis in one embodiment of the present invention. Detailed Implementation
[0025] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0026] In the description of this invention, it should be understood that the terms "longitudinal," "radial," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.
[0027] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0028] like Figure 1 As shown, an embodiment of the present invention provides a strategy optimization method based on trajectory structure analysis, including:
[0029] S1. Obtain the task environment description, construct structured prompts through the environment adaptation layer, generate procedural strategy code based on the structured prompts using a language model, perform syntax checks on the procedural strategy code, record genealogy information and evaluate sandbox games. If the verification fails, a retry mechanism is triggered. After all passes, proceed to adversarial execution and trajectory collection.
[0030] In one embodiment, step S1 further includes the following sub-steps:
[0031] S101. Obtain the task environment description. The task environment adaptation layer receives the task environment description and constructs a structured prompt. The structured prompt includes environment rules, reference strategy codes sorted by win rate, and a historical reflection memory pool. The language model generates procedural strategy codes under the constraints of the structured prompt.
[0032] S102. Perform a syntax check on the generated strategy code, and verify it through abstract syntax tree parsing and bytecode compilation; if it fails, delete the error file and feed back the syntax error information to the language model for regeneration, allowing a maximum of three retries.
[0033] S103. After passing the syntax check, record the lineage information of the procedural strategy code, which includes the creation source, parent identifier, and code difference statistics. Save the strategy code to the local strategy library and put it into the sandbox for game evaluation. The strategy game is run in operating system-level process isolation, with dual protection mechanisms of whole game timeout and single step timeout. If an execution error occurs, the error message is fed back to the language model, triggering a complete strategy regeneration process (including syntax check and lineage recording), with a maximum of three retries allowed. After all verifications are passed, the strategy is ready to enter the adversarial execution and trajectory collection stage.
[0034] S2. The validated strategy is loaded from the local strategy library into the sandbox process of the simulation environment, and multiple rounds of confrontation are conducted against the opponent in the global strategy library. A differentiated opponent selection mechanism is adopted based on the game type. For symmetric games, the opponent is selected by Softmax temperature sampling based on Elo rating, and for asymmetric games, the strongest opponent is selected. The trajectory data is collected frame by frame by the log recorder and the win and loss results of each game are recorded. All data is persistently stored in the log directory in JSON format as the original trajectory data.
[0035] In one embodiment, step S2 further includes the following sub-steps:
[0036] S201. Load the validated strategy from the local strategy library into the simulation environment sandbox process and engage in multiple rounds of competition with opponents in the global strategy library. A differentiated opponent selection mechanism is adopted based on the game type: For symmetric game environments, a Softmax temperature sampling mechanism based on Elo rating is used to select several opponents from the global strategy library, and the average win rate is calculated after multiple rounds of competition. For asymmetric game environments (such as chase-escape scenarios), an independent opponent selection mechanism is used to obtain the strongest opponent for competition.
[0037] S202. Trajectory data is collected for the generated adversarial process. Trajectory data is collected frame by frame at each simulation step using a logger. A differentiated log collection strategy is adopted: For traditional symmetric game environments, the force values and angles of the agent and the opponent, the distance between the agent and the boundary, and the instant reward for each step are recorded. All data are organized in a two-dimensional list, with the outer index corresponding to the game number and the inner index corresponding to the time step. For asymmetric game environments, an independent logger is used to record role-specific information, which includes observation data, discrete actions, and rewards for each step.
[0038] S203. After the trajectory data collection is completed, record the win / loss results and the final cumulative reward for each game; persist all trajectory data in JSON format to form the raw trajectory data and store it in the log directory, with the file name and strategy identifier maintaining a mapping relationship; the mapping relationship supports direct calling by the subsequent S3 trajectory structured modeling module.
[0039] S3. Import the raw trajectory data into the structured processing module, using force value, angle, boundary distance, and instant reward as four-dimensional feature inputs; automatically segment the trajectory based on reward jumps or boundary distance abrupt changes; extract nine statistical features for each segment, calculate the weighted comprehensive anomaly score of the complete trajectory, and extract key decision nodes; generate a structured decision representation containing stage segment statistics, a list of key decision nodes, and global statistics, and persistently store it in the form of an in-memory data structure; use the in-memory data structure for subsequent fault mode clustering and cross-game comparative analysis.
[0040] In one embodiment, step S3 further includes the following sub-steps:
[0041] S301. Import the original trajectory data into the structured processing module to enter the processing flow. The original trajectory data uses the force value, angle, boundary distance and instant reward at each time step as four-dimensional feature input. The segmented trigger detector automatically segments the original trajectory according to the significant jump of the reward signal or the sharp change of the boundary distance, forming several stage segments.
[0042] S302. Extract nine statistical features from each generated stage segment using a feature extractor, including average force value, standard deviation of force value, average angle, rate of change of angle, average boundary distance, minimum boundary distance, boundary approach frequency, cumulative segment reward, and average reward.
[0043] S303. After extracting the nine statistical features of the stage segment, calculate the weighted comprehensive anomaly score for the complete trajectory. The anomaly scorer sums the normalized differences of four dimensions: reward change, boundary distance change, force value change, and angle change, according to their weights. Extract the key decision nodes with the highest scores and label their triggering reasons (such as sudden reward change, sudden drop in boundary distance, drastic change in force output, etc.). Combine the stage segment statistics, the list of key decision nodes, and the global statistics into a structured decision representation and persist it in the form of an in-memory data structure for direct use by the subsequent S4 fault mode clustering and cross-game comparison analysis module.
[0044] S4. Based on structured decision representation, scan all game records in the log directory, filter out failed games with a win rate lower than the threshold (0.4), and extract a 14-dimensional description vector from the trajectory of each game; after standardization, perform K-Means clustering analysis to identify high-frequency failure patterns; calculate the centroid Z-score deviation for each cluster to infer the triggering conditions and error action types; abstract the failure patterns into structured fault templates and persist them.
[0045] In one embodiment, step S4 further includes the following sub-steps:
[0046] S401. Based on structured decision representation, the log scanner scans all game records in the log directory; the failed game filter filters out failed games with a win rate lower than a threshold (e.g., it can be set to 0.4), and extracts a 14-dimensional description vector from the trajectory of each game. The 14-dimensional description vector includes the average force value, the standard deviation of the force value, the average angle, the standard deviation of the angle, the rate of change of the angle, the boundary distance distribution, the reward feature, the difference between the behavior before and after the game, and the number of game steps.
[0047] S402. Perform cluster analysis on the 14-dimensional description vector. After standardization by a standardization processor, perform K-Means cluster analysis to identify high-frequency failure modes. For each cluster, calculate the Z-score deviation of the centroid relative to the global mean using a Z-score calculator. Select the features with the largest deviation as trigger condition descriptions and label the error action types using a rule inferrer. Error action types include, for example, excessive force, frequent approach to the boundary, frequent direction switching, and late-stage collapse.
[0048] S403. After the clustering analysis is completed, the failure scenarios within the cluster are structured, and representative key decision nodes are extracted using a key node extractor. The failure modes are abstracted into structured fault templates. The structured fault templates include cluster identifiers, number of samples, triggering conditions, error action types, cluster centroid feature vectors, representative key nodes, and sample source information. The structured fault templates are persistently stored in the fault template file under the log directory for direct use by the subsequent S5 structured correction and patch synthesis module.
[0049] S5. Convert the structured fault template and decision difference localization results into three-level structured correction instructions. Generate patch synthesis prompts based on the instructions and original code to guide the language model to make minimal local modifications and generate optimized strategy code. Save the optimized code to the local strategy library, calculate the number of changed lines through the unified difference algorithm, establish the mapping relationship between patches and fault templates, and form a traceable patch chain.
[0050] In one embodiment, step S5 further includes the following sub-steps:
[0051] S501. Obtain the structured fault template and decision difference location results, and convert them into three-level structured correction instructions by the correction instruction generator. The three-level structured correction instructions include a target layer that specifies the type of behavior to be modified (such as boundary avoidance logic, force output control, etc.), a location layer that parses the functions, conditional branches and numerical constants in the original strategy code through an abstract syntax tree to determine the modification location, and a constraint layer that specifies the numerical range that must be met after modification (such as force value interval, minimum safe distance, maximum angle change, etc.).
[0052] S502. Patch synthesis is performed on the generated three-level structured correction instructions. The correction instructions and the original code are combined into patch synthesis prompts through the prompt builder. The language model is required to make minimal local modifications and add patch comment markers to the modified lines. Based on this, a complete optimized version of the policy code is generated by the language model. The optimized version of the policy code is saved to the local policy library through the local library storage.
[0053] S503. After patch synthesis is completed, the number of lines of change between the old and new code is calculated using a unified difference algorithm, and patch comments are extracted using an comment extractor. A mapping relationship is established between the difference records and the corresponding fault templates. A traceable patch chain (including parent strategy identifier, fault template identifier, number of lines of change, and patch comments) is formed through the genealogy tracking module, and the patch chain is persistently stored in the genealogy index file for direct use by the subsequent S6 hierarchical evaluation and population update modules.
[0054] S6. Load the optimized strategy code (i.e., candidate strategies including newly generated strategies and patched strategies) into the sandbox process for adversarial testing, and select opponents based on the differences in game type; calculate the promotion criteria through statistical confidence intervals (Bootstrap resampling method or Wilson scoring interval method), include strategies that meet the promotion criteria into the global strategy library and perform population update operations, and enter the iterative optimization process for strategies that do not meet the criteria, generating an inheritance phylogenetic chain to form a complete evolutionary record.
[0055] In one embodiment, step S6 further includes the following sub-steps:
[0056] S601. Load the optimized strategy code into the sandbox process for adversarial testing. For traditional symmetric game environments, select several opponents by sampling the Softmax temperature based on Elo ratings, play against them one by one, and take the average win rate. For asymmetric game environments, select the strategy with the highest rating among the opponent roles for the game.
[0057] S602. The generated adversarial process is guaranteed for stability and results are accumulated. The process isolator ensures that the strategy crash or timeout will not affect the stability of the main system. The strategy that fails to complete the game normally will obtain error information and feed it back to the language model for retry.
[0058] S603. After accumulating the game results, the promotion determination based on the statistical confidence interval is calculated using the confidence interval calculator. For the win rate of consecutive values in multiple games, the Bootstrap resampling method is used to calculate the 90% confidence interval. For small sample or binary win / loss results, it degenerates into the Wilson scoring interval method. The promotion condition is that the lower bound of the confidence interval is not lower than the preset threshold (e.g., it can be set to 0.55).
[0059] S604. Policies that meet the promotion criteria are added to the global policy library through the global library manager, and the following operations are performed simultaneously: copy the policy file to the global directory, set the initial Elo score, and record the promotion event (including the win rate and confidence interval range) in the genealogy index.
[0060] S605. After the strategy promotion is completed, the local strategy library is cleared, and the top-K backfiller selects several strategies with the highest Elo scores from the global library and backfills them into the local strategy library as the reference baseline for the next iteration. Strategies that do not meet the promotion criteria enter the trajectory diagnosis-driven iterative optimization process, and their game logs, fault templates and old code are submitted to the language model to generate improved versions. The new strategies generated by the iteration automatically inherit the lineage chain of their parents to form a complete evolutionary history.
[0061] like Figure 2 As shown, the present invention also provides a strategy optimization system based on trajectory structure analysis, comprising:
[0062] The strategy code generation and trajectory acquisition module 100 is used to obtain the task environment description, build structured prompts through the environment adaptation layer, generate procedural strategy code based on the structured prompts using a language model, perform syntax checks on the procedural strategy code, record genealogy information and evaluate sandbox games. If the verification fails, a retry mechanism is triggered. After all verifications are successful, the adversarial execution and trajectory acquisition are performed.
[0063] The adversarial execution and raw trajectory acquisition module 200 loads the verified strategies from the local strategy library into the simulation environment sandbox process, and conducts multiple rounds of adversarial battles against opponents in the global strategy library. Based on the game type, a differentiated opponent selection mechanism is adopted. For symmetric games, Softmax temperature sampling based on Elo rating is used to select opponents, while for asymmetric games, the strongest opponent is selected. Trajectory data is collected frame by frame through a log recorder and the win and loss results of each game are recorded. All data is persistently stored in JSON format in the log directory as raw trajectory data.
[0064] The trajectory structured modeling module 300 imports the raw trajectory data into the structured processing module, using force, angle, boundary distance, and immediate reward as four-dimensional feature inputs; it automatically segments the trajectory based on reward jumps or boundary distance abrupt changes; it extracts nine statistical features from each segment, calculates the weighted comprehensive anomaly score of the complete trajectory, and extracts key decision nodes; it generates a structured decision representation containing stage segment statistics, a list of key decision nodes, and global statistics, and persists it in memory data structure form.
[0065] The fault mode identification and template extraction module 400, based on structured decision representation, scans all game records in the log directory, filters out losing games with a win rate below a threshold, and extracts a 14-dimensional description vector from the trajectory of each game; after standardization, it performs K-Means clustering analysis to identify high-frequency failure patterns; it calculates the centroid Z-score deviation for each cluster to infer the triggering conditions and erroneous action types; and it abstracts the failure patterns into structured fault templates and persists them.
[0066] The structured correction and patch synthesis module 500 converts the structured fault template and decision difference localization results into three-level structured correction instructions. Based on the instructions and the original code, it generates patch synthesis prompts, guides the language model to make minimal local modifications and generate optimized strategy code; saves the optimized code to the local strategy library, calculates the number of changed lines through a unified difference algorithm, establishes the mapping relationship between patches and fault templates, and forms a traceable patch chain.
[0067] The hierarchical evaluation and population update module 600 loads the optimized strategy code into the sandbox process for adversarial testing and selects opponents based on the differences in game type. It calculates the promotion criteria through statistical confidence intervals. Strategies that meet the promotion criteria are included in the global strategy library and the population update operation is performed. Strategies that do not meet the criteria enter the iterative optimization process and generate an inheritance lineage chain to form a complete evolutionary record.
[0068] This invention, based on the existing large language model-driven strategy generation framework, constructs a closed-loop automatic improvement method with trajectory structure analysis as its core, enabling strategy optimization to shift from relying on subjective language reflection to refined evolution driven by objective data. This method extracts key decision features by performing structured modeling of execution trajectories, mines high-frequency failure modes based on cluster analysis and generates reusable structured templates, and then guides patch synthesis through program structure analysis to achieve precise local correction, ultimately forming an enhanced closed-loop system of "trajectory acquisition—structured processing—failure diagnosis—structured correction—confidence verification—population update".
[0069] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A strategy optimization method based on trajectory structure analysis, characterized in that, include: S1. Obtain the task environment description, construct structured prompts through the environment adaptation layer, and generate procedural strategy code based on the structured prompts using a language model; Syntax checks are performed on the programmatic strategy code; genealogy information is recorded and sandbox game evaluation is conducted. If the verification fails, a retry mechanism is triggered. After all verifications are successful, the adversarial execution and trajectory collection begin. S2. The validated strategy is loaded from the local strategy library into the sandbox process of the simulation environment, and it engages in multiple rounds of competition with the opponent in the global strategy library. A differentiated opponent selection mechanism is adopted based on the game type. For symmetric games, the opponent is selected using Softmax temperature sampling based on Elo rating, and for asymmetric games, the strongest opponent is selected. The trajectory data is collected frame by frame by the log recorder and the win and loss results of each game are recorded. All data is persistently stored in the log directory in JSON format as the original trajectory data. S3. Import the raw trajectory data into the structured processing module, using force, angle, boundary distance, and immediate reward as four-dimensional feature inputs; automatically segment the trajectory based on reward jumps or boundary distance abrupt changes; extract nine statistical features for each segment, calculate the weighted comprehensive anomaly score of the complete trajectory, and extract key decision nodes; generate a structured decision representation containing stage segment statistics, a list of key decision nodes, and global statistics, and persistently store it in the form of an in-memory data structure. S4. Based on structured decision representation, scan all game records in the log directory, filter out losing games with a win rate below the threshold, and extract a 14-dimensional description vector from the trajectory of each game; after standardization, perform K-Means clustering analysis to identify high-frequency failure patterns. Calculate the centroid Z-score deviation for each cluster to infer the triggering conditions and error action types; Abstract failure modes into structured fault templates and persist them. S5. Convert the structured fault template and decision difference localization results into three-level structured correction instructions. Generate patch synthesis prompts based on the instructions and original code to guide the language model to make minimal local modifications and generate optimized strategy code. Save the optimized code to the local strategy library, calculate the number of changed lines through the unified difference algorithm, establish the mapping relationship between patches and fault templates, and form a traceable patch chain. S6. Load the optimized strategy code into the sandbox process for adversarial testing, and select opponents based on the differences in game type; calculate the promotion criteria through statistical confidence intervals, include strategies that meet the promotion criteria into the global strategy library and perform population update operations, and enter the iterative optimization process for strategies that do not meet the criteria, generating an inheritance lineage chain to form a complete evolution record.
2. The strategy optimization method based on trajectory structure analysis according to claim 1, characterized in that, Step S1 also includes the following steps: S101. Obtain the task environment description. The task environment adaptation layer receives the task environment description and constructs a structured prompt. The structured prompt includes environment rules, reference strategy codes sorted by win rate, and a historical reflection memory pool. The language model generates procedural strategy codes under the constraints of the structured prompt. S102. Perform a syntax check on the generated strategy code, verifying it through abstract syntax tree parsing and bytecode compilation; if it fails, delete the error file and feed back the syntax error information to the language model for regeneration, allowing a maximum of three retries; S103. After passing the syntax check, record the lineage information of the procedural strategy code, including its creation source, parent identifier, and code difference statistics. Save the strategy code to the local strategy library and put it into the sandbox for game evaluation. The strategy game is run in operating system-level process isolation, with dual protection mechanisms of whole-game timeout and single-step timeout. If an execution error occurs, the error message is fed back to the language model, triggering a complete strategy regeneration process, with a maximum of three retries allowed. After all verifications are passed, the strategy is ready to enter the adversarial execution and trajectory collection stage.
3. The strategy optimization method based on trajectory structure analysis according to claim 2, characterized in that, Step S2 also includes the following steps: S201. Load the validated strategy from the local strategy library into the simulation environment sandbox process and engage in multiple rounds of competition with opponents in the global strategy library. A differentiated opponent selection mechanism is adopted based on the game type: For symmetric game environments, a Softmax temperature sampling mechanism based on Elo rating is used to select several opponents from the global strategy library, and the average win rate is calculated after multiple rounds of competition with each opponent; For asymmetric game environments, an independent opponent selection mechanism is used to obtain the strongest opponent for competition. S202. Trajectory data is collected during the generated adversarial process. Trajectory data is collected frame by frame at each simulation step using a logger. A differentiated log collection strategy is adopted: For traditional symmetric game environments, the force values and angles between the agent and the opponent, the distance between the agent and the boundary, and the instant reward for each step are recorded. All data are organized in a two-dimensional list, with the outer index corresponding to the game number and the inner index corresponding to the time step. For asymmetric game environments, an independent logger is used to record role-specific information, which includes observation data, discrete actions, and rewards for each step. S203. After the trajectory data collection is completed, record the win / loss results and the final cumulative reward for each game; persist all trajectory data in JSON format to form the raw trajectory data and store it in the log directory, with the file name and policy identifier maintaining a mapping relationship.
4. The strategy optimization method based on trajectory structure analysis according to claim 3, characterized in that, Step S3 also includes the following steps: S301. Import the original trajectory data into the structured processing module to enter the processing flow. The original trajectory data uses the force value, angle, boundary distance and instant reward at each time step as four-dimensional feature input. The segmented trigger detector automatically segments the original trajectory according to the significant jump of the reward signal or the sharp change of the boundary distance, forming several stage segments. S302. Extract nine statistical features from each generated stage segment using a feature extractor, including average force value, standard deviation of force value, average angle, rate of change of angle, average boundary distance, minimum boundary distance, boundary approach frequency, cumulative segment reward, and average reward. S303. After extracting the nine statistical features of the stage segment, calculate the weighted comprehensive anomaly score for the complete trajectory. The anomaly scorer sums the normalized differences of the four dimensions of reward change, boundary distance change, force value change, and angle change according to their weights. Extract the key decision nodes with the highest scores and label their triggering reasons. Combine the stage segment statistics, the list of key decision nodes, and the global statistics into a structured decision representation and persist it in the form of an in-memory data structure.
5. The strategy optimization method based on trajectory structure analysis according to claim 4, characterized in that, Step S4 also includes the following steps: S401. Based on structured decision representation, the log scanner scans all game records in the log directory; the failed game filter filters out failed games with a win rate lower than a threshold, and extracts a 14-dimensional description vector from each game trajectory. The 14-dimensional description vector includes average force value, force value standard deviation, average angle, angle standard deviation, angle change rate, boundary distance distribution, reward features, differences in behavior before and after the game, and game steps. S402. Perform cluster analysis on the 14-dimensional description vector. After standardization by a standardization processor, perform K-Means cluster analysis to identify high-frequency failure patterns. For each cluster, calculate the Z-score deviation of the centroid relative to the global mean using a Z-score calculator. Select the features with the largest deviation as trigger condition descriptions and label the error action types using a rule inferrer. Error action types include excessive force, frequent approach to the boundary, frequent direction switching, and late-stage collapse. S403. After the clustering analysis is completed, the failure scenarios within the cluster are structured, and representative key decision nodes are extracted using a key node extractor; the failure modes are abstracted into structured fault templates; the structured fault templates include cluster identifiers, number of samples, triggering conditions, error action types, cluster centroid feature vectors, representative key nodes, and sample source information; the structured fault templates are persistently stored in the fault template file under the log directory.
6. The strategy optimization method based on trajectory structure analysis according to claim 5, characterized in that, Step S5 also includes the following steps: S501. Obtain the structured fault template and decision difference location results, and convert them into a three-level structured correction instruction by the correction instruction generator; the three-level structured correction instruction includes a target layer that specifies the type of behavior to be modified, a location layer that parses the functions, conditional branches and numerical constants in the original strategy code through an abstract syntax tree to determine the modification location, and a constraint layer that specifies the numerical range that must be satisfied after modification. S502. Patch synthesis is performed on the generated three-level structured correction instructions. The correction instructions and the original code are combined into patch synthesis prompts through the prompt builder. The language model is required to make minimal local modifications and add patch comment markers to the modified lines. Based on this, a complete optimized version of the strategy code is generated by the language model. The optimized version of the strategy code is saved to the local strategy library through the local library storage. S503. After patch synthesis is completed, the number of lines of change between the old and new code is calculated using a unified difference algorithm, and patch comments are extracted using an annotation extractor. A mapping relationship is established between the difference records and the corresponding fault templates. A traceable patch chain is formed through the genealogy tracking module, and the patch chain is persistently stored in the genealogy index file for direct use by the subsequent S6 hierarchical evaluation and population update modules.
7. The strategy optimization method based on trajectory structure analysis according to claim 6, characterized in that, Step S6 also includes the following steps: S601. Load the optimized strategy code into the sandbox process for adversarial testing. For traditional symmetric game environments, select several opponents by Softmax temperature sampling based on Elo ratings, play against them one by one, and take the average win rate. For asymmetric game environments, select the strategy with the highest rating among the opponent roles for the game. S602. The generated adversarial process is guaranteed for stability and results are accumulated. The process isolator ensures that the failure of the strategy or timeout will not affect the stability of the main system. The strategy that fails to complete the game normally will obtain error information and feed it back to the language model for retry. S603. After accumulating the game results, the promotion determination based on the statistical confidence interval is calculated using the confidence interval calculator. For the win rate of consecutive values in multiple games, the Bootstrap resampling method is used to calculate the 90% confidence interval. For small sample or binary win / loss results, it degenerates into the Wilson scoring interval method. The promotion condition is that the lower bound of the confidence interval is not lower than the preset threshold. S604. Strategies that meet the promotion criteria are added to the global strategy library through the global library manager, and the following operations are performed simultaneously: copy the strategy file to the global directory, set the initial Elo score, and record the promotion event (including win rate and confidence interval range) in the genealogy index. S605. After the strategy promotion is completed, the local strategy library is cleared, and the top-K backfiller selects several strategies with the highest Elo scores from the global library and backfills them into the local strategy library as the reference baseline for the next iteration. Strategies that do not meet the promotion criteria enter the trajectory diagnosis-driven iterative optimization process, and their game logs, fault templates and old code are submitted to the language model to generate improved versions. The new strategies generated by the iteration automatically inherit the lineage chain of their parents to form a complete evolutionary history.
8. A strategy optimization system based on trajectory structure analysis, characterized in that, The method is implemented using the strategy optimization method based on trajectory structure analysis as described in any one of claims 1-7, comprising: The strategy code generation and trajectory acquisition module is used to obtain the task environment description, build structured prompts through the environment adaptation layer, generate procedural strategy code based on the structured prompts using a language model, perform syntax checks on the procedural strategy code, record genealogy information and evaluate sandbox games. If the verification fails, a retry mechanism is triggered. After all verifications are successful, the module proceeds to adversarial execution and trajectory acquisition. The adversarial execution and raw trajectory acquisition module loads validated strategies from the local strategy library into the simulation environment sandbox process, and conducts multiple rounds of adversarial battles against opponents in the global strategy library. A differentiated opponent selection mechanism is adopted based on the game type: for symmetric games, softmax temperature sampling based on Elo rating is used to select opponents, while for asymmetric games, the strongest opponent is selected. Trajectory data is collected frame by frame through a log recorder and the win and loss results of each game are recorded. All data is persistently stored in JSON format in the log directory as raw trajectory data. The trajectory structured modeling module imports the raw trajectory data into the structured processing module, using force, angle, boundary distance, and immediate reward as four-dimensional feature inputs; it automatically segments the trajectory based on reward jumps or boundary distance abrupt changes; it extracts nine statistical features from each segment, calculates the weighted comprehensive anomaly score of the complete trajectory, and extracts key decision nodes; it generates a structured decision representation containing stage segment statistics, a list of key decision nodes, and global statistics, and persists it in memory data structure form. The fault mode identification and template extraction module, based on structured decision representation, scans all game records in the log directory, filters out losing games with a win rate below a threshold, and extracts a 14-dimensional description vector from the trajectory of each game. After standardization, K-Means clustering analysis is performed to identify high-frequency failure patterns. The centroid Z-score deviation is calculated for each cluster to infer the triggering conditions and erroneous action types. The failure patterns are abstracted into structured fault templates and persistently stored. The structured correction and patch synthesis module converts the structured fault template and decision difference localization results into three-level structured correction instructions. Based on the instructions and the original code, it generates patch synthesis prompts, guides the language model to make minimal local modifications and generate optimized strategy code. The optimized code is saved to the local strategy library, and the number of changed lines is calculated through a unified difference algorithm. The mapping relationship between patches and fault templates is established to form a traceable patch chain. The hierarchical evaluation and population update module loads the optimized strategy code into the sandbox process for adversarial testing and selects opponents based on the differences in game type. It calculates the promotion criteria through statistical confidence intervals. Strategies that meet the promotion criteria are included in the global strategy library and the population update operation is performed. Strategies that do not meet the criteria enter the iterative optimization process, generating an inheritance lineage chain to form a complete evolutionary record.