Skill consecutive recruitment construction method based on hierarchical reinforcement learning and terminal

Abstract

The invention discloses a hierarchical reinforcement learning-based skill consecutive recruitment construction method and a terminal. The method comprises the following steps of: modeling state data of a virtual environment by using a depth time sequence modeling algorithm, and generating a sub-target sequence according to the state data; generating a first consecutive call sequence based on the sub-target sequence, identifying illegal operations and potential risks in the first consecutive call sequence, and optimizing to generate a second consecutive call sequence; and with the consecutive call effect, the resource consumption and the consecutive call stability as targets, optimizing the second consecutive call sequence by using a reward function, and constructing and generating an optimal skill consecutive call sequence. According to the method, the virtual environment is comprehensively modeled by using the depth time sequence modeling algorithm, the legality and stability of the consecutive call are verified and optimized through the rule constraint and decision search algorithm, illegal operation and potential risks in skill consecutive call are effectively avoided, and the dynamic adaptability of consecutive call construction is improved, so that high coverage rate and high efficiency are realized. And the skill continuous call test efficiency is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of electronic games, and in particular to a skill combo construction method and terminal based on hierarchical reinforcement learning. Background Art

[0002] In massively multiplayer online role-playing games (MMORPGs), boss battles are a core element of game design. The complexity and diversity of skill combos crucially impact game balance and player experience. Because boss battle skill combinations are highly dynamic and random, skill cooldowns, state changes, and combo combinations vary significantly across different battle phases. Traditional automated testing methods struggle to fully cover all skill combinations and battle scenarios, resulting in insufficient combo coverage and inefficient strategy generation.

[0003] Common automated testing methods currently rely primarily on manually written fixed scripts or rule-based test generation tools. These methods often require manual updates to scripts and test logic when game versions are updated or skill parameters are changed, resulting in high maintenance costs and low testing efficiency. Furthermore, scripted testing methods struggle to adapt to the complex demands of dynamic combat scenarios, such as skill cooldown changes, state transitions, and combo optimization, resulting in test coverage rates generally below 30%. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a skill combo construction method and terminal based on hierarchical reinforcement learning to solve the problem of low combo construction coverage.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A skill combo construction method based on hierarchical reinforcement learning, comprising the following steps: S1. Modeling the state data of the virtual environment using a deep temporal modeling algorithm and generating a sub-goal sequence based on the state data; S2. generating a first combo sequence based on the sub-goal sequence, identifying illegal operations and potential risks in the first combo sequence, and optimizing and generating a second combo sequence; S3. Optimize the second combo sequence using a reward function, with combo effect, resource consumption, and combo stability as goals, to construct and generate an optimal skill combo sequence.

[0006] In order to solve the above technical problems, another technical solution adopted by the present invention is: A skill combo construction terminal based on hierarchical reinforcement learning includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed: S1. Modeling the state data of the virtual environment using a deep temporal modeling algorithm and generating a sub-goal sequence based on the state data; S2. generating a first combo sequence based on the sub-goal sequence, identifying illegal operations and potential risks in the first combo sequence, and optimizing and generating a second combo sequence; S3. Optimize the second combo sequence using a reward function, with combo effect, resource consumption, and combo stability as goals, to construct and generate an optimal skill combo sequence.

[0007] The beneficial effects of the present invention are: providing a skill combo construction method and terminal based on hierarchical reinforcement learning, which effectively solves the limitations of traditional script testing through hierarchical reinforcement learning and can dynamically adapt to different combat states and skill parameter adjustments. The virtual environment is comprehensively modeled using a deep temporal modeling algorithm, and a low-level sub-target sequence is generated by a high-level strategy to achieve efficient extraction and dynamic adaptability analysis of skill triggering patterns. When generating the first combo sequence, the legality and stability of the combo are verified and optimized through rule constraints and decision search algorithms, thereby effectively avoiding illegal operations and potential risks in skill combos. The combo effect, resource consumption and stability are comprehensively evaluated through the reward function to achieve comprehensive optimization of the skill combo strategy, improve the dynamic adaptability of combo construction, thereby achieving high coverage and effectively improving the efficiency of skill combo testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Flowchart of a method for constructing skill combos based on hierarchical reinforcement learning in an embodiment of the present invention; Figure 2 A schematic diagram of a skill combo construction terminal based on hierarchical reinforcement learning in an embodiment of the present invention; Description of labels: 1. A skill combo construction terminal based on hierarchical reinforcement learning; 2. Memory; 3. Processor. DETAILED DESCRIPTION

[0009] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following description is given in conjunction with the embodiments and accompanying drawings.

[0010] Please refer to Figure 1 , a skill combo construction method based on hierarchical reinforcement learning, comprising the steps of: S1. Modeling the state data of the virtual environment using a deep temporal modeling algorithm and generating a sub-goal sequence based on the state data; S2. generating a first combo sequence based on the sub-goal sequence, identifying illegal operations and potential risks in the first combo sequence, and optimizing and generating a second combo sequence; S3. Optimize the second combo sequence using a reward function, with combo effect, resource consumption, and combo stability as goals, to construct and generate an optimal skill combo sequence.

[0011] As can be seen from the above description, the beneficial effects of the present invention are: providing a skill combo construction method and terminal based on hierarchical reinforcement learning, which effectively solves the limitations of traditional script testing through hierarchical reinforcement learning and can dynamically adapt to different combat states and skill parameter adjustments. The virtual environment is comprehensively modeled using a deep temporal modeling algorithm to achieve efficient extraction and dynamic adaptability analysis of skill triggering patterns. When generating the first combo sequence, the legality and stability of the combo are verified and optimized through rule constraints and decision search algorithms, thereby effectively avoiding illegal operations and potential risks in skill combos. The combo effect, resource consumption and stability are comprehensively evaluated through the reward function to achieve comprehensive optimization of the skill combo strategy, improve the dynamic adaptability of combo construction, thereby achieving high coverage and effectively improving the efficiency of skill combo testing.

[0012] For example, when a boss battle begins in a virtual environment, the following method is used to construct and test the boss's combos. Testing skill combos for boss battles faces issues such as inconsistent skill cooldowns and dynamically changing skill trigger probabilities. Using the method presented in this paper, skill trigger patterns are modeled and extracted using a deep temporal modeling algorithm (e.g., a bidirectional LSTM), enabling a comprehensive analysis of skill cooldowns and trigger rates. After generating the first combo sequence, a reinforcement learning algorithm and a decision search algorithm are used to filter the legitimacy and risk of the skill combo, generating an optimized second combo sequence. Finally, a multi-dimensional reward function is used to comprehensively optimize combo effectiveness, resource consumption, and stability to generate the optimal skill combo sequence. Experiments have shown that compared to traditional testing methods, this method improves coverage by over 30% and reduces generation time by over 50%.

[0013] In some embodiments, step S1 specifically includes the steps of: S11. Modeling the state data of the virtual environment using a deep temporal modeling algorithm and extracting skill triggering patterns from the state data; S12. Converting the skill triggering pattern into a sequence related to skill combinations through a hierarchical strategy architecture; S13. Adjust the priority within the sequence and generate a sub-goal sequence based on the current combat status, skill cooldown status, and operation behavior.

[0014] As can be seen from the above description, by utilizing a deep temporal modeling algorithm and a hierarchical strategy architecture, this method can accurately extract skill triggering patterns and skill combination sequences, enabling comprehensive modeling of skill effects and strategy generation. The deep temporal modeling algorithm, a bidirectional long short-term memory network (BiLSTM), has significant advantages in capturing skill triggering characteristics. It can extract the causal relationship of skill triggering using a forward LSTM and simultaneously capture the subsequent feedback of skill effects using a backward LSTM, forming a comprehensive representation of skill triggering patterns.

[0015] During the feature extraction phase, input data includes boss status characteristics (such as health percentage, skill cooldown status, and player location coordinates) and environmental parameters (such as combat stage thresholds and weather influence coefficients). Using the bidirectional time series modeling capabilities of bidirectional LSTM, dynamic prediction of skill trigger patterns and feature weight analysis are achieved. The feature data format is a three-dimensional tensor (number of samples × number of time steps × number of features), such as (1000, 10, 5), which accurately represents the temporal changes in skill triggering in complex combat scenarios.

[0016] The bidirectional LSTM network architecture consists of two layers of bidirectional LSTM modules. The first layer extracts temporal features, the second layer abstracts high-level patterns, and finally, a fully connected layer outputs the skill trigger probability distribution. The network's loss function uses categorical cross entropy to effectively improve skill prediction accuracy. Model optimization and an action masking mechanism prevent duplicate predictions and false triggering of skills during cooldown, further improving the model's reliability and accuracy.

[0017] During training, a dynamic curriculum learning strategy is employed, gradually transitioning from simple combat scenarios to complex nested scenarios, gradually improving the model's generalization and stability. Furthermore, an action masking mechanism is introduced to prevent repeated use of the same skill during skill cooldown periods, enhancing the rationality of strategy execution. Feature engineering is employed to capture skill trigger patterns through LSTM hidden state vectors. In particular, when health drops below 50%, the hidden state vector significantly favors dimensions corresponding to AoE skills. To address mana deficiency, when mana falls below a threshold for three consecutive time steps, the hidden state prioritizes healing skills to ensure combat endurance.

[0018] Sequence priority adjustment is flexibly controlled using the Option-Critic algorithm. The Option network outputs the sub-goal termination probability, while the Critic network evaluates the long-term benefits of the termination decision. The termination threshold is dynamically adjusted based on environmental conditions (such as boss health and skill cooldown) and training progress (such as current coverage). When coverage is insufficient, the sub-goal duration is extended. When high-risk scenarios (such as mana depletion) are detected, sub-goals are terminated prematurely to avoid ineffective use cases. This dynamic strategy adjustment ensures comprehensive coverage and efficiency of skill combo testing.

[0019] For example, in boss battles, skill trigger patterns were generated using a deep time series modeling algorithm. The data simulation generated 10,000 skill trigger sequences, including features such as health, mana, player distance, skill cooldown, and phase markers. During the skill trigger modeling process, t-SNE dimensionality reduction analysis of latent state features revealed that when health dropped below 50%, the latent states clustered into the area of AOE skills. Weight analysis revealed that the neuron weight corresponding to the health feature in the second LSTM layer was significantly higher than that of other features (approximately 0.7).

[0020] In practical applications, the model successfully predicted skill triggering patterns during boss battle dungeon testing, including prioritizing AOE skills from 50% to 30% health and triggering healing skills when mana drops below 20% and there are no healing skills on cooldown. The model achieved a skill trigger prediction accuracy of 92.3%, significantly outperforming the 78.5% achieved by traditional RNN models. By modeling and extracting features from skill triggering patterns, the accuracy of skill combo strategies and combat stability were effectively improved.

[0021] In some embodiments, step S2 specifically includes the steps of: S21, generating a first combo sequence using a reinforcement learning algorithm according to the sub-goal sequence; S22, filtering illegal operations in the first combo sequence in combination with a rule constraint mechanism; S23, using a decision search algorithm to perform a risk assessment on the first combo sequence and avoid potential risks in the first combo sequence; S24: Generate a second combo sequence based on the rule constraint mechanism and the processing effect of the decision search algorithm.

[0022] As can be seen from the above description, by combining a reinforcement learning algorithm with a rule-based constraint mechanism, skill combo sequences are effectively generated and illegal operations are avoided, ensuring the rationality and compliance of skill combos. The reinforcement learning algorithm is used to generate the first combo sequence, making the skill combo dynamic and flexible. The rule-based constraint mechanism is used to filter combos, significantly reducing the probability of illegal operations in skill combos. A decision-making search algorithm is then used to assess and mitigate potential risks in skill combos, ensuring the feasibility and stability of skill combos in actual operations. Finally, based on the processing results of the rule-based constraint and risk-avoidance mechanism, an optimized second combo sequence is generated, ensuring the efficiency and reliability of skill combos in complex combat scenarios.

[0023] Specifically, by employing the Proximal Policy Optimization (PPO) algorithm to generate the first combo sequence, we effectively address the challenges of low skill combo generation efficiency and lack of strategic flexibility in traditional testing methods. The PPO algorithm, combined with the Action Mask mechanism, generates a binary mask during skill cooldown, preventing illegal operations such as re-casting skills during cooldown, thus ensuring the rationality and compliance of skill combo sequences.

[0024] Furthermore, by incorporating Monte Carlo Tree Search (MCTS) to perform backtracking analysis on high-risk combos, we can detect boundary conditions where the continuous release of mana-intensive skills leads to mana depletion, ensuring the stability and feasibility of combo sequences in complex combat scenarios. MCTS evaluates the value of high-risk combo paths through four phases: path selection, expansion, simulation, and backpropagation. It dynamically updates the Q-value of parent nodes, ensuring the effectiveness and security of strategy optimization.

[0025] Furthermore, in the PPO network architecture, the Actor network generates skill release probability distributions, while the Critic network evaluates the current state value and guides strategy optimization. The action masking mechanism enables real-time filtering of illegal actions based on skill cooldown status. For example, when Fire Wall is on cooldown, its output probability is suppressed, ensuring that the combo sequence complies with operational rules. Combined with the MCTS backtracking mechanism, this further enhances the detection of high-risk paths.

[0026] For example, in a certain MMORPG, the challenges of skill combos in boss battles lie in skill cooldowns and insufficient mana. Using the PPO algorithm and an action mask mechanism, a Fire Wall combo sequence was generated. Input states included the boss's health at 50%, mana at 80%, and a normal skill cooldown. The resulting skill combo sequence was "Fire Wall" → "Meteor" → "Fire Wall," meeting skill cooldown constraints and resource rationality. To detect mana depletion risk, MCTS was used to backtrack the path of five consecutive high-mana-consuming skills (such as Meteor) and detect the boundary condition where mana reaches zero. Using the backpropagation algorithm, the selection priority of this path was lowered to ensure combo stability. The resulting skill combo strategy achieved a 92.3% trigger success rate and 85% resource utilization in combat tests, effectively improving the robustness and rationality of the combo strategy. Through the combined application of PPO and MCTS algorithms, the present invention has significant advantages in skill combo generation and risk detection. It can achieve precise optimization and risk control of skill combo strategies in a dynamic combat environment, and effectively avoid test failure problems caused by skill conflicts and resource exhaustion.

[0027] In some embodiments, step S3 specifically includes the steps of: S31, testing the second combo sequence with the combo effect, resource consumption, and combo stability as the objectives; S32. Optimize the test results based on a multi-dimensional reward function, adjust the reward value according to the combo effect and stability, optimize the second combo strategy, and construct an optimal skill combo sequence.

[0028] As can be seen from the above description, this method ensures the stability and feasibility of skill combos in complex scenarios by optimizing combo effectiveness, resource consumption, and combo stability. By testing the second combo sequence, combo effectiveness data and performance metrics are obtained. This combo effect is then comprehensively optimized using a multi-dimensional reward function, significantly improving the output efficiency and execution stability of skill combinations. The reward function dynamically adjusts to different skill effects and resource allocations, reducing the cost of reconfiguring skill combo strategies after version changes. The resulting optimal skill combo sequence maintains a balance between combo effectiveness and resource utilization in various combat scenarios.

[0029] Specifically, the reward function effectively enhances the accuracy and comprehensiveness of low-level strategies in test case generation by introducing reward and penalty rules, particularly enabling precise optimization of skill combination effects in complex combat scenarios. These reward and penalty rules, through the synergistic effect of the PPO algorithm and the MCTS backtracking mechanism, dynamically adjust the selection weights of combo strategies, ensuring a balanced balance between stability and effectiveness in the generated skill combo sequences.

[0030] The basic reward mechanism improves skill combination coverage through positive incentives. During combo generation, each new skill combination discovered (such as "alternating Fire + Ice") receives a basic reward (+0.1 points) that directly increases the path's priority. The PPO network uses gradient optimization to incentivize the Actor network to generate uncovered skill combinations. In practical applications, for example, when a low-level strategy generates a new combination of "Fire Wall → Meteor → Frostbolt" in a game instance, the Critic network evaluates its coverage gain, significantly increasing the Q value of that path and the probability of subsequent generation. This basic reward mechanism has increased skill combination coverage from 67% to 98%, enabling dynamic optimization of combo strategies.

[0031] Penalties include logical conflicts and performance bottlenecks. The logical conflict penalty (-0.5 points) suppresses contradictory behavior (such as triggering both a buff and a debuff simultaneously) by reducing the priority of offending paths. The performance bottleneck penalty (-0.3 points) triggers a performance warning and reduces the selection weight of the path if the CPU usage of a single combo exceeds 50ms. By using the gradient descent of the PPO network and MCTS backtracking flags, test failures caused by combo conflicts or high resource consumption are reduced.

[0032] Furthermore, an exploration reward mechanism (+0.2 points) drives the PPO network to generate unconventional combos. For example, it provides incentives for skill parameter edge values (e.g., damage value > 200), ensuring comprehensive exploration and coverage of these edge values. This incentive strategy enables the system to discover skill combinations and numerical boundaries that are difficult to cover with traditional methods, ensuring comprehensive skill combos and risk control.

[0033] For example, in a boss battle scenario, the system generates a series of skill combos and monitors their effects and performance in real time. When a "Fire + Lightning" combo is first triggered, the system awards a +0.1 point bonus, increasing the probability of this combo appearing in subsequent combos by 30%. When generating "Frenzy State (Attack +50%) → Healing (Defense +20%)," the system detects that the defense gain overrides the attack gain, incurring a -0.5 point logical conflict penalty and lowering the system's priority. In high-performance overhead combo detection, for example, if "Meteor x 5" causes the CPU to consume 60ms of special effects rendering, a performance bottleneck penalty of -0.3 points is imposed. MCTS then marks this path as inefficient and limits the number of consecutive uses.

[0034] Through the comprehensive application of reward and punishment mechanisms, the method of the present invention effectively improves the efficiency and accuracy of generating skill combo strategies, avoids the problems of redundant combos and resource exhaustion, and demonstrates high stability and flexibility in complex combat environments.

[0035] In some embodiments, step S3 further includes step S4: S4. Automated execution verification is performed based on the optimal skill combo sequence, combo effects and performance indicators are monitored, and the reward function is adjusted based on captured anomalies.

[0036] As can be seen from the above description, automated verification steps ensure the stability and accuracy of optimal skill combo sequences in real-world combat environments. This method can execute skill combos in a virtual environment and monitor their effectiveness, resource consumption, and performance metrics. For abnormal situations (such as combo failures, resource exhaustion, and performance bottlenecks), the system automatically captures and analyzes the exception logs, feeding them back to the reward function optimization module for parameter adjustment. This closed-loop verification mechanism optimizes skill combo strategies while improving the robustness and stability of skill combinations in various scenarios.

[0037] Specifically, step S4 includes: S41, converting the optimal skill combo sequence into an operation instruction stream, and executing the combo operation in a virtual environment; S42. Monitor skill combo effects, resource consumption, and performance stability, and generate a verification report; S43. When capturing abnormal or illegal operations, record the operation log and performance data, and feed it back to the reward function optimization module to adjust the reward weight and combo strategy parameters.

[0038] For example, a digital twin of a boss battle was built within a virtual combat environment. Skill parameters, including cooldowns and trigger conditions, were synchronized using the Unity engine to ensure the skill model was consistent with actual combat data. Skill combo sequences were then transmitted to the test platform in real time at a 60Hz frequency via the gRPC protocol, ensuring the real-time and accuracy of skill triggering.

[0039] The JSON-formatted skill combo sequence generated by the lower layer is then parsed and converted into an executable instruction stream, which is then run in a virtual environment. Assertion rules are set, including damage effect and buff conflict verification, to ensure that the skill combo effect meets expectations. Furthermore, the combo effect is monitored in real time during skill execution. If an anomaly is detected (such as mana depletion or skill conflict), a log is immediately triggered to save and record the operation path. Furthermore, system performance indicators are recorded in real time, including CPU usage (threshold ≤ 50ms / time) and memory peak (threshold ≤ 2GB). If these thresholds are exceeded, an alarm mechanism is triggered.

[0040] Next, skill combination coverage is calculated and scenarios with insufficient coverage are analyzed. Bug reports are categorized and labeled, and recorded according to severity levels (1-3). Detected skill combo bugs (such as skill parameter overflows) are fed back to the reward function module, dynamically updating reward weights. Based on coverage and bug feedback, the reinforcement learning model is retrained. If coverage falls below the expected threshold, the training level is automatically downgraded to improve training stability.

[0041] This involves comprehensively improving the adaptability and stability of skill combo strategies in real-world environments through test case execution and verification. By building an environment and automating execution, we achieve real-time and accurate verification of skill combo operations. Performance monitoring and exception capture mechanisms ensure the stability of skill combo execution, and a feedback mechanism automatically transmits exception and defect information back to the strategy optimization module, forming a closed-loop strategy improvement loop. Version iteration testing ensures the continued effectiveness and reusability of skill combo strategies across game updates through cross-version adaptation and regression verification.

[0042] Please refer to Figure 2 A skill combo construction terminal 1 based on hierarchical reinforcement learning includes a memory 2, a processor 3, and a computer program stored on the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, the steps in a skill combo construction method based on hierarchical reinforcement learning are completed.

[0043] In summary, the present invention provides a method and terminal for constructing skill combos based on hierarchical reinforcement learning. Through the hierarchical reinforcement learning method, it effectively solves the problems of insufficient coverage, poor dynamic adaptability, and low efficiency in use case generation in skill combo testing. Through the hierarchical architecture of high-level decision modules and low-level strategy modules, and utilizing a deep temporal modeling algorithm and a bidirectional LSTM neural network, it achieves accurate modeling and prediction of skill triggering patterns, dynamically adjusts sub-goal generation and combo strategy optimization, and effectively improves the rationality and accuracy of skill combo strategies.

[0044] The dynamic adjustment mechanism of the Option-Critic algorithm ensures the flexibility and effectiveness of test case generation by adjusting sub-goal termination conditions in real time during skill combination generation. The algorithm dynamically adjusts skill combo duration based on current coverage and risk status, effectively avoiding combo failures and resource exhaustion, significantly improving skill coverage. By collaboratively calculating sub-goal termination probabilities and long-term benefits, the sustainability and stability of skill combo strategies are significantly improved.

[0045] In low-level strategy generation, this invention uses a proximal policy optimization (PPO) algorithm to generate initial combo sequences. This, combined with an action mask mechanism, prevents repeated use of illegal skills during skill cooldowns, ensuring the compliance and rationality of combo sequences. Monte Carlo tree search (MCTS) is also introduced to perform backtracking analysis on high-risk combo paths, assessing the resource consumption and stability risks associated with these actions. This allows for real-time adjustment of strategy priorities to ensure the stability and efficiency of combo strategies.

[0046] The introduction of a reward and penalty mechanism has effectively improved the dynamic adjustment capabilities and optimization levels of combo strategies. Regarding the reward mechanism, basic rewards are used to positively incentivize skill combination coverage, driving the model to proactively explore new skill combinations. By exploring reward strategies, we conduct comprehensive testing of skill parameter edge values to ensure the adaptability and robustness of skill combinations under boundary conditions. Regarding the penalty mechanism, we implement negative regulation for logical conflicts and performance bottlenecks. Logical conflict penalties are used to prevent program crashes caused by skill conflicts, while performance bottleneck penalties effectively reduce the frequency of use of high-resource-consuming skills, ensuring the stability of the test environment.

[0047] Overall, the present invention achieves high efficiency, accuracy, and stability in skill combo testing through a hierarchical reinforcement learning architecture and a multi-dimensional optimization mechanism. In practical applications, compared to traditional methods, test coverage has increased from 30% to over 90%, the skill combo strategy generation speed has increased by 8 times, the combo accuracy rate has reached 92.3%, and the resource optimization rate has increased by 40%. Through the comprehensive application of reward and punishment mechanisms and engineering optimization strategies, the reliability and effectiveness of skill combo strategies have been further improved, demonstrating excellent technical advantages and practical application value.

[0048] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A skill combo construction method based on hierarchical reinforcement learning, characterized by: Including steps: S1. Modeling the state data of the virtual environment using a deep temporal modeling algorithm and generating a sub-goal sequence based on the state data; S2. generating a first combo sequence based on the sub-goal sequence, identifying illegal operations and potential risks in the first combo sequence, and optimizing and generating a second combo sequence; S3. Optimize the second combo sequence using a reward function, with combo effect, resource consumption, and combo stability as goals, to construct and generate an optimal skill combo sequence.

2. The method for constructing skill combos based on hierarchical reinforcement learning according to claim 1, characterized in that: The step S1 specifically includes the following steps: S11. Modeling the state data of the virtual environment using a deep temporal modeling algorithm and extracting skill triggering patterns from the state data; S12. Converting the skill triggering pattern into a sequence related to skill combinations through a hierarchical strategy architecture; S13. Adjust the priority within the sequence and generate a sub-goal sequence based on the current combat status, skill cooldown status, and operation behavior.

3. The method for constructing skill combos based on hierarchical reinforcement learning according to claim 1, characterized in that: The step S2 specifically includes the following steps: S21, generating a first combo sequence using a reinforcement learning algorithm according to the sub-goal sequence; S22, filtering illegal operations in the first combo sequence in combination with a rule constraint mechanism; S23, using a decision search algorithm to perform a risk assessment on the first combo sequence and avoid potential risks in the first combo sequence; S24: Generate a second combo sequence based on the rule constraint mechanism and the processing effect of the decision search algorithm.

4. The method for constructing skill combos based on hierarchical reinforcement learning according to claim 1, characterized in that: The step S3 specifically includes the following steps: S31, testing the second combo sequence with the combo effect, resource consumption, and combo stability as the objectives; S32. Optimize the test results based on a multi-dimensional reward function, adjust the reward value according to the combo effect and stability, optimize the second combo strategy, and construct an optimal skill combo sequence.

5. The method for constructing skill combos based on hierarchical reinforcement learning according to claim 1, characterized in that: Step S3 is followed by step S4: S4. Automated execution verification is performed based on the optimal skill combo sequence, combo effects and performance indicators are monitored, and the reward function is adjusted based on captured anomalies.

6. A skill combo construction terminal based on hierarchical reinforcement learning, characterized by: The invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are performed: S1. Modeling the state data of the virtual environment using a deep temporal modeling algorithm and generating a sub-goal sequence based on the state data; S2. generating a first combo sequence based on the sub-goal sequence, identifying illegal operations and potential risks in the first combo sequence, and optimizing and generating a second combo sequence; S3. Optimize the second combo sequence using a reward function, with combo effect, resource consumption, and combo stability as goals, to construct and generate an optimal skill combo sequence.

7. The skill combo construction terminal based on hierarchical reinforcement learning according to claim 6, characterized in that: The step S1 specifically includes the following steps: S11. Modeling the state data of the virtual environment using a deep temporal modeling algorithm and extracting skill triggering patterns from the state data; S12. Converting the skill triggering pattern into a sequence related to skill combinations through a hierarchical strategy architecture; S13. Adjust the priority within the sequence and generate a sub-goal sequence based on the current combat status, skill cooldown status, and operation behavior.

8. The skill combo construction terminal based on hierarchical reinforcement learning according to claim 6, characterized in that: The step S2 specifically includes the following steps: S21, generating a first combo sequence using a reinforcement learning algorithm according to the sub-goal sequence; S22, filtering illegal operations in the first combo sequence in combination with a rule constraint mechanism; S23, using a decision search algorithm to perform a risk assessment on the first combo sequence and avoid potential risks in the first combo sequence; S24: Generate a second combo sequence based on the rule constraint mechanism and the processing effect of the decision search algorithm.

9. The skill combo construction terminal based on hierarchical reinforcement learning according to claim 6, characterized in that: The step S3 specifically includes the following steps: S31, testing the second combo sequence with the combo effect, resource consumption, and combo stability as the objectives; S32. Optimize the test results based on a multi-dimensional reward function, adjust the reward value according to the combo effect and stability, optimize the second combo strategy, and construct an optimal skill combo sequence.

10. The skill combo construction terminal based on hierarchical reinforcement learning according to claim 6, characterized in that: Step S3 is followed by step S4: S4. Automated execution verification is performed based on the optimal skill combo sequence, combo effects and performance indicators are monitored, and the reward function is adjusted based on captured anomalies.

Images