Reinforcement learning based dynamic generation and optimization of game npc behavior

Abstract

The application provides a game NPC behavior dynamic generation and optimization system based on reinforcement learning, and relates to the technical field of computer games.The method comprises the following steps: collecting player position, resource distribution and tribe attribute data in a current game environment to construct a multidimensional environment state space of autonomous NPC tribes; forming an initial behavior strategy through a policy network of a reinforcement learning model based on the multidimensional environment state space; collecting execution effect data of the initial behavior strategy in the game environment, evaluating the policy effect through a value network of the reinforcement learning model to obtain an evaluation result, performing policy optimization iteration based on the evaluation result, and forming an optimized behavior strategy; and dynamically generating a targeted tactical behavior combination according to the optimized behavior strategy, wherein the tactical behavior combination comprises a dynamic ambush strategy and a phased defense breaking strategy.The application can improve the immersion and challenge of the game.

Description

Technical Field

[0001] This invention relates to the field of computer game technology, and in particular to a system for dynamically generating and optimizing game NPC behavior based on reinforcement learning. Background Technology

[0002] In open-world strategy games and multiplayer online battle arenas, the behavioral intelligence of autonomous NPC tribes directly determines the game's immersion and challenge.

[0003] As game environments become more dynamic and player behavior becomes more free, the limitations of traditional methods are becoming increasingly apparent. On one hand, environmental perception is often one-dimensional; most systems rely solely on simple conditions such as whether the player's location triggers boundaries, failing to integrate multi-dimensional data such as resource distribution, tribe attributes, and spatial topology. This results in NPCs having a weak understanding of complex environments. For example, when a player is simultaneously near a resource-rich area and on the edge of a tribe's territory, the NPC cannot adjust its behavioral priorities based on the overall situation. On the other hand, behavioral strategies lack adaptive evolution: initial strategies are manually written by developers and cannot be optimized based on actual effects, such as resource acquisition efficiency and player responses. Furthermore, fixed tactical patterns, such as only one or two preset attack paths, make it difficult to counter diverse player tactics.

[0004] More importantly, traditional methods generally lack group coordination mechanisms and closed-loop update capabilities. Even if multiple autonomous NPC tribes meet the conditions of spatial proximity and resource competition, they cannot form a coordinated defense or joint attack, often resulting in each fighting its own battle and being defeated one by one by the players. At the same time, when the game environment changes dynamically, such as when resources are depleted or player base defenses are upgraded, NPC behavior cannot respond in real time and still uses the initial strategy, resulting in behavior that is out of touch with the environment, which players jokingly call more rigid than robots. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a dynamic generation and optimization system for game NPC behavior based on reinforcement learning, which can enable cross-tribe collaboration and real-time closed-loop response to environmental changes, thereby enhancing the immersion and challenge of the game.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: Firstly, a method for dynamically generating and optimizing game NPC behavior based on reinforcement learning, the method comprising: In the current game environment, by collecting player location, resource distribution, and tribe attribute data, a multi-dimensional environmental state space of autonomous NPC tribes is constructed; Based on a multi-dimensional environmental state space, an initial behavioral policy is formed through a policy network of a reinforcement learning model. Data on the execution effect of the initial behavioral strategy in the game environment is collected. The effect of the strategy is evaluated through the value network of the reinforcement learning model to obtain the evaluation results. Based on the evaluation results, the strategy is optimized and iterated to form an optimized behavioral strategy. Based on the optimized behavior strategy, targeted tactical behavior combinations are dynamically generated. The tactical behavior combinations include dynamic ambush strategies and phased defense penetration strategies. In the phased defense penetration strategy, the central axis of the polygonal shape of the base defense area is extracted, and the weak points of the defense are identified based on the topology of the central axis, so as to plan the phased attack path. When multiple autonomous NPC tribes are detected to meet the cooperative conditions of spatial proximity and resource competition, a tribal cooperative behavior scheme is formed based on optimized behavior strategies. Tactical behavior combinations and tribal collaborative behavior schemes are transformed into behavioral instruction sequences that can be recognized by the game engine. By parsing and executing the instruction sequences, autonomous NPC tribes are driven to complete corresponding behavioral performances. The system monitors the current game environment's operational status in real time. When the environmental state characteristics change, it triggers an update mechanism for the environmental state space. By reconstructing the multi-dimensional environmental state space, a closed-loop behavior optimization process is formed.

[0007] Furthermore, by collecting player location, resource distribution, and tribe attribute data within the current game environment, a multi-dimensional environmental state space for autonomous NPC tribes is constructed, including: A horizontal ray is emitted from the player's current position in the current game environment, and the horizontal ray extends horizontally to the farthest point of the tribal territory boundary polygon. Calculate the number of intersections between the horizontal ray and each boundary of the tribal territory polygon, and record the precise coordinates of the intersection points. Based on the calculated number of intersections, spatial relationships are determined according to the parity rule. That is, when the number of intersections is odd, the player is determined to be inside the tribe's territory; when the number of intersections is even, the player is determined to be outside the tribe's territory, thus obtaining spatial relative positional relationship data. The spatial relative positional relationship data is integrated with player position, resource distribution and tribe attribute data to form a feature set of a multi-dimensional environmental state space; The feature set is normalized and transformed into a standardized state vector that can be recognized by the reinforcement learning model, so as to complete the construction of a multi-dimensional environmental state space.

[0008] Furthermore, based on a multi-dimensional environmental state space, an initial behavioral policy is formed through a policy network of a reinforcement learning model, including: The standardized state vector is input into the policy network of the reinforcement learning model, and state features are extracted through a multi-layer neural network. Based on state characteristics, a behavioral strategy framework is formed that includes resource monitoring intensity, patrol range, and behavior priority. Based on the spatial relative position of the player and the tribe's territory, the parameter configuration in the behavior strategy framework is dynamically adjusted to form a parameter-optimized behavior strategy. By extracting player location data, resource distribution data, and tribe attribute data from the feature set, the optimized behavior strategy is refined and improved to form a complete initial behavior strategy.

[0009] Furthermore, data on the execution effect of the initial behavioral strategy in the game environment is collected. The effect of the strategy is evaluated through the value network of a reinforcement learning model to obtain evaluation results. Based on the evaluation results, the strategy is iterated and optimized to form an optimized behavioral strategy, including: Collect data on the execution effect of the initial behavioral strategy in the game environment. The execution effect data includes the success rate of NPC behavior execution, resource acquisition efficiency, and player interaction response data. The performance data is input into the value network of the reinforcement learning model, and the value evaluation score of the behavioral strategy is obtained through the value network calculation. Based on the value assessment score, the advantage function value of the current behavioral strategy is analyzed and calculated to identify the dimensions of the behavioral strategy that need to be optimized, and the analysis and calculation results of the advantage function value are obtained. Based on the analysis and calculation results of the dominance function value, the parameter configuration of the policy network is updated by using the policy gradient method to obtain the updated policy network; Based on the updated policy network, a new behavioral policy is formed. The newly formed behavioral policy is then iteratively optimized through multiple rounds to obtain an optimized behavioral policy.

[0010] Furthermore, based on the optimized behavioral strategy, targeted tactical behavioral combinations are dynamically generated. These combinations include dynamic ambush strategies and phased breaching strategies. In the phased breaching strategy, the central axis of the polygonal shape of the base's defense area is extracted, and weak points in the defense are identified based on the topological structure of the central axis. This is used to plan phased attack paths, including: Based on optimized behavioral strategies, the tactical objectives that autonomous NPC tribes need to execute and their priority order are determined. Based on the tactical objectives and the phased defense breaching tactical requirements in the current game environment, the polygonal boundary features of the base defense area are extracted. By calculating the central axis of the polygonal shape of the base defense area, a topological representation of the defense structure is established, namely the central axis topological representation. Based on the topological representation of the central axis, the connectivity and node importance of the defense area are analyzed to identify the weak links in the defense system. Based on the identified defensive weaknesses, phased attack paths are planned to determine the attack targets and routes at each stage, ultimately forming a phased defense breaching strategy. By integrating phased defense penetration strategies with dynamic ambush strategies, a complete tactical action combination plan is formed.

[0011] Furthermore, when multiple autonomous NPC tribes are detected to meet the cooperative conditions of spatial proximity and resource competition, a tribal cooperative behavior scheme is formed based on optimized behavioral strategies, including: Based on the optimized behavior strategy, the spatial proximity and resource competition among autonomous NPC tribes are evaluated. When both the preset spatial proximity threshold and the preset resource competition threshold are met, it is determined that there is a cooperative condition among the autonomous NPC tribes. For multiple autonomous NPC tribes that meet the conditions for cooperation, a cross-tribe information sharing mechanism will be established. The information sharing mechanism includes sharing player location information, synchronizing resource distribution status, and reporting tactical execution progress. Based on the shared information in the information sharing mechanism, a collaborative attack scheduling strategy is designed, which includes the coordination of attack time windows, the allocation of attack directions, and the configuration of attack intensity. By integrating information sharing mechanisms with coordinated attack scheduling strategies, a complete tribal coordinated behavior scheme is formed; Conflict detection and resource optimization are performed on tribal collaborative behavior schemes to ensure the coordination and consistency of tribal behaviors in time and space.

[0012] Furthermore, tactical behavior combinations and tribal collaborative behavior schemes are transformed into behavioral command sequences recognizable by the game engine. These command sequences are then parsed and executed to drive autonomous NPC tribes to perform corresponding behaviors, including: Analyze tactical behavior combinations and tribal cooperative behavior schemes to extract the basic behavioral units, which include movement, attack, waiting, and cooperative behavior types. Based on basic behavioral units, a behavior priority mapping relationship is established to assign execution priority and triggering conditions to each behavioral unit; Based on the behavior priority mapping relationship, a sequence of behavior instructions that can be recognized by the game engine is formed. The sequence of behavior instructions includes spatial coordinates, time parameters and execution order. The sequence of behavioral instructions is parsed in real time and converted into underlying control commands to obtain the converted control commands; Execute the transformed control commands to drive the autonomous NPC tribes to perform corresponding behaviors according to the instruction sequence.

[0013] Furthermore, the system monitors the current game environment's operational status in real time. When environmental state characteristics change, it triggers an environment state space update mechanism. By reconstructing a multi-dimensional environment state space, a closed-loop behavior optimization process is formed, including: Continuously monitor key features in the environmental state space, including changes in player location, updates to resource distribution status, and dynamics of tribe relationships, to obtain monitoring results; Based on the monitoring results, the magnitude of change in environmental state characteristics is calculated. When the magnitude of change exceeds a preset threshold, it is marked as a valid environmental state change. Based on changes in the effective environmental state, the environmental state space update mechanism is triggered, and based on the environmental state space update mechanism, the environmental state space reconstruction process is initiated. During the process of reconstructing the environment state space, player location, resource distribution, and tribe attribute data are re-collected to construct a new multi-dimensional environment state space. The new multi-dimensional environmental state space is input into the policy network of the reinforcement learning model, and the behavior policy generation process is restarted to form a closed-loop processing flow that is continuously optimized.

[0014] Secondly, a reinforcement learning-based system for dynamically generating and optimizing game NPC behavior includes: The building module is used to construct a multi-dimensional environmental state space for autonomous NPC tribes in the current game environment by collecting data on player location, resource distribution, and tribe attributes; based on the multi-dimensional environmental state space, an initial behavioral policy is formed through a policy network of a reinforcement learning model. The iteration module is used to collect data on the execution effect of the initial behavior strategy in the game environment, evaluate the effect of the strategy through the value network of the reinforcement learning model, obtain the evaluation results, and perform strategy optimization iteration based on the evaluation results to form an optimized behavior strategy. The planning module is used to dynamically generate targeted tactical behavior combinations based on the optimized behavior strategy. The tactical behavior combinations include dynamic ambush strategy and phased defense penetration strategy. In the phased defense penetration strategy, the central axis of the polygonal shape of the base defense area is extracted, and the weak points of the defense are identified based on the topology of the central axis, so as to plan the phased attack path. The detection module is used to form a tribal collaborative behavior scheme based on optimized behavior strategies when multiple autonomous NPC tribes are detected to meet the collaborative conditions of spatial proximity and resource competition. The execution module is used to transform tactical behavior combinations and tribal collaborative behavior schemes into a sequence of behavioral instructions that the game engine can recognize. By parsing and executing the instruction sequence, it drives autonomous NPC tribes to complete the corresponding behavioral performance. The trigger module is used to monitor the current game environment's running status in real time. When the environmental state characteristics change, it triggers the environmental state space update mechanism, and reconstructs the multi-dimensional environmental state space to form a closed-loop behavior optimization process.

[0015] Thirdly, a computing device, comprising: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0016] Fourthly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0017] The above-described solution of the present invention has at least the following beneficial effects: Employing multi-dimensional environmental state space construction technology, integrating player location, resource distribution, tribe attributes, and relative spatial position, this approach overcomes the limitations of traditional methods, such as single-dimensional environmental perception and weak understanding of complex environments, enabling NPCs to achieve comprehensive and accurate perception of the game environment. A reinforcement learning-driven strategy generation and optimization mechanism is used, with a strategy network generating initial strategies, a value network evaluating effects, and a strategy gradient iteratively updating, overcoming the static and rigid nature of traditional behavioral strategies and their inability to adapt and evolve based on execution results, allowing for dynamic iterative optimization of NPC behavioral strategies. Dynamic tactical combination generation technology is employed, combining the topology of defense areas to plan phased breaching paths and integrating dynamic ambush strategies, overcoming the fixed tactical patterns and single attack paths of traditional methods, enabling flexible generation of targeted tactics. A tribe collaboration mechanism is adopted, determining collaboration conditions based on spatial proximity and resource competition, establishing cross-tribe information sharing and attack scheduling, overcoming the problem of traditional multiple autonomous NPC tribes fighting independently and lacking group collaboration, forming collaborative defense and joint attack capabilities among tribes. A closed-loop update process is used, monitoring environmental state changes in real time, triggering state space reconstruction and strategy regeneration, overcoming the problems of lagging environmental response and disconnect between behavior and environment in traditional methods, achieving real-time matching of NPC behavior and the dynamic environment. By integrating and applying the above-mentioned technical means, this invention comprehensively enhances the behavioral intelligence and adaptability of autonomous NPC tribes, effectively increasing the immersion and challenge of the game. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the method for dynamically generating and optimizing game NPC behavior based on reinforcement learning, provided in an embodiment of the present invention.

[0019] Figure 2This is a schematic diagram of a game NPC behavior dynamic generation and optimization system based on reinforcement learning provided by an embodiment of the present invention. Detailed Implementation

[0020] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0021] like Figure 1 As shown, embodiments of the present invention propose a method for dynamic generation and optimization of game NPC behavior based on reinforcement learning, the method comprising the following steps: Step 1: In the current game environment, construct a multi-dimensional environmental state space for autonomous NPC tribes by collecting player location, resource distribution, and tribe attribute data; Step 2: Based on the multi-dimensional environmental state space, an initial behavioral policy is formed through the policy network of the reinforcement learning model; Step 3: Collect data on the execution effect of the initial behavior strategy in the game environment, evaluate the effect of the strategy through the value network of the reinforcement learning model, obtain the evaluation results, and perform strategy optimization iteration based on the evaluation results to form an optimized behavior strategy; Step 4: Based on the optimized behavior strategy, dynamically generate targeted tactical behavior combinations. The tactical behavior combinations include dynamic ambush strategy and phased defense penetration strategy. In the phased defense penetration strategy, the central axis of the polygonal shape of the base defense area is extracted, and the weak points of the defense are identified based on the topology of the central axis, so as to plan the phased attack path. Step 5: When multiple autonomous NPC tribes are detected to meet the cooperative conditions of spatial proximity and resource competition, a tribal cooperative behavior plan is formed based on the optimized behavior strategy. Step 6: Transform the tactical behavior combinations and tribal collaborative behavior schemes into a sequence of behavioral instructions that the game engine can recognize. By parsing and executing the instruction sequences, the autonomous NPC tribes are driven to complete the corresponding behavioral performances. Step 7: Monitor the current game environment's running status in real time. When the environmental state characteristics change, trigger the environmental state space update mechanism. By reconstructing the multi-dimensional environmental state space, a closed-loop behavior optimization process is formed.

[0022] In this embodiment of the invention, by constructing a multi-dimensional environmental state space, autonomous NPC tribes can comprehensively integrate and perceive key environmental information such as player location, resource distribution, and tribe attributes, overcoming the limitations of traditional single-dimensional perception and improving their understanding of complex game environments. Utilizing the policy and value networks of a reinforcement learning model, initial behavioral strategies are scientifically generated and dynamically optimized based on execution results. This prevents NPC behavioral strategies from becoming fixed and rigid, allowing them to continuously improve with actual interaction effects, enhancing the effectiveness and rationality of behavioral execution. Furthermore, by dynamically generating tactical behavioral combinations including dynamic ambush and phased defense breaches, and combining this with the base defense area topology to plan attack paths, tactics are enhanced. It is targeted and flexible, avoiding the problem that traditional fixed tactical patterns cannot cope with the diverse strategies of players; it forms a collaborative behavior plan for multiple autonomous NPC tribes that meet the conditions for cooperation, promoting the tribes to shift from fighting alone to cooperating, improving the group combat capability, and increasing the sense of layering and challenge of game combat; it transforms tactics and collaborative plans into instruction sequences that the game engine can recognize, ensuring that the strategy can be effectively implemented into the actual behavior of NPCs, and ensuring the feasibility of the method; by monitoring changes in environmental state in real time and triggering state space updates, a closed-loop optimization process is formed, so that NPC behavior can respond to the dynamics of the game environment in real time, avoiding the disconnect between behavior and environment, and always maintaining adaptability to the current game scene.

[0023] In a preferred embodiment of the present invention, step 1 above may include: Step 1.1: In the current game environment, a horizontal ray is emitted from the player's current position. The horizontal ray extends horizontally to the farthest point of the tribal territory boundary polygon. Specifically, this includes: First, obtaining the player character's current precise 3D coordinates in the game world coordinate system. This coordinate point serves as the starting point for the horizontal ray. The emission direction of the horizontal ray is strictly set along the horizontal direction of the game world coordinate system, specifically parallel to the X-axis of the coordinate system, thus eliminating the influence of vertical height (Z-axis) differences on the ray detection results. The endpoint of the ray's extension is set to the farthest point of the tribal territory boundary polygon. To achieve this, the 3D coordinates of all vertices of the tribal territory boundary polygon in the game world coordinate system must be obtained first. Then, by calculating the straight-line distance between each vertex and the player's current coordinate point in the horizontal direction (X-axis and Y-axis), the vertex with the largest distance is selected, and its coordinates are used as the endpoint of the ray's extension, ensuring that the ray can completely cover the area from the player's position to the farthest point of the tribal territory boundary.

[0024] Step 1.2: Calculate the number of intersections between the horizontal ray and each boundary of the tribal territory polygon, and record the precise coordinate data of the intersection points. Specifically, this includes: obtaining all boundary segments contained in the polygon constituting the tribal territory boundary. Each boundary segment is determined by the three-dimensional coordinates of two adjacent vertices of the polygon in the game world coordinate system. For example, if the vertices of the polygon are arranged in order as vertex 1, vertex 2, vertex 3 up to vertex n, then the boundary segments include the segment connecting vertex 1 and vertex 2, the segment connecting vertex 2 and vertex 3, up to the segment connecting vertex n and vertex 1. Next, based on the geometric intersection judgment logic, calculate the intersection of the emitted horizontal ray with each boundary segment one by one. During the calculation, the intersection points of the ray's backward extension line (i.e., the direction extending from the endpoint to the starting point) with the boundary segments must be excluded. Only valid intersection points in the direction extending from the player's position to the farthest end of the tribal territory boundary are retained. Count the total number of all valid intersection points and record the precise three-dimensional coordinates of each valid intersection point in the game world coordinate system. The precision of the coordinate data must match the smallest calculation unit of the game engine, generally at the centimeter level.

[0025] Step 1.3: Based on the calculated number of intersections, spatial relationships are determined according to the parity rule. When the number of intersections is odd, the player is determined to be inside the tribe's territory; when the number of intersections is even, the player is determined to be outside the tribe's territory. This yields spatial relative positional relationship data. Specifically, based on the statistically significant number of valid intersections, the parity rule is applied to determine the spatial relative positional relationship between the player and the boundary polygon of the tribe's territory. This rule is applicable to irregular polygonal territory boundary scenes common in open-world games and can reliably determine the inclusion relationship between points and polygons. The specific judgment logic is as follows: when the number of valid intersections is odd... This indicates that the number of times the horizontal ray crosses the boundary of the tribe's territory polygon after originating from the player's position is odd. According to geometric principles, this means the player's current position is inside the tribe's territory polygon. When the number of valid intersections is even, the number of times the ray crosses the boundary is even, meaning the player's current position is outside the tribe's territory polygon. These results are then linked to the player's current coordinates to form spatial relative positional relationship data that includes the player's position and the spatial relationship between the tribe's territory. For example, player A's current coordinates correspond to being inside tribe B's territory, and player C's current coordinates correspond to being outside tribe D's territory.

[0026] Step 1.4 integrates the spatial relative positional relationship data with player position, resource distribution, and tribe attribute data to form a feature set of a multi-dimensional environmental state space. Specifically, this includes: first, determining the specific content of each type of data to be integrated; player position data, in addition to the player's current 3D coordinates, also includes the player character's movement speed and direction, with the movement direction represented by angle values ​​in the game world coordinate system; resource distribution data covers the 3D coordinates, resource type, remaining quantity, and refresh time of all resource points in the game environment, with resource types specifically including iron ore, wood, and energy; tribe attribute data includes the current population, military strength level, various resource reserves, and territory size of the autonomous NPC tribe, with military strength... The quantity level is determined by the number of soldiers and their equipment level within the tribe; the reserves of various resources correspond to the resource types in the resource distribution data, and the size of the territory is represented by the area of ​​the boundary polygon; then, the spatial relative position relationship data is used as the core correlation dimension, and player location data, resource distribution data, and tribe attribute data are matched and integrated with this dimension; for example, when the spatial relative position relationship is that the player is located inside the tribe's territory, the player's movement direction, the distribution and remaining quantity of resource points within 500 meters of the player, and the tribe's current military strength level are simultaneously correlated; finally, a feature set of a multi-dimensional environmental state space is formed, in which each data unit contains complete correlation content of four types of information: spatial relative position, player status, resource status, and tribe status.

[0027] Step 1.5 involves normalizing the feature set to transform it into a standardized state vector that can be recognized by the reinforcement learning model, thereby completing the construction of a multi-dimensional environmental state space. Specifically, this includes analyzing the magnitude differences of various types of data in the feature set; the player's movement speed typically ranges from 0 to 10 meters per second. The number of tribe members typically ranges from 0 to 1000, while the remaining resources typically range from 0 to 10000 units. The difference in magnitude between different data types can lead to an imbalance in data weights when inputting into a reinforcement learning model, affecting the model's accurate analysis of features. To address this issue, all data in the feature set needs to be normalized. First, the theoretical range of each data type within the game's settings is determined. The theoretical minimum for player movement speed is 0 meters per second, and the maximum is 10 meters per second. The theoretical minimum for tribe population is 0, and the maximum is 1000. Then, the current actual values ​​of each data type are converted to standardized values ​​between 0 and 1 using a uniform ratio. This conversion process ensures that the relative size of the data remains unchanged, eliminating only the difference in magnitude. Finally, all the normalized values ​​are arranged in a fixed order: spatial relative position data, player position data, resource distribution data, and tribe attribute data, forming an ordered numerical sequence. This sequence is the standardized state vector that the reinforcement learning model can directly read and analyze, thus completing the construction of the multi-dimensional environmental state space.

[0028] The specific construction and training process of the reinforcement learning model is as follows: The construction of the reinforcement learning model begins with network structure design to adapt to the processing requirements of a multi-dimensional environmental state space. First, the policy network adopts a multi-layer neural network architecture. The input layer dimension is consistent with the normalized state vector dimension, receiving a normalized numerical sequence containing spatial relative position, player state, resource state, and tribe state. The hidden layer uses a three-layer non-linear transformation structure, strengthening the correlation between features of different dimensions through activation functions, such as cross-mapping player movement direction with resource point distribution features. The output layer outputs feature vectors related to the behavioral policy framework, providing a foundation for forming core dimensions such as resource monitoring intensity, patrol range, and behavior priority. This structural design ensures that the policy network can extract key decision features from complex environmental states. The value network and policy network adopt a matched multi-layer architecture. The input layer receives execution effect data, including indicators such as NPC behavior execution success rate, resource acquisition efficiency, and player interaction response. The hidden layer uses weighted calculations to highlight the influence weight of different effect indicators; for example, the weight of player interaction response data is higher than that of simple behavior success rate. The output layer outputs a value evaluation score from 0 to 100, realizing a quantitative evaluation of the behavioral policy effect.

[0029] The training process of the reinforcement learning model is deeply integrated with the generation and optimization process of the behavioral policy. In the initial training phase, the constructed first multi-dimensional environment state space is input into the policy network to generate the initial behavioral policy and execute it in the game environment. Simultaneously, execution effect data is collected and input into the value network to obtain the initial value evaluation score. Based on this score, the initial advantage function value is calculated, and the performance difference between the current policy and the random policy is compared to identify directions that need to be prioritized for optimization, such as resource monitoring dimensions. Then, the weight parameters of the corresponding dimensions in the policy network are adjusted using the policy gradient method, such as increasing the weight ratio of resource distribution data in the hidden layer, completing the first round of parameter updates.

[0030] Entering the iterative training phase, each training round is based on multiple rounds of iterative optimization. The newly generated behavioral strategies from the updated policy network are implemented, and the process of collecting performance data, calculating value scores, and analyzing dominance function values ​​is repeated. As iterations progress, the parameters of the policy network are continuously adjusted towards better performance. For example, when players frequently avoid fixed patrol routes, dominance function analysis will reveal deficiencies in the patrol range dimension. The weights of parameters related to player movement trajectory features in the policy network will be increased, making the generated patrol range more dynamically adaptable. Continuous optimization of training is achieved through a closed-loop processing flow. When changes in the effective environmental state are detected and a new multi-dimensional environmental state space is reconstructed, the new state space is input into the policy network and value network, triggering model retraining. At this time, the model not only adjusts the policy generation logic based on the new environmental data but also recalibrates the evaluation criteria through the value network. For example, in resource-depleted scenarios, the weight of the value network on resource acquisition efficiency will decrease, while the weight on the success rate of defensive actions will increase, ensuring that model training is always synchronized with the dynamic changes in the game environment. Ultimately, this enables the reinforcement learning model to continuously generate behavioral strategies adapted to complex dynamic environments, improving the behavioral intelligence of autonomous NPC tribes.

[0031] In this embodiment of the invention, by emitting a horizontal ray from the player's position and calculating the number of intersections with the polygon of the tribe's territory boundary, combined with parity determination rules, the spatial relative positional relationship between the player and the tribe's territory can be accurately determined. Compared with the traditional simple boundary trigger judgment method, this significantly improves the accuracy of positional perception, allowing NPCs to more accurately identify the area where the player is located. Integrating this spatial relative positional relationship data with player position, resource distribution, and tribe attribute data into a feature set further enriches the dimensions of the environmental state space, breaking the limitations of traditional single-dimensional perception, and enabling NPCs to understand the current game environment by comprehensively considering multiple types of key information. By normalizing the feature set to obtain a standardized state vector, it ensures that the multi-dimensional data format is unified and the scale is adapted, enabling it to be effectively identified and processed by the reinforcement learning model, providing reliable basic data support for generating reasonable behavior strategies based on the model.

[0032] In a preferred embodiment of the present invention, step 2 above may include: Step 2.1 involves inputting the standardized state vector into the policy network of the reinforcement learning model. This is achieved through a multi-layer neural network to extract state features. Specifically, the standardized state vector is input into the policy network of the reinforcement learning model. This policy network consists of a multi-layer neural network with an input layer, hidden layers, and an output layer. The input layer receives all numerical sequences from the standardized state vector and passes them to the hidden layer. The hidden layer performs a non-linear transformation on the input vector data using the ReLU activation function, gradually filtering redundant information and strengthening the association of key features. The output layer outputs a set of state features based on the processing results of the hidden layer, containing key information about various dimensions of the game environment. These state features cover the spatial correlation between the player and the tribe's territory, the distribution density and remaining quantity of various resources, and the adaptation of the tribe's own attributes to the surrounding environment.

[0033] Step 2.2: Based on state characteristics, a behavioral strategy framework is formed, including resource monitoring intensity, patrol range, and behavior priority. Specifically, this includes: Based on the extracted state characteristics, the behavioral strategy framework is constructed. The core components of this framework include resource monitoring intensity, patrol range, and behavior priority. The initial setting of resource monitoring intensity refers to resource distribution data and tribe resource reserves in the state characteristics. When a certain type of resource is densely distributed around the tribe's territory and the tribe's reserves of that type of resource are low, the monitoring intensity of the corresponding resource is initially set to a higher level. The initial boundary of the patrol range is determined based on the size of the tribe's territory and the spatial distance between the player and the territory in the state characteristics. Typically, the initial patrol range is formed by extending a certain distance outward from the tribe's territory boundary. The initial ranking of behavior priorities combines the player threat level in the state characteristics. The player threat level is comprehensively judged by characteristics such as the distance between the player's location and the territory, and the player's equipment level, as well as the tribe's current core needs. The tribe's current core needs are determined by characteristics such as population size and resource gap in the tribe's attributes. For example, when the player threat level is high, the priority of attack behavior is initially set higher than resource gathering behavior. Finally, the initial settings of the above three components are integrated to form the basic behavioral strategy framework.

[0034] Step 2.3: Based on the spatial relative position of the player and the tribe's territory, dynamically adjust the parameter configuration in the behavior strategy framework to form a parameter-optimized behavior strategy. Specifically, this includes: acquiring data on the spatial relative position of the player and the tribe's territory, and dynamically adjusting the parameter configuration in the behavior strategy framework based on this data; if the spatial relative position is that the player is inside the tribe's territory, it indicates that the player poses a direct threat to the tribe. In this case, adjust the coverage density of the patrol range to set one patrol observation point every 10 meters, with an initial density of one every 20 meters. At the same time, reduce the monitoring intensity of non-core resources to concentrate computing power on the player's dynamics, and increase the priority of attack response behavior by two levels, for example, from third to first; if the spatial relative position is that the player is outside the tribe's territory and is more than a preset safe distance from the territory boundary, extend the patrol range 50 meters outside the territory to expand the alert range, increase the resource monitoring intensity of resource-rich areas around the territory, and appropriately increase the priority of resource collection behavior in the behavior priority. Through the above parameter adjustments for different spatial positions, a parameter-optimized behavior strategy is formed.

[0035] Step 2.4 involves refining and improving the optimized behavioral strategy by extracting player location data, resource distribution data, and tribe attribute data from the feature set, forming a complete initial behavioral strategy. Specifically, this includes extracting player location data, resource distribution data, and tribe attribute data from the feature set. Player location data focuses on extracting the player's real-time movement direction and speed; resource distribution data focuses on extracting the specific coordinates and remaining resource quantities of various resource points; and tribe attribute data focuses on extracting the tribe's current number of soldiers and their equipment levels. Using the extracted player movement direction data, the patrol range is refined. For example, when a player moves eastward towards the territory boundary, the patrol route will be adjusted accordingly. The key area of ​​the patrol route was adjusted to the eastern boundary of the territory; based on the specific coordinates and remaining quantity of resource points, a dedicated monitoring frequency was set for resource points with high remaining quantity in the resource monitoring intensity parameter, such as scanning once every 30 seconds, and once every 60 seconds for ordinary resource points; referring to the number of soldiers and equipment level of the tribe, the intensity of the attack behavior parameter in the behavior priority was refined, for example, when the number of soldiers exceeds 100 and the equipment level reaches level 3, the execution intensity of the attack behavior is set to full attack mode, and if the number of soldiers is less than 50, it is set to restraint attack mode. Through the refinement and improvement of each parameter, a complete initial behavior strategy including specific rules for each link such as resource monitoring, patrol, and behavior execution is finally formed.

[0036] In this embodiment of the invention, standardized state vectors are input into the policy network of a reinforcement learning model. A multi-layer neural network is used to accurately extract key state features adapted to the game environment, providing a reliable foundation for constructing behavioral strategies and avoiding the problem of incomplete capture of complex information in traditional feature extraction. Based on these state features, a policy framework is formed that includes resource monitoring intensity, patrol range, and behavioral priority, giving NPC behavioral strategies a structured core dimension and overcoming the limitations of fragmented behavioral logic in traditional methods. The framework parameters are dynamically adjusted according to the spatial relationship between the player and the tribe's territory. For example, patrol density is increased and attack response level is raised when the player is within the territory, while resource monitoring intensity is optimized when the player is outside the territory, allowing the policy parameters to flexibly adapt to the scenario and enhance their targeting. By extracting player location, resource distribution, and tribe attribute data, the optimized strategy is refined and improved. For example, patrol focus is adjusted based on player movement direction, and monitoring priority is set based on resource density, making the initial behavioral strategy more realistic and avoiding the problem of poor generalization in traditional manually written strategies. These steps progress step by step, gradually forming a scientific and reasonable initial behavioral strategy, improving its adaptability to the dynamic game environment and its execution effectiveness.

[0037] In a preferred embodiment of the present invention, step 3 above may include: Step 3.1: Collect execution effect data of the initial behavioral strategy in the game environment. This data includes NPC behavior execution success rate, resource acquisition efficiency, and player interaction response data. Specifically, this involves: collecting three core execution effect data in real time during the execution of the initial behavioral strategy in the game environment; NPC behavior execution success rate is calculated separately for each specific behavior included in the strategy, such as resource gathering, patrol and alert, and attack response. For example, the success standard for resource gathering is successfully reaching a resource point and completing one gathering action; the success standard for patrol and alert behavior is covering the entire patrol area within a preset time without missing any player entry events. The ratio of the number of successful executions per unit time to the total number of executions for each behavior is calculated as the percentage of successful executions. The success rate of action execution; resource acquisition efficiency is calculated using a 10-minute in-game statistical period, comparing the total amount of various resources actually obtained by the autonomous NPC tribe through resource gathering actions within that period with the total amount of resource acquisition targets planned in the strategy for that period. This is combined with the NPC unit time cost consumed in resource gathering to form a comprehensive resource acquisition efficiency data; player interaction reaction data is obtained by capturing changes in the player character's behavior when interacting with the autonomous NPC tribe. For example, when the NPC launches an attack, whether the player switches defensive equipment, calls for ally support, or adjusts the base's defensive layout, etc., are all recorded as player interaction reaction data. Finally, these three types of data are integrated to form a complete set of execution effect data.

[0038] Step 3.2 involves inputting the execution effect data into the value network of the reinforcement learning model. The value network calculates the value assessment score of the behavioral strategy. Specifically, this includes: inputting the execution effect data set into the value network of the reinforcement learning model. This value network employs a multi-layer neural network architecture that matches the policy network structure. The input layer receives various specific indicators from the execution effect data and converts them into numerical forms that the network can process. The hidden layer uses a linear activation function to weight the data, assigning different weight coefficients to NPC behavior execution success rate, resource acquisition efficiency, and player interaction reaction data. The weight coefficients for player interaction reaction data are set according to the degree of impact of player behavior changes on the game process; for example, the reaction weight for a player calling for ally support is higher than the reaction weight for switching defensive equipment. The output layer, based on the weighted calculation results from the hidden layer, outputs a value assessment score from 0 to 100. A higher score indicates a better overall performance of the current behavioral strategy in the game environment. 60 points is the basic passing score for the strategy; a score below 60 points indicates significant flaws in the strategy that require optimization. This score provides an objective quantitative assessment of the effectiveness of the behavioral strategy.

[0039] Step 3.3: Based on the value assessment score, the dominance function value of the current behavioral strategy is analyzed and calculated to identify the dimensions of the behavioral strategy that need to be optimized. The analysis and calculation results of the dominance function value are obtained, specifically including: calculating the dominance function value of the current behavioral strategy based on the value assessment score. This function value is used to measure the performance difference of the current strategy compared to historical iteration strategies in each dimension. During the calculation, a benchmark value is first determined, usually the average of the value assessment scores of the previous three iteration strategies. Then, the difference between the current strategy's value assessment score and the benchmark value is calculated. If the difference is positive, the dominance function value is positive, indicating that the current strategy is generally superior to the historical strategy. The strategy involves analyzing the advantage function value. If the difference is negative, the advantage function value is negative, indicating a regression in the current strategy. Furthermore, the advantage function value is broken down and analyzed according to different dimensions of the behavioral strategy, such as resource monitoring, patrol range, and attack response. By comparing the proportion of each dimension's sub-advantage function value to the overall advantage function value, the dimension with the greatest impact on the overall score is identified. If the sub-advantage function value corresponding to resource acquisition efficiency is negative and its proportion exceeds 40%, then the resource monitoring intensity dimension is determined to be the dimension that needs priority optimization. The final result is an analysis and calculation of the advantage function value, including the priority optimization dimension, the second-best optimization dimension, and the optimization direction for each dimension.

[0040] Step 3.4: Based on the analysis and calculation results of the dominance function value, the parameter configuration of the strategy network is updated using the strategy gradient method to obtain the updated strategy network. Specifically, this includes: updating the strategy network parameter configuration using the strategy gradient method based on the analysis and calculation results of the dominance function value; adjusting the weight parameters of the identified priority optimization dimensions in the strategy network. For example, if the priority optimization dimension is resource monitoring intensity, and the analysis results show that the current monitoring intensity is insufficient, resulting in low resource acquisition efficiency, then the weight coefficient of resource distribution data in the hidden layer of the strategy network is increased, while the weight coefficient of non-core dimensions, such as patrol route details, is decreased to balance network computing resources; the magnitude of the parameter update is determined by the absolute value of the dominance function value. The larger the absolute value of the dominance function value, the larger the magnitude of the parameter adjustment. For example, when the dominance function value of the resource monitoring dimension is -20 (larger absolute value), the weight coefficient of this dimension is increased by 20%, and when the dominance function value is -5 (smaller absolute value), the weight coefficient is increased by only 5%; through the above targeted parameter adjustments, a strategy network with updated parameter configuration is obtained to ensure that the network can better adapt to the game environment's needs for strategy.

[0041] Step 3.5: Based on the updated policy network, a new behavioral policy is formed. This new policy is then iteratively optimized through multiple rounds to obtain an optimized behavioral policy. Specifically, this involves: re-inputting the standardized state vector into the updated policy network; through multi-layer neural processing, generating a new set of behavioral policies that include core dimensions such as resource monitoring intensity, patrol range, and behavior priority; executing this new behavioral policy in the game environment; and repeating the process from Step 3.1 to Step 3.4, which involves collecting data on the execution effect of the new policy, inputting it into the value network to calculate the evaluation score, analyzing the advantage function value to identify optimization dimensions, and updating the policy network parameters to complete one complete iterative optimization. This process is repeated for multiple rounds. After each round, the value evaluation score of the current policy is compared with that of the previous policy. When the score fluctuation range of three consecutive rounds is less than 5 points and the score is stable above 85 points, the iteration process is stopped. The behavioral policy generated in the last round is the optimized behavioral policy.

[0042] In this embodiment of the invention, by collecting execution effect data of the initial behavioral strategy, covering multiple aspects such as NPC behavior execution success rate, resource acquisition efficiency, and player interaction response, the performance of the strategy in the actual game environment can be comprehensively reflected, avoiding the one-sidedness of traditional methods that rely solely on local feedback to judge the merits of the strategy. Inputting this data into the value network of the reinforcement learning model, the calculated value assessment score quantifies the strategy's effectiveness, transforming the evaluation of the strategy from subjective judgment to objective data support, thus improving the accuracy of the evaluation. Analyzing the dominance function value based on the value assessment score can accurately identify dimensions in the behavioral strategy that need optimization, such as insufficient resource monitoring intensity or unreasonable patrol range, avoiding inefficiency caused by blind adjustments. The strategy gradient method is used to update the strategy network parameters based on the analysis results, enabling the network to specifically improve its shortcomings and enhance its adaptability to the game environment. Through multiple iterations, the behavioral strategy is reformulated and optimized, allowing it to continuously evolve with actual execution results, breaking free from the limitations of a fixed initial strategy. The final optimized behavioral strategy is more aligned with the dynamic game scenario, effectively improving the rationality and adaptability of NPC behavior.

[0043] In a preferred embodiment of the present invention, step 4 above may include: Step 4.1, based on the optimized behavior strategy, determine the tactical objectives and their priority order for the autonomous NPC tribe. Specifically, this includes: first, extracting the core guidelines for the autonomous NPC tribe's actions from the optimized behavior strategy, including potential directions such as resource contention, territory expansion, and player base suppression; then, analyzing the specific tactical objectives corresponding to each direction in light of the current game environment. For example, when resource acquisition has a high priority in the optimized behavior strategy and the player base stores a large amount of target resources, breaking through the player base's defenses and seizing the core resource repository will be set as the key tactical objective; when the tribe's territory is frequently harassed by players, clearing... The tactical objectives are: first, to patrol the area around the territory and establish an outer perimeter security zone; then, to prioritize the tactical objectives based on their urgency, difficulty of implementation, and impact on the tribe's development. Urgency is determined by the timeliness of the threat posed by player actions to the tribe. For example, when a player is about to launch an attack, defensive tactical objectives take precedence over resource objectives. Difficulty of implementation is assessed in conjunction with the tribe's current troop strength and equipment level. Impact weight is based on the tribe's short-term development needs. For example, when resources are scarce, resource objectives have a higher weight. Finally, the tactical objectives and their priority order are determined. For example, the first priority is to break through the player's base defense and seize resources, and the second priority is to establish an outer perimeter security zone around the territory.

[0044] Step 4.2: Based on the tactical objectives and the phased breaching tactics required in the current game environment, extract the polygonal boundary features of the base defense area. Calculate the central axis of the polygonal shape of the base defense area to establish a topological representation of the defense structure, i.e., the central axis topological representation. Specifically, this includes: Based on the core tactical objectives, if the objective involves attacking the player base, then, for the phased breaching tactics, first obtain the 3D model data of the player base defense area, extracting the polygonal boundary features constituting the defense area, including the vertex coordinates of the boundaries, the length and tilt angle of each side, and the distribution of defensive facilities such as walls and turrets on the boundaries; based on these boundary features, calculate the central axis of the polygonal shape of the base defense area. During the calculation, first use a geometric algorithm to find the skeleton line of the polygon, i.e., the line connecting all points inside the polygon and equidistant from the boundary, then smooth the skeleton line, removing redundant branches caused by boundary irregularities, retaining the main line running through the core of the defense area and key branches, forming the central axis of the base defense area; based on this central axis, mark the key nodes on the axis, such as axis intersections, endpoints connecting defensive facilities, and the connection relationships of each segment of the axis, establishing a topological representation of the defense structure, i.e., the central axis topological representation.

[0045] Step 4.3, based on the central axis topological representation, analyze the connectivity and node importance of the defense area to identify weak links in the defense system. Specifically, this includes: analyzing the connectivity and node importance of the defense area based on the established central axis topological representation; connectivity analysis mainly examines whether the connections between each segment of the central axis are continuous, whether there are breaks or segments with only a single connection. For example, if a segment of the axis is connected to other axes only through one node, the defense area corresponding to that segment is prone to becoming an isolated area after the connection is severed, indicating weak connectivity; node importance analysis involves statistically analyzing the number of axes connected to each node, the strength of the surrounding defense facilities such as the number of turrets, the thickness of the walls, and the position of the node in the overall defense structure, such as whether it is located at the connection node between the core resource repository and the outside. The more connecting axes, the weaker the surrounding defense facilities, and the more critical the node's supporting role in the defense system, the less important the node is; conversely, the weaker the node is, the lower its importance. By combining areas with weak connectivity and nodes with low importance, weak links in the defense system are identified, such as wall segments where connectivity is broken, and areas near axis branch nodes with low importance and few surrounding defense facilities.

[0046] Step 4.4: Based on the identified defensive weaknesses, plan phased attack paths to determine the attack targets and routes for each phase, ultimately forming a phased defense breaching strategy. Specifically, this includes: planning phased attack paths based on the identified defensive weaknesses, proceeding from the outside in and gradually advancing; the first phase target is the outermost weak point, such as a section of the wall with weak connectivity at the edge of the base's defensive area. The planned route starts from the tribe's rally point, avoiding known patrol routes and strong defensive structures, such as high-level turret clusters, moving along terrain-concealed areas, such as forests and depressions, towards the target wall section, ensuring exposure during the approach. The probability is the lowest. After the first phase objective is achieved, if the target wall section is destroyed and a breach is opened, the second phase attack target is set as the low-importance node inside the breach. The route is to enter the defense area along the opened breach, advance based on the branch axes in the topological representation of the central axis, avoid the core defense node, and at the same time clear small defense facilities along the way. Subsequent phases follow the same pattern. The target of each phase is to target the new weak links exposed after the breakthrough of the previous phase, until finally approaching the core target, such as the resource depot or command center. The attack targets of each phase, the specific defense locations and facilities and the route of advance are clearly defined, and the detailed spatial coordinate path is formed, which finally forms a phased defense breaching strategy.

[0047] Step 4.5 integrates the phased breaching strategy with the dynamic ambush strategy to form a complete tactical action combination plan. Specifically, this includes: first, designing a dynamic ambush strategy. The core of this strategy is to determine the ambush location and trigger timing based on player behavior patterns in the game environment, such as common routes players use to support base defense, resource transport routes, patrol return routes, and the execution progress of the phased breaching strategy. Ambush locations are preferentially chosen from concealed terrain along players' likely routes, such as the back of high ground or beside narrow passages. The trigger timing is set when the phased breaching strategy reaches a critical point, such as after the first phase of breaching the wall, players are likely to arrive via the support route, at which point the ambush is triggered. Then, the dynamic ambush strategy is integrated with the phased breaching strategy... The phased breaching strategy is integrated, clarifying the synergistic relationship between the two: the attack actions of each phase in the phased breaching strategy serve as the main tactical advancement direction, while the dynamic ambush strategy serves as an auxiliary interference means. For example, when executing the second phase attack, ambushes are set up along the player's possible support routes to slow down the player's support speed and buy time for the breaching action; when the breaching strategy encounters strong resistance from the player, the ambush strategy is triggered to distract the player's attention and create new breakthrough opportunities for the breaching force; during the integration process, the time nodes and spatial positions of the two need to be coordinated to avoid tactical conflicts, such as the ambush point being too close to the breaching route, leading to accidental attacks, ultimately forming a complete tactical action combination scheme that includes the main breaching path and auxiliary ambush interference.

[0048] In this embodiment of the invention, polygonal boundary features of the base defense area are extracted to meet the needs of phased breaching. A topological representation of the defense structure is established by calculating the central axis, breaking the limitations of traditional methods that rely solely on surface location to determine defense. This allows NPCs to gain a deeper understanding of the internal structure of the defense system. Based on the analysis of the connectivity and node importance of the defense area using topological representation, weak points in the defense can be accurately identified, such as areas with loosely connected defense nodes or insufficient defense force. This avoids the indiscriminate and inefficient impact on the defense system seen in traditional attacks. Phased attack paths are planned based on these weak points, clarifying the attack targets and routes at each stage. The resulting phased breaching strategy can gradually dismantle the enemy's defenses, offering greater flexibility and better handling of complex base defense layouts than traditional fixed attack paths. Integrating the phased breaching strategy with a dynamic ambush strategy forms a complete tactical behavior combination scheme, enriching the NPC's tactical modes. This allows for both orderly frontal breaching and disruptive player deployment through ambushes, overcoming the shortcomings of traditional tactical modes that are singular and easily countered by players. This effectively enhances the diversity and competitiveness of NPC tactics, increasing the game's challenge and strategic depth.

[0049] In a preferred embodiment of the present invention, step 5 above may include: Step 5.1: Based on the optimized behavior strategy, assess the spatial proximity and resource competition among autonomous NPC tribes. When both the preset spatial proximity threshold and the preset resource competition threshold are met, it is determined that there is a cooperative condition between the autonomous NPC tribes. Specifically, this includes: based on the optimized behavior strategy, first extracting the action range and resource demand type of each autonomous NPC tribe in the strategy, and then assessing the spatial proximity and resource competition among the tribes based on this; when assessing spatial proximity, calculate the horizontal straight-line distance between the centers of the two tribes' territories in the game world coordinate system, and compare this distance with the preset spatial proximity threshold. This threshold is set according to the game map scale and the regular activity radius of the tribes. In open-world maps, it is usually set to 500 meters. If the distance is less than the threshold, it is considered spatially close; assessment When determining the level of resource competition, first count the top three priority resource types required in each tribe's optimization strategy, such as iron ore and energy cores. Then calculate the overlap of resource types required by different tribes. Combine this with the ratio of the total reserves of this type of resource in the game environment to the total demand of the tribes to determine the resource competition index. When the index exceeds the preset resource competition threshold, such as when there are two overlapping resource types and the resource shortage rate exceeds 40%, the level of resource competition is determined to be up to standard. When two tribes simultaneously meet the spatial proximity threshold and the resource competition threshold, it is determined that there is a synergy condition between them. For example, if the territorial distance between tribe A and tribe B is 300 meters, which is less than the 500-meter threshold, and both tribes have iron ore and energy as their main needs, with two overlapping resources and a resource shortage rate of 45% exceeding the threshold, then it is determined that there is a synergy condition between the two tribes.

[0050] Step 5.2: For multiple autonomous NPC tribes that meet the cooperation conditions, establish a cross-tribe information sharing mechanism. This mechanism includes player location information sharing, resource distribution status synchronization, and tactical execution progress reporting. Specifically, it includes: establishing a cross-tribe information transmission channel and a unified information sharing mechanism for multiple autonomous NPC tribes that meet the cooperation conditions. This mechanism includes real-time interaction of three core types of information: Regarding player location information sharing, each tribe will synchronize the real-time collected player 3D coordinates, movement direction, and movement speed data every 5 seconds through the information channel to ensure that all cooperating tribes obtain consistent player dynamics; Regarding resource distribution status synchronization, it integrates the resource point data detected by each tribe, including resource type, remaining quantity, coordinate location, and estimated refresh time. When a tribe collects a resource point, causing a change in quantity, it immediately updates the data to other tribes; Regarding tactical execution progress reporting, each tribe reports its own tactical execution status in stages, such as completing the first stage of breaching defenses or completing 60% of the resource collection task. The report includes the current stage completion rate, obstacles encountered (e.g., player interception), and the next plan, ensuring that cooperating tribes keep track of each other's progress.

[0051] Step 5.3: Based on the shared information in the information sharing mechanism, design a coordinated attack scheduling strategy. This strategy includes coordinating attack time windows, allocating attack directions, and configuring attack intensity. Specifically, it involves: designing a coordinated attack scheduling strategy based on real-time shared information in the information sharing mechanism; coordinating attack time windows based on player status, using shared player location and movement data to determine if a player is within the base's defense range and whether they have accompanying support troops. When a player returns to base and support troops are not yet in place, a unified attack time window is set, such as launching an attack in 1 minute. Each clan must complete attack preparation 5 seconds before the window opens to ensure synchronized action. The allocation of attack directions is based on shared player base defense data, such as identified weaknesses. The base defense area is divided into multiple attack directions, such as east, south, west, north, and northwest. The direction is allocated according to the current troop strength and equipment characteristics of each tribe. For example, tribes with stronger troops are responsible for defending the more solid but core passageway on the east side, while tribes with flexible tactics are responsible for the weaker section of the northwest wall. The configuration of attack intensity takes into account the number of soldiers, equipment level, and tactical execution capabilities of each tribe. Tribes with more troops and higher equipment levels undertake the main attack tasks, such as investing 60% of their troops, while other tribes undertake the tasks of containment or flanking support, such as investing 30% to 40% of their troops, to ensure that the attack intensity matches the tribe's capabilities.

[0052] Step 5.4 integrates the information sharing mechanism with the collaborative attack scheduling strategy to form a complete tribal collaborative behavior plan. Specifically, this includes: integrating the cross-tribal information sharing mechanism with the collaborative attack scheduling strategy into a complete tribal collaborative behavior plan; firstly, determining the service targets of the information sharing mechanism, i.e., prioritizing information transmission to support key nodes in the collaborative attack scheduling, such as increasing the synchronization frequency of player location information to once every 2 seconds in the 10 seconds before the attack time window is determined; after the attack direction is allocated, each tribe needs to synchronize its own troop deployment position through the information sharing mechanism to ensure accurate direction execution; then, binding the execution steps of the collaborative attack scheduling strategy with information update nodes, for example, when a tribe completes its pre-attack preparations, it needs to report its readiness through the information sharing mechanism, and only after all tribes have reported will the attack time window be triggered; during the attack, if a tribe encounters a player counterattack causing progress delays, it needs to report the obstruction in real time, and other tribes adjust their attack intensity or direction support based on the shared information; through this process binding, information sharing and attack scheduling form a closed loop, ultimately forming a complete tribal collaborative behavior plan that includes information interaction rules, attack execution steps, and the division of responsibilities among tribes.

[0053] Step 5.5 involves conflict detection and resource optimization allocation for the tribal collaborative behavior plan to ensure the consistency of tribal behaviors in time and space. Specifically, this includes conflict detection and resource optimization allocation for the formed tribal collaborative behavior plan. Conflict detection is conducted from both time and space dimensions. In terms of time, it checks whether the attack preparation time and attack initiation time of each tribe overlap or the interval is too short. For example, if tribe A needs 5 minutes to prepare and tribe B plans to launch an attack in 3 minutes, there is a time conflict. In terms of space, it checks whether the attack routes of each tribe intersect or whether the rally points overlap. For example, if the attack routes of tribes A and B both need to pass through the same narrow passage, congestion is likely to occur. For detected conflicts, adjustments are made to... The attack time difference can be addressed by delaying Clan B's attack by 2 minutes or optimizing the route, such as replanning a detour for Clan B. Resource optimization and allocation are crucial for resources such as troops, equipment, and supplies involved in the coordinated attack. First, the resource inventory and needs of each clan are tallied. Then, resources are allocated according to the attack responsibilities. For example, the clan undertaking the main attack is given priority for high-level equipment and more supplies, while support clans are allocated flexible combat equipment. Simultaneously, it is ensured that the resource input ratio of each clan matches its attack contribution. For instance, the resource input ratio of the main attacking clan should not exceed 60% of its own inventory to avoid excessive consumption. Through reasonable resource allocation, the actions of each clan are ensured to be coordinated in both time and space, improving the success rate of coordinated attacks.

[0054] In this embodiment of the invention, the spatial proximity and resource competition of each autonomous NPC tribe are evaluated based on optimized behavioral strategies. Collaboration conditions are accurately determined through preset thresholds, preventing tribes from missing effective collaboration opportunities or forming meaningless collaborations, thus ensuring the necessity and rationality of collaboration. A cross-tribe information sharing mechanism is established for tribes that meet the conditions, enabling real-time communication of player locations, resource distribution, and tactical progress. This breaks the traditional isolation of tribe information, reduces collaboration errors caused by information asymmetry, and provides comprehensive support for collaborative actions. Based on shared information, a collaborative attack scheduling strategy is designed to coordinate attack time, allocation direction, and configuration intensity, allowing attacks from various tribes to form a combined force. This avoids the problems of scattered attacks and easy individual responses by players caused by traditional independent warfare, enhancing overall attack deterrence. Information sharing and attack scheduling are integrated into a complete collaborative solution, providing a clear and unified execution framework for collaborative actions, overcoming the limitations of fragmented and difficult-to-implement traditional collaborative logic. The plan involves conflict detection and resource optimization to proactively avoid behavioral conflicts between tribes in time and space, rationally allocate resources to avoid waste, ensure the consistency of collaborative actions, promote autonomous NPC tribes to shift from fighting alone to cooperating, enhance group confrontation capabilities, enrich game group interaction scenarios, and improve strategy and challenge.

[0055] In a preferred embodiment of the present invention, step 6 above may include: Step 6.1 involves analyzing tactical behavior combinations and tribal collaborative behavior schemes to extract basic behavioral units. These basic behavioral units include movement, attack, waiting, and cooperative behavior types. Specifically, this includes: performing structured analysis on the tactical behavior combination schemes and tribal collaborative behavior schemes, breaking down all action flows described in the schemes; classifying behaviors according to their action attributes during analysis, identifying recurring basic action patterns, and extracting them as basic behavioral units; movement behavior types refer to the displacement actions of NPCs from one spatial location to another, including specific forms such as linear movement and circling movement; attack behavior types refer to the offensive actions initiated by NPCs against targets, covering ranged attacks and melee attacks; waiting behavior types refer to the actions of NPCs maintaining a stationary state at a designated location, including temporary standby and defensive garrisoning; cooperative behavior types refer to actions completed by multiple autonomous NPC tribes in cooperation, such as alternating cover and resource transfer; after analysis, a set of basic behavioral units containing the above four types of behavior is formed, with each unit corresponding to the action description explicitly stated in the scheme.

[0056] Step 6.2: Based on basic behavioral units, establish a behavior priority mapping relationship to assign execution priority and triggering conditions to each behavioral unit. Specifically, this includes: establishing a behavior priority mapping relationship based on basic behavioral units and considering possible scene changes in the game environment; the priority division is based on the urgency of the behavior, its contribution to the task objective, and the level of environmental threat. Behavioral units with high urgency, high contribution, or dealing with high threats are assigned higher priority; for example, attack behavior has the highest priority when a player is detected entering the core area of ​​the territory, and movement behavior has a higher priority than waiting behavior when rapid support for friendly forces is needed; at the same time, triggering conditions are set for each behavioral unit. The triggering condition for attack behavior is that the player enters the preset attack range and the NPC is in combat; the triggering condition for waiting behavior is that the previous task is completed and no new instructions are received; the triggering condition for cooperative behavior is that a friendly force request for cooperation is received and the current task priority is low; the triggering condition for movement behavior is that the target location is determined and the path is unobstructed; finally, a mapping relationship table containing the corresponding priority value and triggering condition description for each behavioral unit is formed.

[0057] Step 6.3: Based on the behavior priority mapping relationship, a sequence of behavior instructions that the game engine can recognize is formed. The sequence of behavior instructions includes spatial coordinates, time parameters, and execution order. Specifically, it includes: converting basic behavior units into a sequence of behavior instructions that the game engine can directly recognize based on the established behavior priority mapping relationship; each instruction contains three core elements: spatial coordinates refer to the precise three-dimensional coordinates of the target position or range of action in the game world coordinate system, such as the endpoint coordinates of movement behavior and the target coordinates of attack behavior; time parameters include the start time, duration, and interval of behavior execution, such as the continuous attack duration of attack behavior being 10 seconds and the interval between two attacks being 2 seconds; the execution order is arranged from high to low behavior priority. When the triggering conditions of multiple behavior units are met simultaneously, the instruction with higher priority is executed first. If the priorities are the same, they are ordered according to the order of triggering time. For example, when attack behavior and movement behavior are triggered simultaneously and the attack priority is higher, the attack instruction is placed before the movement instruction in the instruction sequence, finally forming an ordered instruction sequence data with a format adapted to the game engine's instruction receiving specifications.

[0058] Step 6.4 involves real-time parsing of the behavior instruction sequence to convert it into underlying control commands. This process includes: real-time parsing of the behavior instruction sequence, synchronized with the game engine's frame rate, processing the currently executing instruction in the instruction sequence each frame; firstly, extracting the spatial coordinates from the instructions and converting them into local coordinates used by the game engine's underlying system; then, converting the time parameters into timestamps recognizable by the engine to ensure the execution time of the behavior is synchronized with the engine clock; finally, determining the execution priority of the current instruction based on the execution order, excluding low-priority instructions interrupted by higher-priority instructions; and ensuring that the converted underlying control commands conform to the game engine's API interface requirements. For example, movement control commands include parameters such as target coordinates, movement speed, and acceleration, while attack control commands include parameters such as target ID, attack type, and damage value, ensuring the engine can directly read and respond to these commands.

[0059] Step 6.5 executes the transformed control commands to drive the autonomous NPC tribes to perform corresponding behaviors according to the command sequence. Specifically, this includes: receiving the transformed underlying control commands and executing them sequentially according to the command sequence; when executing a movement command, the engine drives the NPC character's skeletal animation system to play walking or running animations, while simultaneously calculating the movement path based on spatial coordinates and time parameters, and adjusting the character's position coordinates in the game world in real time; when executing an attack command, an attack animation is triggered and damage effects are calculated, while the target's status data is updated synchronously; when executing a wait command, the character's movement and attack actions are paused, and a standby animation is played; when executing a cooperation command, the actions of multiple NPC characters are coordinated synchronously, such as when an NPC from tribe A performs a cover animation, an NPC from tribe B synchronously performs a dash action; during execution, the command execution status is monitored in real time, and when a command is completed, the next command in the sequence is automatically called to ensure that the autonomous NPC tribes' behavior is completely matched with the command sequence, ultimately realizing the complete implementation of tactical behavior combinations and tribal cooperative behavior schemes.

[0060] In this embodiment of the invention, tactical behavior combinations and tribal collaboration schemes are analyzed, and basic behavioral units such as movement, attack, waiting, and cooperation are extracted. Complex strategies are broken down into basic modules, avoiding the problem of difficulty in translating complex structures into actual behaviors, and providing a concise basis for command generation. A priority mapping relationship is established based on the basic behavioral units, assigning execution priority and triggering conditions to each unit. For example, attack behavior has higher priority than waiting when encountering a player, and cooperation requires satisfying friendly signals, avoiding behavioral conflicts or disordered sequences and improving execution logic. Based on the mapping, an engine-recognizable command sequence containing spatial coordinates, time parameters, and execution order is formed, solving the adaptation problem between strategy and engine, ensuring that strategy requirements are accurately translated into a standardized format, and bridging the gap from design to execution. The command sequence is parsed in real time as underlying control commands, allowing abstract commands to be grounded in concrete control signals, ensuring effective response from the underlying system and avoiding execution failures due to format issues. Executing control commands drives autonomous NPC tribes to complete corresponding behaviors, truly transforming tactical and collaborative schemes into actual actions, ensuring a complete closed loop from strategy design to implementation, overcoming the limitations of traditional strategies and behaviors being disconnected, improving the accuracy and consistency of NPC behavior, and enhancing the realism of game interaction.

[0061] In a preferred embodiment of the present invention, step 7 above may include: Step 7.1 involves continuously monitoring key features in the environmental state space, including player position changes, resource distribution status updates, and tribal relationship dynamics. The monitoring results are obtained through the following steps: Based on the game's frame rate, key features in the environmental state space are continuously monitored every 100 milliseconds; when monitoring player position changes, the player's 3D coordinates and movement direction in the game world coordinate system are recorded in real time, and the deviation between the current coordinates and the coordinates at the previous monitoring time is calculated. Simultaneously, it tracks whether the player enters a new area, such as a resource point or tribal territory; when monitoring resource distribution status updates, the focus is on changes in the remaining quantity of each resource point. When a resource point is collected or refreshed, its quantity change value and change time are recorded, and the availability status of the resource point, such as whether it is depleted, is updated synchronously; when monitoring tribal relationship dynamics, the interaction frequency between autonomous NPC tribes, such as the number of coordinated attacks or conflicts, is statistically analyzed. Combined with changes in the overlapping area of ​​tribal territories, the relationship between tribes is determined to be either friendly or hostile. Finally, all the monitored data is summarized to form the monitoring results.

[0062] Step 7.2: Based on the monitoring results, calculate the change range of environmental state characteristics. When the change range exceeds a preset threshold, it is marked as a valid environmental state change. Specifically, this includes: calculating the change range of each key characteristic based on the obtained monitoring results; when calculating the change range of the player's position, using the Euclidean distance formula to calculate the straight-line distance between the player's current position and the previous stable position, where the previous stable position refers to the coordinates where the player has not moved for 30 consecutive seconds; when calculating the change range of resource distribution status, using the initial total amount of resource points as a benchmark, calculating the percentage difference between the current remaining amount and the initial total amount, such as the initial reserves of a certain iron ore. 1000 units, currently 600 units remaining, change range is 40%; when calculating the dynamic change range of tribe relations, it is measured by the change value of the tribe interaction friendliness score. The friendliness score ranges from -100 to 100, with positive values ​​indicating friendliness and negative values ​​indicating hostility. The change range is the absolute value of the difference between the current score and the score 24 hours ago; preset change thresholds for each feature are set, with the player location change threshold set to 50 meters, the resource distribution change threshold set to 30%, and the tribe relations change threshold set to 40 points. When the change range of a feature exceeds the corresponding threshold, it is marked as a valid environmental state change.

[0063] Step 7.3: Based on the effective environmental state changes, trigger the environmental state space update mechanism. Based on the environmental state space update mechanism, initiate the reconstruction process of the environmental state space. Specifically, this includes: when an effective environmental state change is marked, immediately trigger the environmental state space update mechanism. This mechanism has pre-set trigger rules. As long as any one of the following is marked as an effective change: player position change, resource distribution update, or tribe relationship dynamics, the reconstruction process of the environmental state space will be initiated. When the reconstruction process is initiated, automatically pause the execution of the current behavior strategy generated based on the old state space, release the cached data related to the old state space, and send a state update signal to the reinforcement learning model to notify the model to prepare to receive new environmental state data. This ensures that the reconstruction process is not interfered with by old data and provides a clean operating environment for the construction of the new state space.

[0064] Step 7.4: During the reconstruction process of the environment state space, player location, resource distribution, and tribe attribute data are re-collected to construct a new multi-dimensional environment state space. Specifically, this includes: re-collecting data according to the collection standards in steps 1.1 to 1.4 during the environment state space reconstruction process; obtaining the player's current precise 3D coordinates, movement speed, and movement trajectory within the last 5 minutes when re-collecting player location data; scanning all resource points in the game environment and recording their latest 3D coordinates, resource type, remaining quantity, and expected next refresh time when re-collecting tribe attribute data; updating the current population, military strength level, resource reserves, and territory boundary polygon vertex coordinates of each autonomous NPC tribe when re-collecting tribe attribute data; and integrating the re-collected three types of data with the latest spatial relative positional relationship data of players and tribe territories, using the normalization method to convert them into a standardized numerical sequence to construct a new multi-dimensional environment state space that accurately reflects the actual situation of the current game environment.

[0065] Step 7.5 involves inputting the new multi-dimensional environment state space into the reinforcement learning model's policy network and restarting the behavior policy generation process to form a continuously optimized closed-loop processing flow. Specifically, this includes: inputting the constructed new multi-dimensional environment state space into the reinforcement learning model's policy network and restarting the behavior policy generation process; that is, extracting new state features through the policy network, forming and optimizing behavior policies, designing tactical behavior combinations and tribal collaboration schemes, and converting them into control commands executable by the game engine to drive autonomous NPC tribes to make behaviors adapted to the new environment; after each new policy is executed, steps 7.1 to 7.4 are restarted to continuously monitor environmental changes and update the state space. This cycle repeats, forming a continuously optimized closed-loop processing flow of environment monitoring, state updating, policy generation, and behavior execution, ensuring that the behavior of autonomous NPC tribes can respond to the dynamic changes of the game environment in real time and avoid becoming disconnected from the environment due to policy rigidity.

[0066] In this embodiment of the invention, key features such as player position changes, resource distribution updates, and tribe relationship dynamics in the environmental state space are continuously monitored. This allows for real-time capture of dynamic changes in the game environment, avoiding the lag in environmental change perception common in traditional methods and providing timely information support for strategy adjustments. The magnitude of state feature changes is calculated based on the monitoring results, and only changes exceeding a preset threshold are marked as valid, filtering out meaningless minor fluctuations and avoiding frequent updates that consume resources, ensuring the targeted and necessary nature of updates. Valid changes trigger an environmental state space update mechanism and initiate a reconstruction process, breaking the limitations of a fixed traditional state space and allowing it to dynamically adjust with the environment, always adapting to the actual scenario. During reconstruction, player position, resource distribution, and tribe attribute data are re-collected to construct a new multi-dimensional environmental state space, ensuring data timeliness and accuracy, and providing a suitable input foundation for reinforcement learning models. By inputting the new state space into the policy network and restarting the behavior policy generation process, a continuous optimization closed loop of monitoring, judgment, updating, and generation is formed. This allows NPC behavior policies to iterate continuously with changes in the environment, overcoming the shortcomings of traditional initial policies that remain unchanged and are out of touch with the environment. This effectively improves the NPC's adaptability to the dynamic game environment, making its behavior more flexible and intelligent, and enhancing the game's immersion and challenge.

[0067] like Figure 2 As shown, embodiments of the present invention also provide a game NPC behavior dynamic generation and optimization system based on reinforcement learning, including: The building module is used to construct a multi-dimensional environmental state space for autonomous NPC tribes in the current game environment by collecting data on player location, resource distribution, and tribe attributes; based on the multi-dimensional environmental state space, an initial behavioral policy is formed through a policy network of a reinforcement learning model. The iteration module is used to collect data on the execution effect of the initial behavior strategy in the game environment, evaluate the effect of the strategy through the value network of the reinforcement learning model, obtain the evaluation results, and perform strategy optimization iteration based on the evaluation results to form an optimized behavior strategy. The planning module is used to dynamically generate targeted tactical behavior combinations based on the optimized behavior strategy. The tactical behavior combinations include dynamic ambush strategy and phased defense penetration strategy. In the phased defense penetration strategy, the central axis of the polygonal shape of the base defense area is extracted, and the weak points of the defense are identified based on the topology of the central axis, so as to plan the phased attack path. The detection module is used to form a tribal collaborative behavior scheme based on optimized behavior strategies when multiple autonomous NPC tribes are detected to meet the collaborative conditions of spatial proximity and resource competition. The execution module is used to transform tactical behavior combinations and tribal collaborative behavior schemes into a sequence of behavioral instructions that the game engine can recognize. By parsing and executing the instruction sequence, it drives autonomous NPC tribes to complete the corresponding behavioral performance. The trigger module is used to monitor the current game environment's running status in real time. When the environmental state characteristics change, it triggers the environmental state space update mechanism, and reconstructs the multi-dimensional environmental state space to form a closed-loop behavior optimization process.

[0068] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for dynamic generation and optimization of game NPC behavior based on reinforcement learning, characterized in that, The method includes: In the current game environment, by collecting player location, resource distribution, and tribe attribute data, a multi-dimensional environmental state space of autonomous NPC tribes is constructed; Based on a multi-dimensional environmental state space, an initial behavioral policy is formed through a policy network of a reinforcement learning model. Data on the execution effect of the initial behavioral strategy in the game environment is collected. The effect of the strategy is evaluated through the value network of the reinforcement learning model to obtain the evaluation results. Based on the evaluation results, the strategy is optimized and iterated to form an optimized behavioral strategy. Based on the optimized behavioral strategy, targeted tactical behavior combinations are dynamically generated. These combinations include dynamic ambush strategies and phased breaching strategies. In the phased breaching strategy, the central axis of the polygonal shape of the base defense area is extracted, and weak points in the defense are identified based on the topological structure of the central axis. This is used to plan phased attack paths, including: determining the tactical objectives and their priority order for the autonomous NPC tribe based on the optimized behavioral strategy; extracting the polygonal boundary features of the base defense area based on the tactical objectives and the phased breaching tactical requirements in the current game environment; calculating the central axis of the polygonal shape of the base defense area to establish a topological representation of the defense structure, i.e., the central axis topological representation; analyzing the connectivity and node importance of the defense area based on the central axis topological representation to identify weak points in the defense system; planning phased attack paths based on the identified weak points to determine the attack targets and routes at each stage, ultimately forming the phased breaching strategy; and integrating the phased breaching strategy with the dynamic ambush strategy to form a complete tactical behavior combination scheme. When multiple autonomous NPC tribes are detected to meet the cooperative conditions of spatial proximity and resource competition, a tribal cooperative behavior scheme is formed based on optimized behavior strategies. Tactical behavior combinations and tribal collaborative behavior schemes are transformed into behavioral instruction sequences that can be recognized by the game engine. By parsing and executing the instruction sequences, autonomous NPC tribes are driven to complete corresponding behavioral performances. The system monitors the current game environment's operational status in real time. When the environmental state characteristics change, it triggers an update mechanism for the environmental state space. By reconstructing the multi-dimensional environmental state space, a closed-loop behavior optimization process is formed.

2. The method for dynamic generation and optimization of game NPC behavior based on reinforcement learning according to claim 1, characterized in that, In the current game environment, by collecting player location, resource distribution, and tribe attribute data, a multi-dimensional environmental state space for autonomous NPC tribes is constructed, including: A horizontal ray is emitted from the player's current position in the current game environment, and the horizontal ray extends horizontally to the farthest point of the tribal territory boundary polygon. Calculate the number of intersections between the horizontal ray and each boundary of the tribal territory polygon, and record the precise coordinates of the intersection points. Based on the calculated number of intersections, spatial relationships are determined according to the parity rule. That is, when the number of intersections is odd, the player is determined to be inside the tribe's territory; when the number of intersections is even, the player is determined to be outside the tribe's territory, thus obtaining spatial relative positional relationship data. The spatial relative positional relationship data is integrated with player position, resource distribution and tribe attribute data to form a feature set of a multi-dimensional environmental state space; The feature set is normalized and transformed into a standardized state vector that can be recognized by the reinforcement learning model, so as to complete the construction of a multi-dimensional environmental state space.

3. The method for dynamic generation and optimization of game NPC behavior based on reinforcement learning according to claim 2, characterized in that, Based on a multi-dimensional environmental state space, an initial behavioral policy is formed through a policy network of a reinforcement learning model, including: The standardized state vector is input into the policy network of the reinforcement learning model, and state features are extracted through a multi-layer neural network. Based on state characteristics, a behavioral strategy framework is formed that includes resource monitoring intensity, patrol range, and behavior priority. Based on the spatial relative position of the player and the tribe's territory, the parameter configuration in the behavior strategy framework is dynamically adjusted to form a parameter-optimized behavior strategy. By extracting player location data, resource distribution data, and tribe attribute data from the feature set, the optimized behavior strategy is refined and improved to form a complete initial behavior strategy.

4. The method for dynamic generation and optimization of game NPC behavior based on reinforcement learning according to claim 3, characterized in that, Data on the execution effect of the initial behavioral strategy in the game environment is collected. The effect of the strategy is evaluated through the value network of a reinforcement learning model to obtain the evaluation results. Based on the evaluation results, the strategy is iterated and optimized to form an optimized behavioral strategy, including: Collect data on the execution effect of the initial behavioral strategy in the game environment. The execution effect data includes the success rate of NPC behavior execution, resource acquisition efficiency, and player interaction response data. The performance data is input into the value network of the reinforcement learning model, and the value evaluation score of the behavioral strategy is obtained through the value network calculation. Based on the value assessment score, the advantage function value of the current behavioral strategy is analyzed and calculated to identify the dimensions of the behavioral strategy that need to be optimized, and the analysis and calculation results of the advantage function value are obtained. Based on the analysis and calculation results of the dominance function value, the parameter configuration of the policy network is updated by using the policy gradient method to obtain the updated policy network; Based on the updated policy network, a new behavioral policy is formed. The newly formed behavioral policy is then iteratively optimized through multiple rounds to obtain an optimized behavioral policy.

5. The method for dynamic generation and optimization of game NPC behavior based on reinforcement learning according to claim 4, characterized in that, When multiple autonomous NPC tribes are detected to meet the cooperative conditions of spatial proximity and resource competition, a tribal cooperative behavior scheme is formed based on optimized behavioral strategies, including: Based on the optimized behavior strategy, the spatial proximity and resource competition among autonomous NPC tribes are evaluated. When both the preset spatial proximity threshold and the preset resource competition threshold are met, it is determined that there is a cooperative condition among the autonomous NPC tribes. For multiple autonomous NPC tribes that meet the conditions for cooperation, a cross-tribe information sharing mechanism will be established. The information sharing mechanism includes sharing player location information, synchronizing resource distribution status, and reporting tactical execution progress. Based on the shared information in the information sharing mechanism, a collaborative attack scheduling strategy is designed, which includes the coordination of attack time windows, the allocation of attack directions, and the configuration of attack intensity. By integrating information sharing mechanisms with coordinated attack scheduling strategies, a complete tribal coordinated behavior scheme is formed; Conflict detection and resource optimization are performed on tribal collaborative behavior schemes to ensure the coordination and consistency of tribal behaviors in time and space.

6. The method for dynamic generation and optimization of game NPC behavior based on reinforcement learning according to claim 5, characterized in that, Tactical behavior combinations and tribal collaborative behavior schemes are transformed into behavioral command sequences that the game engine can recognize. By parsing and executing these command sequences, autonomous NPC tribes are driven to perform corresponding behaviors, including: Analyze tactical behavior combinations and tribal cooperative behavior schemes to extract the basic behavioral units, which include movement, attack, waiting, and cooperative behavior types. Based on basic behavioral units, a behavior priority mapping relationship is established to assign execution priority and triggering conditions to each behavioral unit; Based on the behavior priority mapping relationship, a sequence of behavior instructions that can be recognized by the game engine is formed. The sequence of behavior instructions includes spatial coordinates, time parameters and execution order. The sequence of behavioral instructions is parsed in real time and converted into underlying control commands to obtain the converted control commands; Execute the transformed control commands to drive the autonomous NPC tribes to perform corresponding behaviors according to the instruction sequence.

7. The method for dynamic generation and optimization of game NPC behavior based on reinforcement learning according to claim 6, characterized in that, The system monitors the current game environment's operational status in real time. When environmental state characteristics change, it triggers an environment state space update mechanism. By reconstructing a multi-dimensional environment state space, a closed-loop behavior optimization process is formed, including: Continuously monitor key features in the environmental state space, including changes in player location, updates to resource distribution status, and dynamics of tribe relationships, to obtain monitoring results; Based on the monitoring results, the magnitude of change in environmental state characteristics is calculated. When the magnitude of change exceeds a preset threshold, it is marked as a valid environmental state change. Based on changes in the effective environmental state, the environmental state space update mechanism is triggered, and based on the environmental state space update mechanism, the environmental state space reconstruction process is initiated. During the process of reconstructing the environment state space, player location, resource distribution, and tribe attribute data are re-collected to construct a new multi-dimensional environment state space. The new multi-dimensional environmental state space is input into the policy network of the reinforcement learning model, and the behavior policy generation process is restarted to form a closed-loop processing flow that is continuously optimized.

8. A system for dynamically generating and optimizing game NPC behavior based on reinforcement learning, wherein the system implements the method as described in any one of claims 1 to 7, characterized in that, include: The building module is used to construct a multi-dimensional environmental state space for autonomous NPC tribes in the current game environment by collecting data on player location, resource distribution, and tribe attributes. Based on a multi-dimensional environmental state space, an initial behavioral policy is formed through a policy network of a reinforcement learning model. The iteration module is used to collect data on the execution effect of the initial behavior strategy in the game environment, evaluate the effect of the strategy through the value network of the reinforcement learning model, obtain the evaluation results, and perform strategy optimization iteration based on the evaluation results to form an optimized behavior strategy. The planning module is used to dynamically generate targeted tactical behavior combinations based on the optimized behavior strategy. The tactical behavior combinations include dynamic ambush strategy and phased defense penetration strategy. In the phased defense penetration strategy, the central axis of the polygonal shape of the base defense area is extracted, and the weak points of the defense are identified based on the topology of the central axis, so as to plan the phased attack path. The detection module is used to form a tribal collaborative behavior scheme based on optimized behavior strategies when multiple autonomous NPC tribes are detected to meet the collaborative conditions of spatial proximity and resource competition. The execution module is used to transform tactical behavior combinations and tribal collaborative behavior schemes into a sequence of behavioral instructions that the game engine can recognize. By parsing and executing the instruction sequence, it drives autonomous NPC tribes to complete the corresponding behavioral performance. The trigger module is used to monitor the current game environment's running status in real time. When the environmental state characteristics change, it triggers the environmental state space update mechanism, and reconstructs the multi-dimensional environmental state space to form a closed-loop behavior optimization process.

Images