A method, system, and storage medium for track generation based on reinforcement learning-guided generative adversarial networks

Abstract

This invention discloses a method, system, and storage medium for track generation based on reinforcement learning-guided generative adversarial networks (GANs), belonging to the fields of procedural content generation (PCG) and game artificial intelligence. First, a skiing game environment, SkiEnv, is constructed; a PPO agent is trained and frozen as an implicit difficulty estimator; a reward prediction network with a residual attention structure is trained based on agent interaction data as a differentiable GAN discriminator; a controllable generator is constructed, taking random noise and target difficulty as input, and outputting track parameters, and is trained using a hybrid loss function including score matching loss and diversity regularization loss, along with a dynamic normalization strategy; finally, the generator is deployed in a real-time system, dynamically generating track parameters according to the user-specified difficulty and loading them into SkiEnv for rendering. This invention achieves end-to-end optimization from reinforcement learning feedback to the generator, ensuring content diversity while precisely controlling track difficulty, and exhibiting good real-time performance and controllability.

Description

Technical Field

[0001] This invention belongs to the field of procedural content generation and game artificial intelligence technology, specifically relating to a track generation method, system and storage medium based on reinforcement learning-guided generative adversarial networks. Background Technology

[0002] Procedural Content Generation (PCG) technology automatically generates game levels, maps, items, and other content through algorithms, significantly reducing game development costs and improving replayability. Traditional PCG methods typically rely on manually defined rule sets (such as L-systems and cellular automata) or search-based optimization algorithms (such as genetic algorithms). These methods perform well in static scenes, but struggle to meet real-time requirements in high-speed dynamic games (such as skiing and racing) and cannot effectively utilize player interaction feedback for dynamic difficulty adjustments.

[0003] In recent years, deep generative models (such as Generative Adversarial Networks (GANs)) have been introduced into the PCG field to generate new content by learning from existing level data. However, most existing methods only focus on the reproduction of visual styles, lacking modeling of the playability and difficulty controllability of the generated content. Furthermore, the evaluation process usually adopts ex-post manual rules or non-differentiable tests, failing to directly integrate player behavior feedback into the generator's optimization process.

[0004] Reinforcement learning (RL) has made significant progress in game AI, enabling RL agents to learn complex decision-making strategies through interaction with their environment. Existing research has attempted to use RL for level evaluation, for example, measuring level difficulty by the performance of RL agents on generated levels. However, this evaluation mechanism is non-differentiable, making end-to-end joint optimization with the generative model difficult. Generative Playing Networks (GPNs) were the first to attempt to use feedback from RL agents to train the generator, but they rely on heuristic reward shaping, lack a clear difficulty control mechanism, and have a coarse-grained optimization process, limiting the quality and controllability of the generated content.

[0005] To address the aforementioned issues, this invention proposes a method that trains a differentiable reward prediction network as the discriminator of a GAN, transforming the behavioral feedback of the RL agent into continuous gradient signals to achieve end-to-end optimization of the generator parameters. Simultaneously, by using a hybrid loss function and a dynamic normalization strategy, the diversity of generated content is ensured while precisely controlling the difficulty of the track. Summary of the Invention

[0006] The purpose of this invention is to provide a track generation method, system, and storage medium based on reinforcement learning-guided generative adversarial networks (PCGs) to overcome the problems of existing PCG methods, such as difficulty in responding to player behavior in real time, lack of differentiable evaluation mechanisms, and inaccurate difficulty control.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A track generation method based on reinforcement learning-guided generative adversarial networks includes the following steps:

[0009] Step 1: Construct the game environment, which defines a state space, an action space, and a composite reward function. The composite reward function is used to provide training feedback for the reinforcement learning agent and to quantitatively evaluate the difficulty of the track.

[0010] Step 2: The reinforcement learning agent is trained using a reinforcement learning algorithm in the game environment, and the trained reinforcement learning agent is frozen as a track difficulty evaluator.

[0011] Step 3: Randomly sample multiple sets of track parameters, use the frozen reinforcement learning agent to interact with the game environment to obtain the corresponding cumulative reward, and construct a paired dataset containing the track parameters and the cumulative reward; train a differentiable evaluation model based on the paired dataset, the differentiable evaluation model is used to realize the differentiable mapping from the track parameters to the cumulative reward;

[0012] Step 4: Construct a generator network. The input of the generator network is random noise and the target difficulty value, and the output is track parameters. Use the differentiable evaluation model as the optimization guide model of the generator network, and train the generator network using a hybrid loss function. The hybrid loss function includes at least a first loss component for making the difficulty of the generated track approximate the target difficulty value and a second loss component for maintaining the diversity of the generated track content.

[0013] Step 5: Deploy the trained generator network in the real-time game system. The system receives the target difficulty value specified by the user, concatenates it with the random noise, and inputs it into the generator network to generate the corresponding track parameters. The generated track parameters are then loaded into the game environment to render a track that meets the difficulty requirements in real time.

[0014] Furthermore, in step 1, the state space is a 7-dimensional continuous vector, where the first two dimensions represent the current speed and speed decay coefficient of the game character, respectively; and the last five dimensions represent the lateral position of the skier and the lateral and longitudinal coordinates of the obstacles and props closest to the game character, respectively.

[0015] The action space is a discrete action space, which includes staying still, moving left, and moving right;

[0016] The composite reward function The total reward is:

[0017]

[0018] in, The base motion reward is determined based on the current speed of the game character. The reward for lateral exploration is determined based on the lateral position of the game character. The penalty for being stationary is determined based on the zero-speed state of the game character. To determine the sharp turn penalty based on the turning change rate of the game character, The event reward is determined based on the interaction results between the game character and obstacles or props.

[0019] Furthermore, in step 2, the reinforcement learning algorithm is a proximal policy optimization algorithm, employing an Actor-Critic network.

[0020] In the game environment, multiple rounds of interaction are performed, and trajectory data is collected in each round. Based on the policy loss function and value loss function of the near-end policy optimization algorithm, and with preset discount factors, preset GAE parameters, preset pruning coefficients, preset batch sizes, and entropy regularization coefficients used to promote exploration, the network parameters are updated until the cumulative reward converges. After training, the parameters of the reinforcement learning agent are frozen, and the cumulative reward obtained by the agent on any track is used as the difficulty score of that track.

[0021] Furthermore, in step 3, the track parameters are N-dimensional vectors, which include the position information of multiple obstacles and multiple props and their respective time offset information;

[0022] The paired dataset is formed by randomly sampling multiple sets of track parameters, inputting them into the game environment, running the complete game with the frozen reinforcement learning agent, and recording the cumulative rewards.

[0023] Furthermore, the differentiable evaluation model is a reward prediction network, and training the reward prediction network specifically includes:

[0024] A reward prediction network is constructed with a residual network as the backbone and a spatial attention module embedded after the feature extraction layer. The reward prediction network takes the track parameters as input and outputs a single predicted reward value.

[0025] After training, the reward prediction network is fixed as the optimization guide model for the generator network.

[0026] Furthermore, in step 4, the generator network input is M-dimensional random noise. and 1-dimensional target difficulty value The output is the track parameters. The generator network employs a multilayer perceptron structure, with the hidden layer dimension decreasing progressively, and the output layer using the Tanh activation function; the training of the generator network specifically includes:

[0027] A dynamic normalization strategy is employed to transform the original predicted reward output by the differentiable evaluation model. The Gaussian function maps to a predetermined numerical range, and the mapping characteristics of the Gaussian function change dynamically with the increase of the number of training steps.

[0028] Using a hybrid loss function Calculate the training loss.

[0029]

[0030] in, For score matching loss, For diversity regularization loss, This is a diversity regularization coefficient, used to balance difficulty control and content diversity;

[0031] The generator is trained using the Adam optimizer, iterating until convergence.

[0032] Furthermore, the score matching loss The formula used to approximate the optimal value of the normalized predicted reward is:

[0033]

[0034] in, For batch size, For the first Normalized prediction reward for each generated sample;

[0035] The diversity regularization loss The formula is:

[0036]

[0037] in, and The first The and the first A vector of track parameters for each generated sample. For the Euclidean norm, This is a diversity scaling parameter used to control the sensitivity of the similarity penalty between samples.

[0038] Furthermore, the normalized predicted reward Calculated using the following Gaussian mapping function:

[0039]

[0040] in, The original predicted reward output by the differentiable evaluation model. To reward scaling factor, The center position parameter of the Gaussian function. is the width parameter of the Gaussian function;

[0041] The width parameter With training steps Attenuation occurs in the following manner:

[0042]

[0043] in, This is the initial value for the width parameter. This is the attenuation amount. Total training steps The minimum value of the width parameter is used; the decay strategy encourages exploration with a wide Gaussian distribution in the early stage of training, and achieves fine optimization with a narrow Gaussian distribution in the later stage of training.

[0044] A computer device / apparatus / system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a track generation method based on a reinforcement learning-guided generative adversarial network.

[0045] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a track generation method based on a reinforcement learning-guided generative adversarial network.

[0046] The beneficial effects of this invention are as follows:

[0047] By introducing a differentiable reward prediction network, the behavioral feedback of the reinforcement learning agent is transformed into continuous gradient signals, enabling direct backpropagation from player interaction to generator parameters. This solves the problem of non-differentiable evaluation in traditional PCG games. A hybrid loss function is employed to precisely control the difficulty of generated tracks while preventing pattern collapse through diversity regularization, ensuring the richness of generated content. A dynamic normalization strategy simulates the learning process, encouraging exploration in the early stages of training and focusing on accuracy in the later stages, improving training stability and generation quality. The generated tracks can be dynamically adjusted according to the user-specified difficulty value, suitable for dynamic difficulty balancing and personalized content generation in games, exhibiting good real-time performance and controllability. Attached Figure Description

[0048] Figure 1 This is the overall architecture diagram of the system of the present invention;

[0049] Figure 2 This is a schematic diagram of the SkiEnv game environment of the present invention;

[0050] Figure 3 This is the PPO agent training curve diagram of the present invention;

[0051] Figure 4 This is a schematic diagram of the reward prediction network of the present invention;

[0052] Figure 5 This is an example of a track generated by the generator of this invention (comparison of different difficulties). Detailed Implementation

[0053] The present invention will now be further described with reference to the accompanying drawings.

[0054] Figure 1 This is a diagram illustrating the overall architecture of the track generation system based on reinforcement learning-guided generative adversarial networks (GANs) of this invention. The technical solution includes: constructing a skiing game environment (SkiEnv); training a PPO agent and freezing it as an implicit difficulty estimator; training a reward prediction network with a residual attention structure as a differentiable GAN discriminator based on agent interaction data; constructing a controllable generator that takes random noise and target difficulty as input, outputs track parameters, and is trained using a hybrid loss function containing score matching loss and diversity regularization loss, along with a dynamic normalization strategy; deploying the generator in a real-time system to dynamically generate track parameters according to user-specified difficulty and load them into SkiEnv for rendering. This invention achieves end-to-end optimization of reinforcement learning feedback to the generator, ensuring content diversity while precisely controlling track difficulty, and exhibiting good real-time performance and controllability.

[0055] Figure 2 This is a schematic diagram of the SkiEnv skiing game environment of the present invention. The diagram shows the core elements of the game interface, including the skier (red square), obstacles (gray squares), and items (green squares). Obstacles and items move downwards at a constant speed from the top of the screen, while the skier, located at the bottom of the screen, only moves horizontally, simulating the visual effect of skiing downhill. This environment is developed based on Pygame, follows the OpenAI Gym interface specification, and supports parametric map resetting and repeatable evaluation.

[0056] Figure 3This is a training curve diagram of the PPO agent of this invention. The figure shows the cumulative reward as a function of training steps for three independent training sessions (random seeds of 10, 20, and 30 respectively). All three curves rise rapidly from the low reward region and eventually converge to a similar high-performance level (approximately 1200 points), verifying the training stability and effectiveness of the PPO algorithm in discrete actions and high-variance reward environments.

[0057] Figure 4 This is a schematic diagram of the reward prediction network of the present invention. The network uses a ResNet as its backbone and embeds a spatial attention module after the feature extraction layer. The input is a 40-dimensional track parameter vector, which passes through multiple convolutional layers and residual connections, and finally outputs a single predicted reward value through a fully connected layer. This structure can effectively capture high-order interactions between parameters and locally important features, achieving high-precision fitting of PPO agent scoring behavior.

[0058] Figure 5 The image shows a comparison of example tracks generated by the generator of this invention. The images depict two typical tracks with target difficulties of 750 points (high difficulty) and 1200 points (low difficulty), respectively. In the high-difficulty track on the left, obstacles (gray) are densely distributed and adjacent to items (green), forming multiple "trap areas"; in the low-difficulty track on the right, obstacles are sparse and items are easily obtainable. Although the visual differences are subtle, through the quantification of the reward function (collisions -100 points, collection +100 points), these layout differences are sufficient to lead to significant score changes.

[0059] This invention relates to a method, system, and storage medium for track generation based on reinforcement learning-guided generative adversarial networks. It can be used for dynamic difficulty adjustment and personalized track generation in skiing video games. The specific steps are as follows:

[0060] Step 1: Construct the skiing game environment SkiEnv, define the state space, action space and compound reward function, which are used for training reinforcement learning agents and quantitatively evaluating track difficulty;

[0061] The SkiEnv environment, developed using the Pygame and OpenAI Gym interfaces, simulates a dynamic scenario where skiers dodge obstacles and collect items as they descend a mountain. The environment is designed to provide a parameterizable and reproducibly evaluable testing platform for subsequent reinforcement learning agent training and quantitative analysis of track difficulty.

[0062] Specifically, the environment includes the following core elements:

[0063] State space: A 7-dimensional continuous vector representing key information about the skier's current position. The first two dimensions represent the skier's current speed (normalized to [-1,1]) and speed decay coefficient (normalized to [0,1]), respectively; the last five dimensions represent the skier's lateral position (normalized to [-1,1]) and the lateral and longitudinal coordinates of the nearest obstacles and obstacles (both normalized to [-1,1]). This state design enables the agent to perceive local information about the surrounding environment and make reasonable decisions.

[0064] Action Space: A discrete action space containing three selectable actions: 0 - stay still, 1 - move left, 2 - move right. This simple lateral movement control method conforms to the basic operation of a skiing game while reducing the complexity of reinforcement learning policy search;

[0065] Reward Function: To guide the agent in learning efficient and smooth gliding strategies, this invention designs a composite reward function, whose total reward consists of multiple components:

[0066]

[0067] in, The basic movement reward encourages agents to maintain a high forward speed; The reward for lateral exploration incentivizes agents to try different paths to avoid getting trapped in local optima; As a stagnation penalty, a small negative reward is given when the agent's speed is zero to prevent it from stagnating due to excessive caution; As a penalty for abrupt changes, a large negative reward is given when the agent's direction changes abruptly, in order to encourage a smooth turn; As an event reward, colliding with obstacles deducts 100 points, while collecting items adds 100 points. This reward mechanism fosters a "skilled skier" behavior pattern that prioritizes efficiency, smoothness, and resource collection, with the cumulative reward directly reflecting the challenge level of the track.

[0068] Step 2: Train the proximal policy optimization PPO reinforcement learning agent in the SkiEnv environment, and freeze the trained agent as an implicit track difficulty estimator.

[0069] Within the constructed SkiEnv environment, this invention employs the Proximal Policy Optimization (PPO) algorithm to train a reinforcement learning agent, enabling it to proficiently complete gliding tasks. After training, the agent's parameters are frozen and used as a standardized tool for subsequent evaluation of track difficulty. The PPO algorithm is chosen based on its stability and efficiency in continuous and high-dimensional action spaces, and its specific implementation is as follows:

[0070] Network architecture: The Actor-Critic architecture is adopted, in which both the Actor network and the Critic network are multilayer perceptrons (MLPs) with two hidden layers, each with 128 neurons, and the activation function is ReLU; the Actor network outputs the probability distribution of three actions, and the Critic network outputs the state value estimate.

[0071] Hyperparameter settings: The optimizer uses Adam, and the learning rate is set to... Discount factor GAE parameters Cutting factor The batch size is 64, and the entropy coefficient is 0.01 to encourage exploration. These parameters have been tuned to ensure stable convergence of the agent in a discrete action, high-variance reward environment.

[0072] Loss function:

[0073] The Actor loss function is:

[0074]

[0075] The Critic loss function is:

[0076]

[0077] Total loss:

[0078]

[0079] in This is the strategy entropy, used to facilitate exploration.

[0080] Training process: Multiple rounds of interaction are performed in the SkiEnv environment, with trajectory data collected and network parameters updated in each round. For example... Figure 3 As shown, after approximately 20,000 training steps, the agent's cumulative reward converged to approximately 1200 points, and multiple independent training sessions (with different random seeds) all achieved similar performance, indicating that the training process was stable and reliable. After training, the agent's parameters were frozen, and the cumulative reward it obtained on any track was used as the "difficulty score" for that track—the lower the score, the more difficult the track. This mechanism provides an objective and repeatable difficulty metric for subsequent generators.

[0081] Step 3: Randomly sample track parameters, use the frozen PPO agent to interact with SkiEnv to obtain the corresponding cumulative reward, and construct a {track parameters, reward} paired dataset; train a differentiable evaluation model, i.e., a reward prediction network, based on this paired dataset. The reward prediction network adopts a residual structure and spatial attention mechanism to realize the differentiable mapping from track parameters to rewards; the trained reward prediction network serves as the discriminator of the generative adversarial network, with fixed parameters.

[0082] Since the scoring process of the PPO agent's interaction with the environment is non-differentiable (involving random sampling of the policy network and deterministic physical simulation of the environment), it cannot be directly used for gradient backpropagation in the generator. Therefore, this invention proposes training a reward prediction network that can directly predict the cumulative reward obtained by the PPO agent on a given track based on the track parameters, thus achieving a differentiable mapping from parameters to reward. This network subsequently serves as the discriminator in a Generative Adversarial Network (GAN), and its parameters remain fixed during generator training. The specific construction process is as follows:

[0083] Track parameters are represented as follows: Each track contains 10 obstacles and 10 power-ups. Each element is described by its horizontal position (normalized to [-1,1]), vertical position (normalized to [-1,1]), and time offset (normalized to [0,1]). Therefore, the track parameters are a 40-dimensional vector (the first 20 dimensions correspond to obstacles, and the last 20 dimensions correspond to power-ups).

[0084] Dataset generation: 10,000 sets of 40-dimensional track parameters were randomly sampled and input into the SkiEnv environment. The PPO agent frozen in step 2 was used to run the complete game, recording the cumulative rewards to form a paired dataset of {track parameters, rewards}. To ensure broad data coverage, the sampling adopted a uniform distribution, covering various possible layout combinations.

[0085] Network Structure: The reward prediction network uses a ResNet as its backbone, with a spatial attention module embedded after the feature extraction layer. The residual structure helps mitigate gradient vanishing, enabling the network to learn deeper features; the spatial attention mechanism automatically focuses on local regions (such as areas with dense obstacles) in the track parameters that have a significant impact on difficulty. The network takes 40-dimensional parameters as input, passes through multiple residual blocks and attention layers, and finally outputs a single predicted reward value through a fully connected layer.

[0086] Training details: Huber loss is used as the loss function to enhance robustness to outliers. The optimizer is Adam, with a learning rate of 1e-3, and training continues until the validation set error converges. Figure 4 As shown, the trained network has a mean absolute error (MAE) of less than 50 on the test set, and a coefficient of determination of... A value above 0.83 indicates that it can accurately fit the scoring behavior of the PPO agent. After training, the network is fixed as the discriminator of the GAN for subsequent generator training.

[0087] Step 4: Construct a generator network with random noise and target difficulty value as input and track parameters as output; train the generator using a hybrid loss function, which includes score matching loss and diversity regularization loss, and introduce a dynamic normalization strategy to map the predicted reward output by the discriminator to a suitable range for optimization; optimize the generator parameters through backpropagation so that the generated tracks approach the target value in difficulty while maintaining content diversity.

[0088] Step 4.1: The generator network input is 10-dimensional random noise. and 1-dimensional target difficulty value (After normalization), the output is 40-dimensional track parameters. The network structure uses a multilayer perceptron, with the hidden layer dimension decreasing layer by layer, and the output layer using the Tanh activation function to constrain the parameters within a reasonable range.

[0089] Step 4.2: Dynamic Normalization Strategy: Normalize the original predicted reward output by the discriminator. Mapped to Gaussian function The interval mapping formula is:

[0090]

[0091] in With training steps Linear decay:

[0092]

[0093] This strategy allows for a larger error (wide Gaussian) in the early stages of training to promote exploration; in the later stages, the distribution is tightened to achieve fine optimization.

[0094] Step 4.3: The hybrid loss function consists of score matching loss and diversity regularization loss;

[0095] Score Matching Loss: This encourages the normalized prediction reward for generated samples to approach the optimal value of 1, as shown in the formula:

[0096]

[0097] in Batch size;

[0098] Diversity regularization loss: penalizes the similarity between generated samples, and the formula is:

[0099]

[0100] Total loss:

[0101]

[0102] Step 4.4: Train the generator using the Adam optimizer, iterating until convergence.

[0103] Furthermore, in step 5, when generating the track in real time, the user can specify any difficulty value (e.g., a score in the range of 600 to 1300), and the generator outputs the corresponding track parameters. The SkiEnv environment can then load these parameters to generate a playable ski track.

[0104] Step 5: After training the generator, deploy it in the real-time game system. When a user (or game AI) needs to generate a track of a specific difficulty, the system first receives the target difficulty value specified by the user (e.g., a score between 600 and 1300), then samples a random noise vector, concatenates the two, and inputs it into the generator. The generator's forward propagation outputs 40-dimensional track parameters, which precisely describe the position and time offset information of 10 obstacles and 10 props. Subsequently, the system calls the `reset(obstacles, props)` function of the `SkiEnv` environment to load these parameters and render a ski track that meets the difficulty requirements in real time. The entire process has minimal computational overhead and can be completed in milliseconds, meeting the needs of real-time games. The generated track can be directly used for dynamic loading of game levels or as difficulty-adaptive training material for AI agents, thus achieving end-to-end automated generation from user intent to playable content.

[0105] In particular, in some preferred embodiments of the present invention, a computer device is also provided, including a memory and a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the track generation method based on reinforcement learning-guided generative adversarial networks in any of the above embodiments.

[0106] In some other preferred embodiments of the present invention, a computer-readable storage medium is also provided, on which a computer program / instruction is stored, wherein when the computer program is executed by a processor, it implements the steps of the track generation method based on reinforcement learning-guided generative adversarial networks in any of the above embodiments.

[0107] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above embodiments of the track generation method based on reinforcement learning-guided generative adversarial networks, which will not be repeated here.

[0108] Computer-readable storage media encompass a variety of types, including persistent and non-persistent, portable and fixed. These media store information using different technologies, and the content can be machine instructions, data structures, program modules, or other types of data. Some typical examples of computer storage media include: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), various types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory and other storage technologies, optical storage media such as CD-ROM and digital video disc (DVD), magnetic storage devices such as magnetic tape and disks, and other non-transferable media used to store information accessible to computing devices. It is important to note that the computer-readable media described herein do not include temporary storage media, such as modulated data signals and carrier waves.

[0109] Those skilled in the art will further recognize that the operation of the module can be achieved using existing technical protocols or programs, without relying on new computer programs themselves. The units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0110] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0111] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A track generation method based on reinforcement learning guided generative adversarial network, characterized in that, Includes the following steps: Step 1: Construct the game environment, which defines a state space, an action space, and a composite reward function. The composite reward function is used to provide training feedback for the reinforcement learning agent and to quantitatively evaluate the difficulty of the track. Step 2: The reinforcement learning agent is trained using a reinforcement learning algorithm in the game environment, and the trained reinforcement learning agent is frozen as a track difficulty evaluator. Step 3: Randomly sample multiple sets of track parameters, use the frozen reinforcement learning agent to interact with the game environment to obtain the corresponding cumulative rewards, and construct a paired dataset containing the track parameters and the cumulative rewards; A differentiable evaluation model is trained based on the paired dataset. The differentiable evaluation model is used to realize a differentiable mapping from the track parameters to the cumulative reward. Step 4: Construct a generator network. The input of the generator network is random noise and the target difficulty value, and the output is track parameters. Use the differentiable evaluation model as the optimization guide model of the generator network, and train the generator network using a hybrid loss function. The hybrid loss function includes at least a first loss component for making the difficulty of the generated track approximate the target difficulty value and a second loss component for maintaining the diversity of the generated track content. Step 5: Deploy the trained generator network in the real-time game system. The system receives the target difficulty value specified by the user, concatenates it with the random noise, and inputs it into the generator network to generate the corresponding track parameters. The generated track parameters are then loaded into the game environment to render a track that meets the difficulty requirements in real time.

2. The method and system for ski course generation based on reinforcement learning guided generative adversarial network according to claim 1, characterized in that: In step 1, the state space is a 7-dimensional continuous vector, where the first two dimensions represent the current speed and speed decay coefficient of the game character, respectively; and the last five dimensions represent the lateral position of the skier and the lateral and longitudinal coordinates of the obstacles and props closest to the game character, respectively. The action space is a discrete action space, which includes staying still, moving left, and moving right; The composite reward function The total reward is: in, The base motion reward is determined based on the current speed of the game character. The reward for lateral exploration is determined based on the lateral position of the game character. The penalty for being stationary is determined based on the zero-speed state of the game character. To determine the sharp turn penalty based on the turning change rate of the game character, The event reward is determined based on the interaction results between the game character and obstacles or props. 3.The method and system for ski course generation based on reinforcement learning guided generative adversarial network according to claim 1, characterized in that: In step 2, the reinforcement learning algorithm is a near-end policy optimization algorithm, which uses an Actor-Critic network. In the game environment, multiple rounds of interaction are performed, and trajectory data is collected in each round. Based on the policy loss function and value loss function of the near-end policy optimization algorithm, and with preset discount factor, preset GAE parameters, preset pruning coefficient, preset batch size, and entropy regularization coefficient to promote exploration, the network parameters are updated until the cumulative reward converges. After training is completed, the parameters of the reinforcement learning agent are frozen, and the cumulative reward it obtains on any track is used as the difficulty score of that track.

4. The method and system for ski course generation based on reinforcement learning guided generative adversarial network according to claim 1, characterized in that: In step 3, the track parameters are N-dimensional vectors, which include the position information and time offset information of multiple obstacles and multiple props. The paired dataset is formed by randomly sampling multiple sets of track parameters, inputting them into the game environment, running the complete game with the frozen reinforcement learning agent, and recording the cumulative rewards.

5. The method and system for ski course generation based on reinforcement learning guided generative adversarial network according to claim 4, characterized in that: The differentiable evaluation model is a reward prediction network, and training the reward prediction network specifically includes: A reward prediction network is constructed with a residual network as the backbone and a spatial attention module embedded after the feature extraction layer. The reward prediction network takes the track parameters as input and outputs a single predicted reward value. After training, the reward prediction network is fixed as the optimization guide model for the generator network.

6. The ski track generation method and system based on reinforcement learning-guided generative adversarial networks according to claim 1, characterized in that: In step 4, the generator network input is M-dimensional random noise. and 1-dimensional target difficulty value The output is the track parameters. The generator network employs a multilayer perceptron structure, with the hidden layer dimension decreasing progressively, and the output layer using the Tanh activation function; the training of the generator network specifically includes: A dynamic normalization strategy is employed to transform the original predicted reward output by the differentiable evaluation model. The Gaussian function maps to a predetermined numerical range, and the mapping characteristics of the Gaussian function change dynamically with the increase of the number of training steps. Adopting a hybrid loss function Computing training loss, wherein, is a fraction matching loss, is a diversity regularization loss, is a diversity regularization coefficient; The generator is trained using the Adam optimizer, iterating until convergence.

7. The ski track generation method and system based on reinforcement learning-guided generative adversarial networks according to claim 6, characterized in that: The score matching loss For making the normalized predicted reward approach the optimal value, the formula is: wherein, is a batch size, is a normalized prediction reward for the th generated sample; The diversity regularizing loss The formula is: in, and The first The and the first A vector of track parameters for each generated sample. For the Euclidean norm, This is a parameter for the diversity scale.

8. The track generation method based on the reinforcement learning guided generative adversarial network according to claim 7, characterized in that: the normalized predicted reward is computed by the following Gaussian mapping function: wherein, is the raw predicted reward output by the differentiable evaluation model, is a reward scaling factor, is a center position parameter of the Gaussian function, is a width parameter of the Gaussian function; The width parameter With the number of training steps Decay in the following way: in, This is the initial value for the width parameter. This is the attenuation amount. Total training steps The minimum value of the width parameter is used; the decay strategy encourages exploration with a wide Gaussian distribution in the early stage of training, and achieves fine optimization with a narrow Gaussian distribution in the later stage of training.

9. A computer apparatus / device / system comprising a memory, a processor, and a computer program stored on the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

10. A computer readable storage medium having stored thereon a computer program, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images