Adaptive adjustment method based on reinforcement learning, computer equipment and medium

Abstract

The invention relates to the technical field of computers, and provides an adaptive adjustment method based on reinforcement learning, computer equipment and a medium. The adaptive adjustment method comprises the following steps: obtaining an input state, player actions and reference actions; and selectively adjusting the tolerance coefficient parameter or maintaining the tolerance coefficient parameter at least based on the difference between the player action and the reference action through an adaptive adjustment module, so that the difference between the player action and the reference action is stabilized in a preset range.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an adaptive adjustment method, computer device and medium based on reinforcement learning. Background Technology

[0002] With the development of applications in artificial intelligence, virtual reality, and augmented reality, coaching systems are widely used to provide instruction and guidance to players, helping them improve training quality and skill levels. Examples include AI Coach systems based on artificial intelligence models. However, different players have varying initial skill levels and learning progress. Therefore, it's necessary to adapt to individual player differences and provide personalized instruction to improve training effectiveness and player experience. Existing computer-aided coaching solutions rely on players setting their own experience and skill levels as reference points, such as beginner, intermediate, and expert levels. However, this static adjustment method and fixed grading lack personalized adjustment mechanisms for individual players and struggles to adapt to skill level improvements in a timely manner, resulting in lag and a poor training experience. Existing teaching and training schemes relying on artificial intelligence technology sometimes employ reinforcement learning (RL) techniques. These schemes learn optimal strategies through trial and error by having an agent interact with the environment. The agent takes actions in the environment and receives rewards as feedback, guiding it to adjust its strategies to maximize long-term cumulative rewards. For example, Chinese patent CN119417671B discloses a method that uses onboard sensors to collect intelligent driving data in real time, inputs this data into a decision model to obtain a standard driving behavior sequence, compares the driver's actual driving behavior sequence with the standard sequence to calculate the driving behavior deviation value, determines a teaching feedback strategy based on the deviation value, and uses the driver's feedback on the driving guidance information to update the training parameters of the current decision model. However, existing reinforcement learning-based teaching and training systems require frequent updates to training parameters based on driver feedback. This leads to frequent interventions in the training process of the reinforcement learning model, making it difficult to fully utilize the teaching model designed according to the highest skill level. It also increases the system adaptation cost when switching between different users, resulting in low automation efficiency and hindering large-scale promotion and application.

[0003] To address these technical challenges, this application proposes an adaptive adjustment method, computer device, and medium based on reinforcement learning. Summary of the Invention

[0004] Firstly, this application provides an adaptive adjustment method based on reinforcement learning. The adaptive adjustment method includes: obtaining an input state, a player action, and a reference action, wherein the reference action is a guided action result obtained by inputting the input state and a tolerance coefficient parameter into the reinforcement learning model under soft boundary condition constraints of the reward function of the reinforcement learning model; the input state at least includes states associated with the player action; and the reference action is used for instructional guidance. Through an adaptive adjustment module, at least based on the gap between the player action and the reference action, the tolerance coefficient parameter is selectively adjusted or maintained to change the decision constraints of the reinforcement learning model, thereby stabilizing the gap between the player action and the reference action within a preset range. The tolerance coefficient parameter is a dynamically adjustable control quantity used to adjust the soft boundary conditions in the reward function of the reinforcement learning model, and the reward function indicates the reward of the reinforcement learning model for taking a given action in a given state.

[0005] Through the first aspect of this application, by utilizing the dynamic adjustment mechanism of the tolerance coefficient parameter, the output reference action and teaching guidance based on the reference action dynamically match the player's current skill level, supporting the provision of real-time and personalized guidance, which is conducive to improving training effect and player experience, significantly reducing the system adaptation cost when switching between different users, and also supporting a personalized adjustment mechanism for the current user. By utilizing the built-in algorithm of the pre-designed adaptive adjustment module, the tolerance coefficient parameter can be automatically adjusted or maintained, improving automation efficiency and facilitating large-scale promotion and application.

[0006] In one possible implementation of the first aspect of this application, the adaptive adjustment module selectively adjusts or maintains the tolerance coefficient parameter, at least based on the gap between the player's action and the reference action, so that the gap between the player's action and the reference action is stabilized within a preset range. This includes: evaluating the behavioral deviation values ​​of the player's action relative to the reference action in multiple dimensions, and then combining the behavioral deviation values ​​of the multiple dimensions to obtain a comprehensive action deviation value of the player's action relative to the reference action; maintaining the tolerance coefficient parameter when the comprehensive action deviation value is within the preset range; and adjusting the tolerance coefficient parameter when the comprehensive action deviation value exceeds the preset range.

[0007] In one possible implementation of the first aspect of this application, the adaptive adjustment module is used to adjust the tolerance coefficient parameter based on a plurality of preset adjustment levels, and the gap between the player action and the reference action includes the distribution of the absolute value of the action deviation of the player action relative to the reference action.

[0008] In one possible implementation of the first aspect of this application, the plurality of discrete gears correspond sequentially from the lowest gear to the highest gear to a plurality of level benchmarks from the highest level to the lowest level, and the adaptive adjustment module is configured to: increase the tolerance coefficient parameter when the mean absolute value of the action deviation is greater than a preset high threshold; and decrease the tolerance coefficient parameter when the mean absolute value of the action deviation is less than a preset low threshold, wherein the preset high threshold and the preset low threshold define the preset range.

[0009] In one possible implementation of the first aspect of this application, the reinforcement learning model is used to provide an artificial intelligence policy with an adjustable level of aggression, wherein the tolerance coefficient parameter is used to dynamically adjust the adjustable level of aggression of the artificial intelligence policy to match the skill level associated with the player's actions.

[0010] In one possible implementation of the first aspect of this application, the adaptive adjustment module is further configured to determine an initial value of the tolerance coefficient parameter based on historical data associated with the player's actions.

[0011] In one possible implementation of the first aspect of this application, the adaptive adjustment module is further configured to selectively adjust the tolerance coefficient parameter or maintain the tolerance coefficient parameter based on the gap between the player action and the reference action and the result indicators related to the task completion status.

[0012] In one possible implementation of the first aspect of this application, when the task completion state does not meet the preset conditions and the gap between the player's action and the reference action exceeds the preset range, the adaptive adjustment module adjusts the tolerance coefficient parameter based on the action deviation statistics to change the policy constraint strength of the reinforcement learning model.

[0013] In one possible implementation of the first aspect of this application, the adaptive adjustment module and the reinforcement learning model together form a closed-loop adjustment structure, wherein the gap between the player action and the reference action is used to adjust the tolerance coefficient parameter, thereby affecting the guiding reference action generated by the reinforcement learning model under soft boundary condition constraints.

[0014] In one possible implementation of the first aspect of this application, the adaptive module is further configured to adjust the tolerance coefficient parameter based on player customization settings.

[0015] In one possible implementation of the first aspect of this application, the method can be applied to teaching guidance in a simulated car driving scenario, wherein the input state is the vehicle state and the track state, the player action is the player's car driving action, the reference action is the reference car driving action, and the soft boundary conditions include at least the upper limit of safe cornering speed, the upper limit of vehicle lateral acceleration, and the soft boundary of track width.

[0016] In one possible implementation of the first aspect of this application, the method can be applied to teaching guidance in a simulated aircraft driving scenario, wherein the input state is the aircraft state and the aircraft flight path state, the player action is the player's aircraft driving action, the reference action is the reference aircraft driving action, and the soft boundary conditions include aircraft speed, aircraft turning speed, and aircraft acceleration.

[0017] In one possible implementation of the first aspect of this application, the method can be applied to teaching guidance in a simulated sports training scenario, wherein the input state is the player's height, player's arm length, and player's posture, the player's action is the player's sports action, the reference action is the reference sports action, and the soft boundary conditions include at least the sports action force, sports action speed, and sports action acceleration.

[0018] In one possible implementation of the first aspect of this application, the hard boundary conditions of the reward function are not adjustable, and the hard boundary conditions of the reward function are preset based on the boundary constraints of the scene associated with the player's action. When the scene associated with the player's action is a simulated car driving scene, the hard boundary conditions of the reward function include a hard boundary of the track width.

[0019] Secondly, this application provides a computer device. The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following operations: obtaining an input state, a player action, and a reference action, wherein the reference action is obtained by inputting the input state and a tolerance coefficient parameter into a reinforcement learning model. The input state includes at least a state associated with the player action, and the reference action is used for instructional guidance. Through an adaptive adjustment module, at least based on the gap between the player action and the reference action, the tolerance coefficient parameter is selectively adjusted or maintained, thereby stabilizing the gap between the player action and the reference action within a preset range. The tolerance coefficient parameter is a dynamically adjustable control quantity used to adjust the soft boundary conditions in the reward function of the reinforcement learning model, and the reward function is used to indicate the reward of the reinforcement learning model for taking a given action in a given state.

[0020] Through the second aspect of this application, by utilizing the dynamic adjustment mechanism of the tolerance coefficient parameter, the output reference action and teaching guidance based on the reference action dynamically match the player's current skill level, supporting the provision of real-time and personalized guidance, which is conducive to improving training effects and player experience, significantly reducing the system adaptation cost when switching between different users, and also supporting a personalized adjustment mechanism for the current user. By utilizing the built-in algorithm of the pre-designed adaptive adjustment module, the tolerance coefficient parameter can be automatically adjusted or maintained, improving automation efficiency and facilitating large-scale promotion and application.

[0021] In one possible implementation of the second aspect of this application, the adaptive adjustment module is used to adjust the tolerance coefficient parameter based on a plurality of preset adjustment levels, and the gap between the player action and the reference action includes the distribution of the absolute value of the action deviation of the player action relative to the reference action.

[0022] In one possible implementation of the second aspect of this application, the adaptive adjustment module and the reinforcement learning model together form a closed-loop adjustment structure, wherein the gap between the player action and the reference action is used to adjust the tolerance coefficient parameter, thereby affecting the guiding reference action generated by the reinforcement learning model under soft boundary condition constraints.

[0023] Thirdly, this application provides a computer-readable storage medium. The computer-readable storage medium stores computer instructions that, when executed on a computer device, cause the computer device to perform operations including: obtaining an input state, a player action, and a reference action, wherein the reference action is obtained by inputting the input state and a tolerance coefficient parameter into a reinforcement learning model, the input state including at least a state associated with the player action, and the reference action used for instructional guidance; and selectively adjusting or maintaining the tolerance coefficient parameter, at least based on the gap between the player action and the reference action, through an adaptive adjustment module, thereby stabilizing the gap between the player action and the reference action within a preset range, wherein the tolerance coefficient parameter is a dynamically adjustable control quantity used to adjust the soft boundary conditions in the reward function of the reinforcement learning model, and the reward function is used to indicate the reward of the reinforcement learning model for taking a given action in a given state.

[0024] Through the third aspect of this application, by utilizing the dynamic adjustment mechanism of the tolerance coefficient parameter, the externally output reference actions and teaching guidance based on the reference actions dynamically match the player's current skill level, supporting the provision of real-time and personalized guidance, which is conducive to improving training effects and player experience, significantly reducing the system adaptation cost when switching between different users, and also supporting a personalized adjustment mechanism for the current user. Utilizing the built-in algorithm of the pre-designed adaptive adjustment module, the tolerance coefficient parameter can be automatically adjusted or maintained, improving automation efficiency and facilitating large-scale promotion and application.

[0025] In one possible implementation of the third aspect of this application, the adaptive adjustment module is used to adjust the tolerance coefficient parameter based on a plurality of preset adjustment levels, and the gap between the player action and the reference action includes the distribution of the absolute value of the action deviation of the player action relative to the reference action.

[0026] In one possible implementation of the third aspect of this application, the adaptive adjustment module and the reinforcement learning model together form a closed-loop adjustment structure, wherein the gap between the player action and the reference action is used to adjust the tolerance coefficient parameter, thereby affecting the guiding reference action generated by the reinforcement learning model under soft boundary condition constraints. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating an adaptive adjustment method based on reinforcement learning, provided for an embodiment of this application; Figure 2 A method based on the embodiments of this application is provided. Figure 1 The diagram shown is a schematic of the teaching and training system. Figure 3 A reference provided for an embodiment of this application Figure 1 The diagram illustrates the process of using adaptive adjustment methods for instruction in simulated driving scenarios. Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0029] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0030] It should be understood that in the description of this application, "at least one" means one or more, and "multiple" means two or more. In addition, the words "first," "second," etc., unless otherwise stated, are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance or order.

[0031] Figure 1 This is a flowchart illustrating an adaptive adjustment method based on reinforcement learning, provided as an embodiment of this application. Figure 1 As shown, the adaptive adjustment method based on reinforcement learning includes the following steps.

[0032] Step S101: Obtain the input state, player action, and reference action, wherein the reference action is a guided action result obtained by inputting the input state and tolerance coefficient parameter into the reinforcement learning model under the soft boundary condition constraint of the reward function of the reinforcement learning model, the input state includes at least the state associated with the player action, and the reference action is used for teaching guidance.

[0033] Step S103: Through the adaptive adjustment module, at least based on the gap between the player's action and the reference action, selectively adjust the tolerance coefficient parameter or maintain the tolerance coefficient parameter to change the decision constraints of the reinforcement learning model, thereby stabilizing the gap between the player's action and the reference action within a preset range. The tolerance coefficient parameter is a dynamically adjustable control quantity used to adjust the soft boundary conditions in the reward function of the reinforcement learning model. The reward function is used to indicate the reward of the reinforcement learning model for taking a given action in a given state.

[0034] See Figure 1The input state can be real-time acquired data or data estimated by an algorithm, such as data obtained by parsing a real-time video stream. Reinforcement Learning (RL) models are based on reinforcement learning techniques. Through the interaction between an agent and the environment, the agent learns the optimal policy through trial and error. The agent takes actions in the environment and receives rewards as feedback, guiding the agent to adjust its policy to maximize long-term cumulative rewards. Here, the reward function indicates the reward for taking a given action in a given state. The reward function can have a series of boundary conditions, which, along with the design of the reward function, influence the agent's decision-making process. Some boundary conditions are hard boundary conditions, which are not adjustable and can be pre-set according to the boundary constraints of the relevant scenario. Some boundary conditions are soft boundary conditions, which are adjustable and can be dynamically adjusted by adjusting the tolerance factor (TF). The tolerance factor is defined as: used to control the aggressiveness of the driving strategy. A smaller TF (True Force) indicates more aggressive driving (e.g., later braking, narrower lanes); a larger TF indicates more conservative driving (e.g., earlier braking, wider lanes). Depending on the specific application scenario of this adaptive adjustment method, corresponding boundary conditions (including soft and hard boundary conditions) and details of reinforcement learning algorithms such as the reward function can be set. Finally, a teaching and training system can be implemented in a specific application scenario, acting as an intelligent agent to provide reference actions based on the input state. Taking the teaching guidance in a simulated car driving scenario as an example, one training project helps players master the optimal line and speed when cornering. For novice players, failing to take the correct line when entering a corner, or failing to adopt an appropriate cornering speed, is a major reason why novice players lose control of the car and may go out of bounds. Novice players may brake too early or too late, or their cornering speed may be too low or too high, or they may choose a line that is too narrow or too wide. Therefore, when the scenario associated with the player's actions is a simulated car driving scenario, the hard boundary conditions of the reward function may include a hard boundary for the track width. If it is a simulated real-world professional track, track data can be obtained through high-precision map data collection, thereby setting hard boundary conditions for the reward function. If it is a virtual track generated in a game, the pre-defined physical boundaries of the virtual track can be obtained through the game's built-in interface as hard boundary conditions for the reward function.Conversely, to achieve the teaching and training objective of assisting players in improving their path planning and vehicle control abilities during cornering, the input states can be set to vehicle state and track state, the player's action to the player's car driving action, and the reference action to the reference car driving action. Furthermore, soft boundary conditions for the reward function can be set, including the upper limit of safe cornering speed, the upper limit of vehicle lateral acceleration, and a soft boundary for track width. Thus, by adjusting the tolerance coefficient parameter, the soft boundary conditions of the reward function can be dynamically adjusted. For example, the lateral acceleration-curvature-velocity relationship can be calculated using a formula, and the upper limit of safe cornering speed can be calculated, based on the vehicle's lateral acceleration limit and the corner curvature. The tolerance coefficient parameter can set an effective proportion of the speed limit value; for example, using 50% of the speed limit value as the actual effective speed limit value, that is, setting the actual soft boundary condition to half of the original speed limit value, or using 100% of the speed limit value as the actual effective speed limit value. Thus, by adjusting the tolerance coefficient parameter, the ratio of the actual effective speed limit value to the original speed limit value can be between 50% and 100%. It should be noted that the tolerance coefficient parameter is used to set the soft boundary conditions of the reward function of the reinforcement learning model, and the tolerance coefficient parameter is selectively adjusted or maintained, at least based on the difference between the player's action and the reference action, through the adaptive adjustment module. In this way, by selectively adjusting or maintaining the tolerance coefficient parameter, at least based on the difference between the player's action and the reference action, the decision constraints of the reinforcement learning model are changed, thereby stabilizing the difference between the player's action and the reference action within a preset range. The tolerance coefficient parameter is a dynamically adjustable control quantity used to adjust the soft boundary conditions in the reward function of the reinforcement learning model, which indicates the reward for taking a given action in a given state. The tolerance coefficient parameter merely alters the soft boundary conditions of the reward function, thus affecting the decision-making process of the reinforcement learning model and consequently the reference action output by the model. This allows the gap between the player's action and the reference action to be stabilized within a preset range. Conversely, by setting corresponding boundary conditions (including soft and hard boundary conditions) and designing details of the reinforcement learning algorithm, such as the reward function, the training process of the reinforcement learning model can be influenced, i.e., the process by which the agent learns the optimal policy.

[0035] In this way, a reinforcement learning model (serving as an instructional model) benchmarked against the highest experience and skill levels can be obtained through training to provide reference actions. In practical applications, the decision-making process of this instructional model is influenced by dynamically adjusting the tolerance coefficient parameter, thereby ensuring that the instructional model provides reference actions under the influence of the dynamically adjusted tolerance coefficient parameter, and the gap between the player's actions and the reference actions remains stable within a preset range. Thus, selectively adjusting or maintaining the tolerance coefficient parameter through an adaptive adjustment module means that frequent changes to the reinforcement learning model's training parameters are unnecessary, avoiding frequent interventions in the training process. This allows full utilization of the instructional model designed according to the highest skill level and enables continuous optimization of the instructional model and related agents. Therefore, unlike static adjustment methods that rely on fixed hierarchies to set the experience and skill levels of the reference objects corresponding to the coaching model, this approach... Figure 1 The adaptive adjustment method based on reinforcement learning, as shown, dynamically adjusts the tolerance coefficient parameter to change the soft boundary conditions of the reward function. This not only adheres to necessary constraints such as hard boundary conditions of the reward function but also avoids affecting the training parameters and training process of the reinforcement learning model. Furthermore, by influencing the decision-making process of the reinforcement learning model, it controls the reference action output by the model, ultimately stabilizing the gap between the player's action and the reference action within a preset range. Thus, by utilizing the dynamic adjustment mechanism of the tolerance coefficient parameter, the reinforcement learning model, along with the related decision-making model and agent, achieve a stable system state. The reference action output and the teaching guidance based on the reference action dynamically match the player's current skill level, supporting real-time and personalized guidance, which is beneficial for improving training effectiveness and player experience. Additionally, because it avoids directly modifying the training parameters of the reinforcement learning model while influencing its decision-making process—for example, without modifying the design of the reward function—but instead changes the reference action output by the model through modifying the soft boundary conditions of the reward function, this means… Figure 1 The reinforcement learning-based adaptive adjustment method and its associated teaching and training system, as illustrated, do not require updating training parameters based on user feedback. Therefore, a single method and system can be adapted to different users. During user interaction, the dynamic adjustment mechanism of the tolerance coefficient parameter mentioned above achieves system stability. This design significantly reduces the system adaptation cost when switching between different users. It also supports personalized adjustment mechanisms for the current user. Using the pre-designed adaptive adjustment module's built-in algorithm, the tolerance coefficient parameter can be automatically adjusted or maintained, improving automation efficiency and facilitating large-scale application.

[0036] Continue reading Figure 1 Depending on the specific application scenario of the adaptive adjustment method, corresponding boundary conditions (including soft and hard boundary conditions) and details of reinforcement learning algorithms such as reward functions can be set. Finally, a teaching and training system can be implemented in a specific application scenario, and the resulting agent can provide reference actions under the optimal policy based on the input state. Therefore, the entire process from data acquisition to device control can be realized. Based on real-time acquired data or data estimated by algorithms, and combined with specific scenarios and situations, personalized teaching guidance and auxiliary interactive experiences can be provided to users. Using a reinforcement learning model, the automated execution process of intervention strategies and timing can be achieved in multiple dimensions, including early state acquisition, policy formulation and output, obtaining player input, and subsequent policy judgment. For example, it can determine when to take over and the judgment strategy for the takeover ratio. This can be achieved through the adaptive adjustment module based on the automated script, determining whether intervention is needed and adjusting the tolerance coefficient parameter, or determining whether intervention is not needed and maintaining the tolerance coefficient parameter. Furthermore, Figure 1The reinforcement learning-based adaptive adjustment method and the teaching and training system based on it, along with the related intelligent agent, can adopt multimodal reminder strategies. For example, they can provide intuitive reminders to users through tactile feedback, or through a combination of visual and auditory reminders, thus lowering the barrier to receiving instructional information for users. Furthermore, adaptive or supervised learning mechanisms can be integrated to continuously optimize the reminder scheme based on different user responses. In addition to using reference actions for teaching guidance, the system output can be further utilized to include both guidance and warning components in user reminders. The guidance component provides instructional guidance, such as showing the optimal path and guiding the user to perform optimal actions (e.g., helping the user turn the steering wheel or apply the brakes). The warning component provides a stronger alert when the user's operation is significantly flawed (e.g., excessive lateral deviation or driving in the wrong direction), such as by vibrating the steering wheel. Therefore, based on the system's reference actions and user reminders, the output strategy can be further determined based on the hardware interacting with the user. For example, after determining the output strategies for the steering wheel and pedals, the interaction parameters (feedback force) of the steering wheel and base can be adjusted based on the actual hardware conditions and user operation. This could include synchronized assistance mechanisms for the steering wheel and pedals. For instance, first, the optimal trajectory and optimal vehicle input are calculated; then, an initial teaching strategy is determined, such as assisting the user to press the pedals 50%; subsequently, the system can automatically or manually adjust the teaching strategy based on the user's response, such as adjusting the proportion of pedal input. Furthermore, the teaching strategy can include complete takeover, assisting the user with partial corrections, adjusting interaction parameters, and issuing warnings for large deviations. Thus, Figure 1The reinforcement learning-based adaptive adjustment method shown can provide a large, end-to-end integrated model for both path planning and vehicle control when the player's actions are associated with a simulated car driving scenario. This means that path planning and vehicle control are not treated as two separate upstream and downstream modules. Instead, it utilizes input states (such as actual or estimated data collected in the game) to output the optimal path plus the optimal vehicle control signal. In practical applications, player actions (e.g., steering wheel and pedal operations) can be acquired through hardware that interacts with the player, and vehicle and game scene states can be obtained through software such as the game itself. This data is then processed by the reinforcement learning model and output to the gaming device according to the teaching strategy, and finally to the user via the hardware. The reinforcement learning model's data processing can include providing multi-modal alerts from visual, auditory, and tactile perspectives, and can also include determining the intervention time (e.g., pre-operation teaching, in-operation teaching, post-operation teaching) and the intervention ratio (e.g., partial intervention, full intervention) based on the teaching strategy. Finally, when the system outputs feedback to the user, it can provide various methods such as visual, auditory, and tactile cues. It can also offer teaching feedback based on the teaching strategy and the actual hardware conditions. Furthermore, it can provide additional warnings—stronger alerts—to notify the user of serious operational problems. In this way, an end-to-end workflow, a teaching strategy mechanism, and multimodal alerts are implemented, with the alert scheme optimized based on user feedback are achieved.

[0037] In short, Figure 1 The reinforcement learning-based adaptive adjustment method shown utilizes a dynamic adjustment mechanism for the tolerance coefficient parameter, outputs reference actions, and provides instructional guidance based on those actions. This dynamically matches the player's current skill level, supports real-time and personalized guidance, and is beneficial for improving training effectiveness and player experience. It significantly reduces the system adaptation cost when switching between different users. Furthermore, it supports a personalized adjustment mechanism for the current user. Using the built-in algorithm of the pre-designed adaptive adjustment module, it can automatically adjust or maintain the tolerance coefficient parameter, improving automation efficiency and facilitating large-scale application.

[0038] Figure 2 A method based on the embodiments of this application is provided. Figure 1 The diagram shows a teaching and training system. Figure 2As shown, the teaching and training system 200 includes two modules: an adaptive adjustment module 210 and a reinforcement learning model 220. The adaptive adjustment module 210 receives player actions and reference actions to output tolerance coefficient parameters. The reinforcement learning model 220 receives the input state and tolerance coefficient parameters, and outputs a reference action based on the input state under the influence of the tolerance coefficient parameters. In some embodiments, the adaptive adjustment module 210 can also receive other variables for outputting the tolerance coefficient parameters, such as whether the player finishes the game within the specified time and player-customized settings.

[0039] See Figure 1 and Figure 2In some embodiments, the adaptive adjustment module selectively adjusts or maintains the tolerance coefficient parameter, at least based on the difference between the player's action and the reference action, so that the difference between the player's action and the reference action is stabilized within a preset range. This includes: evaluating the behavioral deviation values ​​of the player's action relative to the reference action in multiple dimensions, and then combining the behavioral deviation values ​​of the multiple dimensions to obtain a comprehensive action deviation value of the player's action relative to the reference action; maintaining the tolerance coefficient parameter when the comprehensive action deviation value is within the preset range; and adjusting the tolerance coefficient parameter when the comprehensive action deviation value exceeds the preset range. Here, the player's action and the reference action can be broken down into multiple dimensions for quantitative comparison and analysis. For example, assuming the reinforcement learning model is used for instruction in a simulated car driving scenario, player actions and reference actions can be broken down into multiple quantitative indicators such as vehicle position, speed, acceleration, steering wheel angle, accelerator pedal position, and brake pedal position. Alternatively, behavioral deviation values ​​across multiple dimensions, including positional deviation, heading deviation, longitudinal control deviation, steering deviation, and acceleration deviation, can be evaluated. Finally, by integrating this information, a comprehensive action deviation value relative to the reference action can be assessed. For example, the mean and standard deviation can be referenced, and a risk assessment module can be combined. Weighting coefficients can also be used to measure the importance of different dimensions. The resulting comprehensive behavioral deviation value indicates the overall degree of difference between the player's actions and the reference action. This can aid in decision-making for teaching strategies and provide instructional guidance to players, and may also include various modal alerts and warnings. Furthermore, by determining whether the comprehensive action deviation value exceeds a preset range, the tolerance coefficient parameter can be dynamically adjusted, facilitating intelligent and automated judgment of when to intervene and the intervention strategy, providing users with personalized guidance and improving training effectiveness. In some embodiments, the adaptive adjustment module selectively adjusts or maintains the tolerance coefficient parameter, based at least on the difference between the player's action and the reference action, so that the difference between the player's action and the reference action is stabilized within a preset range. This includes: evaluating the behavioral deviation values ​​of the player's action relative to the reference action in multiple dimensions, and then combining the behavioral deviation values ​​of the multiple dimensions to obtain a comprehensive action deviation value of the player's action relative to the reference action; maintaining the tolerance coefficient parameter when the comprehensive action deviation value is within the preset range; and adjusting the tolerance coefficient parameter when the comprehensive action deviation value exceeds the preset range.

[0040] See Figure 1 and Figure 2In some embodiments, the adaptive adjustment module is used to adjust the tolerance coefficient parameter based on multiple preset adjustment levels, and the gap between the player's action and the reference action includes the distribution of the absolute value of the action deviation of the player's action relative to the reference action. In some embodiments, the adaptive adjustment module is used to adjust the tolerance coefficient parameter according to multiple discrete levels, and the gap between the player's action and the reference action includes the distribution of the absolute value of the action deviation of the player's action relative to the reference action. Thus, the tolerance coefficient parameter can be preset with multiple levels, such as from level 1 to level 10, which facilitates subsequent strategy decision-making and dynamic adjustment mechanisms. Recording the distribution of the absolute value of action deviation helps determine when to intervene and the intervention strategy. When the distribution of the absolute value of action deviation is relatively large in most cases, for example by comparing the mean of the absolute value of action deviation with a preset value, this means that the gap between the player's action and the reference action is large. This reflects a large gap between the player's current skill level and the skill level of the coach model corresponding to the reinforcement learning model, and the tolerance coefficient parameter can be dynamically adjusted to match the player's current skill level by taking corresponding adjustment actions. Conversely, when the distribution of the absolute value of the action deviation is relatively small in most cases, for example by comparing the mean of the absolute value of the action deviation with another preset value, it means that the difference between the player's action and the reference action is small. This reflects that the difference between the player's current skill level and the skill level of the coach model corresponding to the reinforcement learning model as the reference is small. The tolerance coefficient parameter can be dynamically adjusted to match the player's current skill level by taking corresponding adjustment actions.

[0041] See Figure 1 and Figure 2In some embodiments, the plurality of discrete gears correspond sequentially from the lowest to the highest gear to multiple level benchmarks from the highest to the lowest level. The adaptive adjustment module is configured to: increase the tolerance coefficient parameter when the mean absolute value of the movement deviation is greater than a preset high threshold; and decrease the tolerance coefficient parameter when the mean absolute value of the movement deviation is less than a preset low threshold, wherein the preset high threshold and the preset low threshold define the preset range. In some embodiments, the plurality of discrete gears correspond sequentially from the lowest to the highest gear to multiple level benchmarks from the highest to the lowest level. The adaptive adjustment module is configured to: increase the tolerance coefficient parameter by one gear when the mean absolute value of the movement deviation is greater than the preset high threshold; and decrease the tolerance coefficient parameter by one gear when the mean absolute value of the movement deviation is less than the preset low threshold, wherein the preset high threshold and the preset low threshold define the preset range. Thus, an adjustable tolerance coefficient mechanism and adaptive adjustment method are implemented, supporting personalized teaching and training strategies for players.

[0042] See Figure 1 and Figure 2 In some embodiments, the reinforcement learning model is used to provide an AI policy with adjustable aggression, and the tolerance coefficient parameter is used to dynamically adjust the adjustable aggression of the AI ​​policy to match the skill level associated with the player's actions. Thus, by dynamically adjusting the tolerance coefficient parameter, the training parameters of the reinforcement learning model do not need to be changed. By influencing the decision-making process of the reinforcement learning model, the adjustable aggression of the AI ​​policy can be controlled. For example, a more aggressive strategy style can be adopted, such as braking later and choosing a narrower line when cornering, or a more conservative strategy style can be adopted, such as braking earlier and choosing a wider line when cornering. The tolerance coefficient parameter can be used to set soft boundary conditions for the reward function, such as setting soft boundaries for track width related to the tolerance coefficient parameter and limiting the track width that the policy can utilize. For example, it can be set that when the tolerance coefficient parameter is small, more aggressive cornering is allowed, while when the tolerance coefficient parameter is large, the driving line is required to maintain a distance from the track boundary to avoid going out of bounds. This design helps players control their cornering speed. High cornering speeds are a major cause of novice players going out of bounds and losing control. This is addressed by setting a speed limit function related to curvature and tolerance parameters. A smaller tolerance parameter results in a higher speed limit, allowing for higher cornering speeds; a larger tolerance parameter results in a lower speed limit, penalizing speeding.

[0043] See Figure 1 and Figure 2In some embodiments, the adaptive adjustment module is further configured to determine an initial value for the tolerance coefficient parameter based on historical data associated with the player's actions. This allows for the reading of the player's past operational history, such as their fastest lap time, to query a lap time list and select the most suitable gear, thus facilitating the matching of player skill levels.

[0044] See Figure 1 and Figure 2 In some embodiments, the adaptive adjustment module is further configured to selectively adjust or maintain the tolerance coefficient parameter based on the gap between the player's action and the reference action, as well as result indicators related to the task completion status. In some embodiments, the adaptive adjustment module is further configured to selectively adjust or maintain the tolerance coefficient parameter based on the gap between the player's action and the reference action, and whether the player finishes the race within the specified time. This helps support a dynamic adjustment mechanism. When a novice player fails to finish the race, the distribution of the absolute value of the action deviation under the initial tolerance coefficient parameter can be statistically analyzed, and corresponding adjustments can be made. After adjustment, lap times and loop optimization can be re-evaluated to improve training effectiveness.

[0045] See Figure 1 and Figure 2 In some embodiments, when the task completion status does not meet preset conditions and the difference between the player's action and the reference action exceeds the preset range, the adaptive adjustment module adjusts the tolerance coefficient parameter based on action deviation statistics to change the policy constraint strength of the reinforcement learning model. In some embodiments, the adaptive adjustment module is further configured to, when the player does not finish the race within the specified time and the difference between the player's action and the reference action exceeds the preset range, increase the tolerance coefficient parameter to adopt a more conservative strategy style, or decrease the tolerance coefficient parameter to adopt a more aggressive strategy style, based on the average action deviation of the player's action relative to the reference action. Thus, a method for adaptively adjusting the tolerance coefficient parameter can be implemented by combining discrete gears, a stored lap time list, and deviation statistics logic, dynamically adjusting based on the lap time list and action deviation distribution. Utilizing the definition of the tolerance coefficient parameter and its influence on track soft boundaries and cornering speed, the tolerance coefficient parameter is used as an input parameter to dynamically adjust the aggressiveness of the artificial intelligence driving strategy, providing users with personalized guidance. This achieves a personalized experience by fine-tuning the AI ​​strategy through a tolerance coefficient to match player skills and reduce loss of control; dynamic adaptation, automatically adjusting based on real-time performance to improve training efficiency and enjoyment; friendliness to novice players, helping them complete races with conservative settings and gradually encouraging improvement with more aggressive ones; and a data-driven adjustment mechanism that uses lap times and deviation data to ensure scientific and reasonable adjustments.

[0046] See Figure 1 and Figure 2 In some embodiments, the adaptive adjustment module and the reinforcement learning model together form a closed-loop adjustment structure. The difference between the player's action and the reference action is used to adjust the tolerance coefficient parameter, thereby influencing the guiding reference action generated by the reinforcement learning model under soft boundary constraints. In some embodiments, the adaptive adjustment module and the reinforcement learning model together provide a negative feedback mechanism. The input of the negative feedback mechanism is the difference between the player's action and the reference action, and the output of the negative feedback mechanism is the reference action. The negative feedback mechanism dynamically adjusts the tolerance coefficient parameter to achieve the reference action following the player's action. Thus, by dynamically adjusting the tolerance coefficient parameter to influence the decision-making process of the teaching model, the teaching model provides a reference action under the dynamically adjusted tolerance coefficient parameter, and the difference between the player's action and the reference action remains stable within a preset range. This negative feedback closed-loop control design, in order to... Figure 2 For example, by adjusting the tolerance coefficient parameter, the reference action output by the reinforcement learning model 220 can be influenced. Finally, a negative feedback loop is used to stabilize the difference between the player's action and the reference action within a preset range. This negative feedback mechanism enables the reference action to follow the player's action. Furthermore, unlike static adjustment methods that rely on fixed grading to set the experience and skill levels of the reference object corresponding to the coaching model, dynamically adjusting the tolerance coefficient parameter to change the soft boundary conditions of the reward function not only adheres to necessary constraints such as hard boundary conditions of the reward function but also avoids affecting the training parameters and process of the reinforcement learning model. Moreover, by influencing the decision-making process of the reinforcement learning model to control the reference action output, a stable reinforcement learning model, along with related decision-making models and agents, dynamically matches the player's current skill level with the output reference action and the teaching guidance based on the reference action, supporting real-time and personalized guidance, which is beneficial for improving training effectiveness and player experience.

[0047] See Figure 1 and Figure 2 In some embodiments, the adaptive module is further configured to adjust the tolerance coefficient parameter based on player-customized settings. This support for player-customized settings helps provide personalized teaching and training guidance.

[0048] See Figure 1 and Figure 2In some embodiments, the method can be applied to teaching guidance in simulated car driving scenarios, wherein the input states are vehicle state and track state, the player action is the player's car driving action, the reference action is the reference car driving action, and the soft boundary conditions include at least the upper limit of safe cornering speed, the upper limit of vehicle lateral acceleration, and a soft boundary of track width. In some embodiments, the reinforcement learning model is used for teaching guidance in simulated car driving scenarios, wherein the input states are vehicle state and track state, the player action is the player's car driving action, the reference action is the reference car driving action, and the soft boundary conditions include the upper limit of safe cornering speed, the upper limit of vehicle lateral acceleration, and a soft boundary of track width. In this way, adaptive teaching in simulated car driving scenarios is realized. The corresponding boundary conditions (including soft and hard boundary conditions) and details of reinforcement learning algorithms such as reward functions are set. Finally, a teaching and training system in specific application scenarios is realized. The resulting agent can provide reference actions under the optimal strategy based on the input state, realizing the entire process from data acquisition to device control. It can provide users with personalized teaching guidance and auxiliary interactive experience based on real-time acquired data or data estimated by algorithms, and can also be combined with specific scenarios and situations.

[0049] See Figure 1 and Figure 2 In some embodiments, the method can be applied to teaching guidance in simulated aircraft piloting scenarios. The input states are the aircraft state and flight path state, the player action is the player's aircraft piloting action, the reference action is a reference aircraft piloting action, and the soft boundary conditions include aircraft speed, aircraft turning speed, and aircraft acceleration. In some embodiments, the reinforcement learning model is used for teaching guidance in simulated aircraft piloting scenarios. The input states are the aircraft state and flight path state, the player action is the player's aircraft piloting action, the reference action is a reference aircraft piloting action, and the soft boundary conditions include aircraft speed, aircraft turning speed, and aircraft acceleration. Thus, adaptive teaching in simulated aircraft piloting scenarios is achieved. This involves setting corresponding boundary conditions (including soft and hard boundary conditions) and designing reinforcement learning algorithm details such as reward functions. Finally, a teaching and training system is implemented in specific application scenarios. The resulting agent can provide reference actions under the optimal policy based on the input state, realizing the entire process from data acquisition to device control. It can provide personalized teaching guidance and auxiliary interactive experiences to users based on real-time acquired data or data estimated through algorithms, and can also be tailored to specific scenarios and situations.

[0050] See Figure 1 and Figure 2In some embodiments, the method can be applied to teaching guidance in simulated sports training scenarios, wherein the input states are player height, player arm length, and player posture; the player action is a player sports action; the reference action is a reference sports action; and the soft boundary conditions include at least the force, speed, and acceleration of the sports action. In some embodiments, the reinforcement learning model is used to teach guidance in simulated sports training scenarios, wherein the input states are player height, player arm length, and player posture; the player action is a player sports action; the reference action is a reference sports action; and the soft boundary conditions include the force, speed, and acceleration of the sports action. In this way, adaptive teaching in simulated sports training scenarios is realized. The corresponding boundary conditions (including soft and hard boundary conditions) and details of reinforcement learning algorithms such as reward functions are set. Finally, a teaching and training system in specific application scenarios is realized. The resulting intelligent agent can provide reference actions under the optimal strategy based on the input state, realizing the entire process from data acquisition to device control. It can provide users with personalized teaching guidance and auxiliary interactive experience based on real-time acquired data or data estimated by algorithms, and can also be combined with specific scenarios and situations.

[0051] See Figure 1 and Figure 2In some embodiments, the hard boundary conditions of the reward function are non-adjustable, and are pre-set based on the boundary constraints of the scene associated with the player's action. When the scene associated with the player's action is a simulated car driving scene, the hard boundary conditions of the reward function include a hard boundary for the track width. The reward function can have a series of boundary conditions, which, together with the design of the reward function, influence the agent's decision-making process. Some boundary conditions are hard boundary conditions, which are non-adjustable and can be pre-set according to the boundary constraints of the relevant scene. Some boundary conditions are soft boundary conditions, which are adjustable and can be dynamically adjusted by adjusting the tolerance coefficient parameter. Depending on the specific application scenario of this adaptive adjustment method, corresponding boundary conditions (including soft and hard boundary conditions) and reinforcement learning algorithm details such as the reward function can be set, ultimately realizing a teaching and training system in a specific application scenario, which can act as an agent to provide reference actions based on the input state. Taking the specific application scenario of this adaptive adjustment method as a teaching guide in a simulated car driving scenario as an example, one training project is to help players master the optimal route and optimal cornering speed when driving a car through a corner. For novice players, failing to take the correct route when entering a corner, or failing to adopt an appropriate cornering speed, is a major reason why novice players lose control of the car and may go out of bounds. Novice players may brake too early or too late, or their cornering speed may be too low or too high, or they may choose a route that is too narrow or too wide. Therefore, when the scenario associated with the player's actions is a simulated car driving scenario, the hard boundary conditions of the reward function can include a hard boundary of the track width. If it is a simulated professional track that exists in reality, track data can be obtained through high-precision map data collection to set the hard boundary conditions of the reward function. If it is a virtual track generated in a game, the pre-defined physical boundaries of the virtual track can be obtained through the game's built-in interface as the hard boundary conditions of the reward function. Conversely, to achieve the teaching and training objective of assisting players in improving their path planning and vehicle control abilities during cornering, the input states can be set to vehicle state and track state, the player's action to the player's car driving action, and the reference action to the reference car driving action. Furthermore, soft boundary conditions for the reward function can be set, including a safe cornering speed limit, a vehicle lateral acceleration limit, and a track width soft boundary. Thus, by adjusting the tolerance coefficient parameters, the soft boundary conditions of the reward function can be dynamically adjusted. For example, the lateral acceleration-curvature-velocity relationship can be calculated using formulas, and the safe cornering speed limit, based on the vehicle's lateral acceleration limit and the corner curvature, can be calculated.The tolerance coefficient parameter can set the effective proportion of the speed limit value. For example, using 50% of the speed limit value as the actual effective speed limit value means setting the actual soft boundary condition to half of the original speed limit value. Alternatively, using 100% of the speed limit value as the actual effective speed limit value. Thus, by adjusting the tolerance coefficient parameter, the proportion of the actual effective speed limit value to the original speed limit value can be between 50% and 100%. It should be noted that the tolerance coefficient parameter is used to set the soft boundary condition of the reward function of the reinforcement learning model. Furthermore, through the adaptive adjustment module, the tolerance coefficient parameter is selectively adjusted or maintained, at least based on the difference between the player's action and the reference action. In this way, the tolerance coefficient parameter only changes the soft boundary condition of the reward function, thus affecting the decision-making process of the reinforcement learning model and consequently affecting the reference action output by the reinforcement learning model. This allows the difference between the player's action and the reference action to be stabilized within a preset range. Conversely, by setting corresponding boundary conditions (including soft and hard boundary conditions) and designing reward functions, the training process of the reinforcement learning model can be influenced, which in turn affects the process by which the agent learns the optimal policy. In this way, a teaching model (or a similar decision-making model based on reinforcement learning techniques) benchmarked against the highest experience and skill levels can be obtained through training to provide reference actions. In practical applications, the decision-making process of this teaching model is influenced by dynamically adjusting the tolerance coefficient parameter, thereby ensuring that the teaching model provides reference actions under the dynamically adjusted tolerance coefficient parameter, and the gap between the player's actions and the reference actions remains stable within a preset range. Thus, selectively adjusting or maintaining the tolerance coefficient parameter through an adaptive adjustment module means that the training parameters of the reinforcement learning model do not need to be frequently modified, thus avoiding frequent intervention in the training process. This allows full utilization of the teaching model designed according to the highest skill level and continuous optimization of the teaching model and the related agent.

[0052] Figure 3 A reference provided for an embodiment of this application Figure 1 The diagram illustrates the process of using an adaptive adjustment method for instruction in simulated driving scenarios. Figure 3 As shown, an exemplary teaching guidance process in a simulated driving scenario includes the following steps.

[0053] Step S301: Perform initial settings. Based on the player's fastest lap time, query the lap time list and select the gear with the most suitable tolerance parameter.

[0054] Step S303: Make dynamic adjustments. If a novice player cannot finish the game, statistically analyze the distribution of the absolute value of the deviation between the player's actions and the reference actions under the initial tolerance coefficient parameters.

[0055] Step S305: When the mean absolute value of the action deviation is greater than the high threshold, the tolerance coefficient parameter is increased by one level to adopt a more conservative strategy style. When the mean absolute value of the action deviation is less than the low threshold, the tolerance coefficient parameter is decreased by one level to adopt a more aggressive strategy style.

[0056] Step S307: After adjusting the tolerance coefficient parameter, re-evaluate the player's lap time, and then iterate to optimize.

[0057] See Figure 3 By utilizing a dynamic adjustment mechanism for the tolerance coefficient parameter, a reinforcement learning model, along with related decision-making models and agents, achieves a stable system state. The output of reference actions and teaching guidance based on these actions dynamically matches the player's current skill level, supporting real-time and personalized guidance, thus improving training effectiveness and player experience. This enables adaptive teaching in simulated car driving scenarios. By setting corresponding boundary conditions (including soft and hard boundary conditions) and designing reward functions, along with other reinforcement learning algorithm details, a teaching and training system is implemented in specific application scenarios. The resulting agent can provide reference actions based on the optimal strategy according to the input state, realizing the entire process from data acquisition to device control. It can provide personalized teaching guidance and auxiliary interactive experiences to users based on real-time acquired data or data estimated through algorithms, combined with specific scenarios and situations.

[0058] See Figure 1 , Figure 2 and Figure 3In some embodiments, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The operations performed by the processor when executing the computer program include: obtaining an input state, a player action, and a reference action, wherein the reference action is obtained by inputting the input state and a tolerance coefficient parameter into a reinforcement learning model, the input state including at least a state associated with the player action, and the reference action used for instructional guidance; and selectively adjusting or maintaining the tolerance coefficient parameter, at least based on the gap between the player action and the reference action, through an adaptive adjustment module, such that the gap between the player action and the reference action is stabilized within a preset range, wherein the tolerance coefficient parameter is a dynamically adjustable control quantity used to adjust the soft boundary conditions in the reward function of the reinforcement learning model, the reward function indicating the reward of the reinforcement learning model for taking a given action in a given state. In some embodiments, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The operations performed by the processor when executing the computer program include: obtaining an input state, a player action, and a reference action, wherein the reference action is obtained by inputting the input state and a tolerance coefficient parameter into a reinforcement learning model, the input state including at least the state associated with the player action, and the reference action used for instructional guidance; and selectively adjusting or maintaining the tolerance coefficient parameter through an adaptive adjustment module, based at least on the gap between the player action and the reference action, thereby stabilizing the gap between the player action and the reference action within a preset range, wherein the tolerance coefficient parameter is used to set the soft boundary conditions of the reward function of the reinforcement learning model, and the reward function is used to indicate the reward of the reinforcement learning model for taking a given action in a given state. In this way, by utilizing the dynamic adjustment mechanism of the tolerance coefficient parameter, the output reference action and teaching guidance based on the reference action dynamically match the player's current skill level, supporting the provision of real-time and personalized guidance, which is conducive to improving training effects and player experience, significantly reducing the system adaptation cost when switching between different users, and also supporting a personalized adjustment mechanism for the current user. Using the built-in algorithm of the pre-designed adaptive adjustment module, the tolerance coefficient parameter can be automatically adjusted or maintained, improving automation efficiency and facilitating large-scale promotion and application.

[0059] See Figure 1 , Figure 2 and Figure 3In some embodiments, referring to the aforementioned computer device, the adaptive adjustment module is used to adjust the tolerance coefficient parameter based on multiple preset adjustment levels, and the gap between the player's action and the reference action includes the distribution of the absolute value of the action deviation of the player's action relative to the reference action. In some embodiments, referring to the aforementioned computer device, the adaptive adjustment module is used to adjust the tolerance coefficient parameter according to multiple discrete levels, and the gap between the player's action and the reference action includes the distribution of the absolute value of the action deviation of the player's action relative to the reference action. Thus, the tolerance coefficient parameter can be preset with multiple levels, for example from level 1 to level 10, which facilitates subsequent strategy decision-making and dynamic adjustment mechanisms. Recording the distribution of the absolute value of action deviation helps determine when to intervene and the intervention strategy. When the distribution of the absolute value of action deviation is relatively large in most cases, for example by comparing the mean of the absolute value of action deviation with a preset value, this means that the gap between the player's action and the reference action is large. This reflects a large gap between the player's current skill level and the skill level of the coach model corresponding to the reinforcement learning model as the reference object. Corresponding adjustment actions can be taken to dynamically adjust the tolerance coefficient parameter to match the player's current skill level. Conversely, when the distribution of the absolute value of the action deviation is relatively small in most cases, for example by comparing the mean of the absolute value of the action deviation with another preset value, it means that the difference between the player's action and the reference action is small. This reflects that the difference between the player's current skill level and the skill level of the coach model corresponding to the reinforcement learning model as the reference is small. The tolerance coefficient parameter can be dynamically adjusted to match the player's current skill level by taking corresponding adjustment actions.

[0060] See Figure 1 , Figure 2 and Figure 3In some embodiments, referring to the aforementioned computer device, the adaptive adjustment module and the reinforcement learning model together form a closed-loop adjustment structure, wherein the gap between the player's action and the reference action is used to adjust the tolerance coefficient parameter, thereby affecting the guiding reference action generated by the reinforcement learning model under soft boundary condition constraints. In some embodiments, referring to the aforementioned computer device, the adaptive adjustment module and the reinforcement learning model together provide a negative feedback mechanism. The input of the negative feedback mechanism is the gap between the player's action and the reference action, and the output of the negative feedback mechanism is the reference action. The negative feedback mechanism dynamically adjusts the tolerance coefficient parameter to achieve the following of the reference action with respect to the player's action. Thus, by dynamically adjusting the tolerance coefficient parameter to influence the decision-making process of the teaching model, the teaching model provides a reference action under the action of the dynamically adjusted tolerance coefficient parameter, and the gap between the player's action and the reference action is stabilized within a preset range. Such a negative feedback closed-loop control design, in order to... Figure 2 For example, by adjusting the tolerance coefficient parameter, the reference action output by the reinforcement learning model 220 can be influenced. Finally, a negative feedback loop is used to stabilize the difference between the player's action and the reference action within a preset range. This negative feedback mechanism enables the reference action to follow the player's action. Furthermore, unlike static adjustment methods that rely on fixed grading to set the experience and skill levels of the reference object corresponding to the coaching model, dynamically adjusting the tolerance coefficient parameter to change the soft boundary conditions of the reward function not only adheres to necessary constraints such as hard boundary conditions of the reward function but also avoids affecting the training parameters and process of the reinforcement learning model. Moreover, by influencing the decision-making process of the reinforcement learning model to control the reference action output, a stable reinforcement learning model, along with related decision-making models and agents, dynamically matches the player's current skill level with the output reference action and the teaching guidance based on the reference action, supporting real-time and personalized guidance, which is beneficial for improving training effectiveness and player experience.

[0061] See Figure 1 , Figure 2 and Figure 3In some embodiments, a computer-readable storage medium stores computer instructions. When the computer instructions are executed on a computer device, the computer device performs the following operations: obtaining an input state, a player action, and a reference action, wherein the reference action is obtained by inputting the input state and a tolerance coefficient parameter into a reinforcement learning model, the input state including at least a state associated with the player action, and the reference action used for instructional guidance; and selectively adjusting or maintaining the tolerance coefficient parameter, at least based on the gap between the player action and the reference action, through an adaptive adjustment module, thereby stabilizing the gap between the player action and the reference action within a preset range, wherein the tolerance coefficient parameter is a dynamically adjustable control quantity used to adjust the soft boundary conditions in the reward function of the reinforcement learning model, the reward function indicating the reward of the reinforcement learning model for taking a given action in a given state. In some embodiments, a computer-readable storage medium stores computer instructions. When the computer instructions are executed on a computer device, the computer device performs the following operations: obtaining an input state, a player action, and a reference action, wherein the reference action is obtained by inputting the input state and a tolerance coefficient parameter into a reinforcement learning model, the input state includes at least the state associated with the player action, and the reference action is used for instructional guidance; and through an adaptive adjustment module, selectively adjusting or maintaining the tolerance coefficient parameter based at least on the gap between the player action and the reference action, thereby stabilizing the gap between the player action and the reference action within a preset range, wherein the tolerance coefficient parameter is used to set the soft boundary conditions of the reward function of the reinforcement learning model, and the reward function is used to indicate the reward of the reinforcement learning model for taking a given action in a given state. In this way, by utilizing the dynamic adjustment mechanism of the tolerance coefficient parameter, the output reference action and teaching guidance based on the reference action dynamically match the player's current skill level, supporting the provision of real-time and personalized guidance, which is conducive to improving training effects and player experience, significantly reducing the system adaptation cost when switching between different users, and also supporting a personalized adjustment mechanism for the current user. Using the built-in algorithm of the pre-designed adaptive adjustment module, the tolerance coefficient parameter can be automatically adjusted or maintained, improving automation efficiency and facilitating large-scale promotion and application.

[0062] See Figure 1 , Figure 2 and Figure 3In some embodiments, referring to the aforementioned computer-readable storage medium, the adaptive adjustment module is used to adjust the tolerance coefficient parameter based on a preset multiple adjustment levels, and the gap between the player's action and the reference action includes the distribution of the absolute value of the action deviation of the player's action relative to the reference action. In some embodiments, referring to the aforementioned computer-readable storage medium, the adaptive adjustment module is used to adjust the tolerance coefficient parameter according to a multiple discrete levels, and the gap between the player's action and the reference action includes the distribution of the absolute value of the action deviation of the player's action relative to the reference action. Thus, the tolerance coefficient parameter can be preset with multiple levels, for example from level 1 to level 10, which facilitates subsequent strategy decisions and dynamic adjustment mechanisms. Recording the distribution of the absolute value of action deviation helps determine when to intervene and the intervention strategy. When the distribution of the absolute value of action deviation is relatively large in most cases, for example by comparing the mean of the absolute value of action deviation with a preset value, this means that the gap between the player's action and the reference action is large. This reflects a large gap between the player's current skill level and the skill level of the coach model corresponding to the reinforcement learning model, which is the reference object. Corresponding adjustment actions can be taken to dynamically adjust the tolerance coefficient parameter to match the player's current skill level. Conversely, when the distribution of the absolute value of the action deviation is relatively small in most cases, for example by comparing the mean of the absolute value of the action deviation with another preset value, it means that the difference between the player's action and the reference action is small. This reflects that the difference between the player's current skill level and the skill level of the coach model corresponding to the reinforcement learning model as the reference is small. The tolerance coefficient parameter can be dynamically adjusted to match the player's current skill level by taking corresponding adjustment actions.

[0063] See Figure 1 , Figure 2 and Figure 3In some embodiments, referring to the aforementioned computer-readable storage medium, the adaptive adjustment module and the reinforcement learning model together form a closed-loop adjustment structure, wherein the gap between the player's action and the reference action is used to adjust the tolerance coefficient parameter, thereby affecting the guiding reference action generated by the reinforcement learning model under soft boundary condition constraints. In some embodiments, referring to the aforementioned computer-readable storage medium, the adaptive adjustment module and the reinforcement learning model together provide a negative feedback mechanism. The input of the negative feedback mechanism is the gap between the player's action and the reference action, and the output of the negative feedback mechanism is the reference action. The negative feedback mechanism dynamically adjusts the tolerance coefficient parameter to achieve the following of the reference action with respect to the player's action. Thus, by dynamically adjusting the tolerance coefficient parameter to influence the decision-making process of the teaching model, the teaching model provides a reference action under the action of the dynamically adjusted tolerance coefficient parameter, and the gap between the player's action and the reference action is stabilized within a preset range. Such a negative feedback closed-loop control design, in order to... Figure 2 For example, by adjusting the tolerance coefficient parameter, the reference action output by the reinforcement learning model 220 can be influenced. Finally, a negative feedback loop is used to stabilize the difference between the player's action and the reference action within a preset range. This negative feedback mechanism enables the reference action to follow the player's action. Furthermore, unlike static adjustment methods that rely on fixed grading to set the experience and skill levels of the reference object corresponding to the coaching model, dynamically adjusting the tolerance coefficient parameter to change the soft boundary conditions of the reward function not only adheres to necessary constraints such as hard boundary conditions of the reward function but also avoids affecting the training parameters and process of the reinforcement learning model. Moreover, by influencing the decision-making process of the reinforcement learning model to control the reference action output, a stable reinforcement learning model, along with related decision-making models and agents, dynamically matches the player's current skill level with the output reference action and the teaching guidance based on the reference action, supporting real-time and personalized guidance, which is beneficial for improving training effectiveness and player experience.

[0064] Figure 4This is a schematic diagram of the structure of a computer device 400 provided in an embodiment of this application. The computer device 400 includes one or more processors 410, a communication interface 420, and a memory 430. The processors 410, the communication interface 420, and the memory 430 are interconnected via a bus 440. Optionally, the computer device 400 may further include an input / output interface 450, which is connected to input / output devices for receiving user-set parameters, etc. The computer device 400 can be used to implement some or all of the functions of the device embodiment or system embodiment in the above-described embodiments of this application; the processor 410 can also be used to implement some or all of the operation steps of the method embodiment in the above-described embodiments of this application. For example, the specific implementation of various operations performed by the computer device 400 can be referred to the specific details in the above embodiments, such as the processor 410 being used to execute some or all of the steps or operations in the above-described method embodiments. For example, in the embodiments of this application, the computer device 400 can be used to implement some or all of the functions of one or more components in the above-described device embodiments. In addition, the communication interface 420 can be used specifically for communication functions necessary to implement the functions of these devices and components, and the processor 410 can be used specifically for processing functions necessary to implement the functions of these devices and components.

[0065] It should be understood that, Figure 4 The computer device 400 may include one or more processors 410, and the multiple processors 410 may cooperate to provide processing power in a parallel connection mode, a serial connection mode, a serial-parallel connection mode, or an arbitrary connection mode; or the multiple processors 410 may form a processor sequence or a processor array; or the multiple processors 410 may be divided into a main processor and an auxiliary processor; or the multiple processors 410 may have different architectures, such as adopting a heterogeneous computing architecture. Furthermore, Figure 4 The structural and functional descriptions of the computer device 400 shown are exemplary and non-limiting. In some exemplary embodiments, the computer device 400 may include... Figure 4 The diagram shows more or fewer components, or combinations of some components, or splitting of some components, or different arrangements of components.

[0066] The processor 410 can have various specific implementations. For example, it can include one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or a data processing unit (DPU). This application embodiment does not impose specific limitations. The processor 410 can also be a single-core or multi-core processor. The processor 410 can be a combination of a CPU and hardware chips. The aforementioned hardware chips can be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLDs can be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof. The processor 410 can also be implemented using logic devices with built-in processing logic, such as FPGAs or digital signal processors (DSPs). The communication interface 420 can be a wired interface or a wireless interface, used to communicate with other modules or devices. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., and the wireless interface can be a cellular network interface or a wireless LAN interface, etc.

[0067] Memory 430 may be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 430 may also be volatile memory, which may be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM). The memory 430 can also be used to store program code and data, so that the processor 410 can call the program code stored in the memory 430 to execute some or all of the operation steps in the above method embodiments, or to execute the corresponding functions in the above device embodiments. Furthermore, the computer device 400 may include, compared to... Figure 4 The number of components displayed may be more or less, or there may be different component configurations.

[0068] Bus 440 can be a Peripheral Component Interconnect Express (PCIe) bus, or an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. Bus 440 can be divided into address bus, data bus, control bus, etc. In addition to the data bus, bus 440 can also include a power bus, control bus, and status signal bus. However, for clarity, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0069] The methods and devices provided in this application are based on the same inventive concept. Since the principles by which the methods and devices solve problems are similar, the embodiments, implementation methods, examples, or methods of implementation of the methods and devices can be referred to each other, and repeated details will not be repeated. This application also provides a system comprising multiple computer devices, the structure of each computer device of which can refer to the structure of the computer devices described above. The functions or operations achievable by this system can refer to the specific implementation steps in the above method embodiments and / or the specific functions described in the above device embodiments, and will not be repeated here.

[0070] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed on a computer device (such as one or more processors), they can implement the method steps described in the above method embodiments. The specific implementation of the above method steps by the processor of the computer-readable storage medium can refer to the specific operations described in the above method embodiments and / or the specific functions described in the above device embodiments, and will not be repeated here.

[0071] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. This application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Embodiments of this application can be implemented wholly or partially by software, hardware, firmware, or any other combination. When implemented in software, the above embodiments can be implemented wholly or partially as a computer program product. This application can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless network communication, microwave, etc.) means. Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that contains one or more sets of available media. Available media can be magnetic media (such as floppy disks, hard disks, and magnetic tapes), optical media, or semiconductor media. Semiconductor media can be solid-state drives, random access memory, flash memory, read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, or any other suitable form of storage medium.

[0072] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. Each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. The steps in the methods of the embodiments of this application can be adjusted in order, combined, or deleted according to actual needs; the modules in the systems of the embodiments of this application can be divided, combined, or deleted according to actual needs. If these modifications and variations of the embodiments of this application fall within the scope of the claims of this application and their equivalents, then this application also intends to include these modifications and variations.

Claims

1. An adaptive adjustment method based on reinforcement learning, characterized in that, The adaptive adjustment method includes: The system obtains an input state, a player action, and a reference action. The reference action is a guided action result obtained by inputting the input state and tolerance coefficient parameters into the reinforcement learning model under the soft boundary condition constraints of the reward function of the reinforcement learning model. The input state includes at least the state associated with the player action, and the reference action is used for teaching guidance. The adaptive adjustment module selectively adjusts or maintains the tolerance coefficient parameter, based at least on the gap between the player's action and the reference action, to change the decision constraints of the reinforcement learning model, thereby stabilizing the gap between the player's action and the reference action within a preset range. The tolerance coefficient parameter is a dynamically adjustable control quantity used to adjust the soft boundary conditions in the reward function of the reinforcement learning model, which indicates the reward of the reinforcement learning model for taking a given action in a given state.

2. The adaptive adjustment method according to claim 1, characterized in that, The adaptive adjustment module selectively adjusts or maintains the tolerance coefficient parameter, based at least on the difference between the player's action and the reference action, so that the difference between the player's action and the reference action is stabilized within a preset range, including: The behavioral deviation values ​​of the player's action relative to the reference action are evaluated in multiple dimensions, and then the combined behavioral deviation values ​​of the multiple dimensions are combined to obtain the comprehensive action deviation value of the player's action relative to the reference action. When the overall action deviation value is within the preset range, the tolerance coefficient parameter is maintained; When the overall action deviation value exceeds the preset range, the tolerance coefficient parameter is adjusted.

3. The adaptive adjustment method according to claim 1, characterized in that, The adaptive adjustment module is used to adjust the tolerance coefficient parameter based on multiple preset adjustment levels, and the gap between the player action and the reference action includes the distribution of the absolute value of the action deviation of the player action relative to the reference action.

4. The adaptive adjustment method according to claim 3, characterized in that, The multiple discrete gears correspond sequentially from the lowest to the highest gear to multiple level references from the highest to the lowest level, and the adaptive adjustment module is used for: When the mean absolute value of the action deviation is greater than a preset high threshold, the tolerance coefficient parameter is increased; When the mean of the absolute value of the action deviation is less than a preset low threshold, the tolerance coefficient parameter is reduced, wherein the preset high threshold and the preset low threshold define the preset range.

5. The adaptive adjustment method according to claim 1, characterized in that, The reinforcement learning model is used to provide an AI policy with adjustable aggressiveness, and the tolerance coefficient parameter is used to dynamically adjust the adjustable aggressiveness of the AI ​​policy to match the skill level associated with the player's actions.

6. The adaptive adjustment method according to claim 1, characterized in that, The adaptive adjustment module is also used to determine the initial value of the tolerance coefficient parameter based on historical data associated with the player's actions.

7. The adaptive adjustment method according to claim 6, characterized in that, The adaptive adjustment module is further configured to selectively adjust the tolerance coefficient parameter or maintain the tolerance coefficient parameter based on the gap between the player's action and the reference action, as well as the result indicators related to the task completion status.

8. The adaptive adjustment method according to claim 7, characterized in that, When the task completion status does not meet the preset conditions and the gap between the player's action and the reference action exceeds the preset range, the adaptive adjustment module adjusts the tolerance coefficient parameter based on the action deviation statistics to change the policy constraint strength of the reinforcement learning model.

9. The adaptive adjustment method according to claim 1, characterized in that, The adaptive adjustment module and the reinforcement learning model together form a closed-loop adjustment structure, wherein the difference between the player action and the reference action is used to adjust the tolerance coefficient parameter, thereby affecting the guiding reference action generated by the reinforcement learning model under soft boundary condition constraints.

10. The adaptive adjustment method according to claim 1, characterized in that, The adaptive module is also used to adjust the tolerance coefficient parameter based on the player's customized settings.

11. The adaptive adjustment method according to claim 1, characterized in that, The method can be applied to teaching and guidance in simulated car driving scenarios. The input states are vehicle state and track state, the player action is the player's car driving action, the reference action is the reference car driving action, and the soft boundary conditions include at least the upper limit of safe cornering speed, the upper limit of vehicle lateral acceleration, and the soft boundary of track width.

12. The adaptive adjustment method according to claim 1, characterized in that, The method can be applied to teaching and guidance in simulated aircraft driving scenarios. The input states are the aircraft state and the aircraft flight path state, the player actions are the player's aircraft driving actions, the reference actions are the reference aircraft driving actions, and the soft boundary conditions include aircraft speed, aircraft turning speed, and aircraft acceleration.

13. The adaptive adjustment method according to claim 1, characterized in that, The method can be applied to teaching guidance in simulated sports training scenarios. The input states are player height, player arm length, and player posture. The player action is the player's sports action. The reference action is the reference sports action. The soft boundary conditions include at least the sports action force, sports action speed, and sports action acceleration.

14. The adaptive adjustment method according to claim 1, characterized in that, The hard boundary conditions of the reward function are not adjustable, and the hard boundary conditions of the reward function are preset based on the boundary constraints of the scene associated with the player's action. When the scene associated with the player's action is a simulated car driving scene, the hard boundary conditions of the reward function include the hard boundary of the track width.

15. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The operations performed by the processor when executing the computer program include: The system obtains an input state, a player action, and a reference action, wherein the reference action is obtained by inputting the input state and a tolerance coefficient parameter into a reinforcement learning model, the input state includes at least the state associated with the player action, and the reference action is used for instructional guidance. Through the adaptive adjustment module, the tolerance coefficient parameter is selectively adjusted or maintained based at least on the gap between the player action and the reference action, so that the gap between the player action and the reference action is stabilized within a preset range. The tolerance coefficient parameter is a dynamically adjustable control quantity used to adjust the soft boundary conditions in the reward function of the reinforcement learning model. The reward function is used to indicate the reward of the reinforcement learning model for taking a given action in a given state.

16. The computer device according to claim 15, characterized in that, The adaptive adjustment module is used to adjust the tolerance coefficient parameter based on multiple preset adjustment levels, and the gap between the player action and the reference action includes the distribution of the absolute value of the action deviation of the player action relative to the reference action.

17. The computer device according to claim 15, characterized in that, The adaptive adjustment module and the reinforcement learning model together form a closed-loop adjustment structure, wherein the difference between the player action and the reference action is used to adjust the tolerance coefficient parameter, thereby affecting the guiding reference action generated by the reinforcement learning model under soft boundary condition constraints.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer device, cause the computer device to perform the following operations: The system obtains an input state, a player action, and a reference action, wherein the reference action is obtained by inputting the input state and a tolerance coefficient parameter into a reinforcement learning model, the input state includes at least the state associated with the player action, and the reference action is used for instructional guidance. Through the adaptive adjustment module, the tolerance coefficient parameter is selectively adjusted or maintained based at least on the gap between the player action and the reference action, so that the gap between the player action and the reference action is stabilized within a preset range. The tolerance coefficient parameter is a dynamically adjustable control quantity used to adjust the soft boundary conditions in the reward function of the reinforcement learning model. The reward function is used to indicate the reward of the reinforcement learning model for taking a given action in a given state.

19. The computer-readable storage medium according to claim 18, characterized in that, The adaptive adjustment module is used to adjust the tolerance coefficient parameter based on multiple preset adjustment levels, and the gap between the player action and the reference action includes the distribution of the absolute value of the action deviation of the player action relative to the reference action.

20. The computer-readable storage medium according to claim 18, characterized in that, The adaptive adjustment module and the reinforcement learning model together form a closed-loop adjustment structure, wherein the difference between the player action and the reference action is used to adjust the tolerance coefficient parameter, thereby affecting the guiding reference action generated by the reinforcement learning model under soft boundary condition constraints.

Images