An AI large model reinforcement learning hyperparameter dynamic adjustment method and device

By adaptively adjusting the hyperparameters of large AI models, the imbalance between exploration and utilization caused by static hyperparameters is resolved, improving the model's learning efficiency and stability, and achieving better training results.

CN120874952BActive Publication Date: 2026-11-10BEISEN CLOUD COMPUTING CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202511007489.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-07-22
Publication Date
2026-11-10
Estimated Expiration
2045-07-22

AI Technical Summary

Technical Problem

The existing large AI models have statically fixed hyperparameters, which leads to an imbalance between exploration and exploitation, unstable policy updates, poor adaptability to environmental changes, lack of flexibility, and affects training results.

Method used

By adaptively adjusting hyperparameters during training, calculating a comprehensive performance score using the difference in performance evaluation values, dynamically adjusting the pruning parameter ε and KL divergence coefficient β, and balancing the weights of reward and loss, adaptive parameter adjustment is achieved.

Benefits of technology

It improves the learning efficiency, stability, and final performance of large AI models. The dynamic adjustment mechanism reduces human intervention and enhances the convergence and generalization capabilities of the models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120874952B_ABST
    Figure CN120874952B_ABST
Patent Text Reader

Abstract

The application provides a method and device for dynamically adjusting hyperparameters of AI large models through reinforcement learning, and relates to the technical field of artificial intelligence, comprising: obtaining a first performance evaluation value of an AI large model in a first time window and a second performance evaluation value of the AI large model in a second time window during training; determining a performance change amount of the AI large model from the first time window to the second time window according to a difference between the first performance evaluation value and the second performance evaluation value; calculating a comprehensive performance score of the AI large model according to the performance change amount, the first performance evaluation value and the second performance evaluation value; determining an adjustment direction of hyperparameters of the AI large model according to the comprehensive performance score and a configured performance threshold; and adjusting the hyperparameters of the AI large model according to a configured adjustment factor and the adjustment direction. The application can ensure stable and efficient learning during the training process and improve the performance of the AI large model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for dynamically adjusting hyperparameters in large AI model reinforcement learning. Background Technology

[0002] In recent years, large-scale language models have made significant progress in understanding and generating natural language, demonstrating powerful capabilities in numerous natural language processing tasks. The DeepSeek AI team proposed the Group Relative Policy Optimization (GRPO) algorithm. GRPO, a policy gradient-based reinforcement learning algorithm, is a variant of Proximal Policy Optimization (PPO). GRPO's performance is highly dependent on its hyperparameter selection, exhibiting significant sensitivity to their values. Two main hyperparameters are the clipping range hyperparameter ε and the KL divergence penalty coefficient β, which are crucial to GRPO's training effectiveness. Currently, the hyperparameter values ​​are statically fixed and unchanging. Unchanging parameter values ​​cannot remain optimal throughout the training process, leading to imbalances in exploration and exploitation, unstable policy updates, poor adaptability to environmental changes, and a lack of necessary flexibility. Because of the model's learning state and data distribution, the training effect changes as training progresses. Summary of the Invention

[0003] The purpose of this application is to provide a method and apparatus for dynamically adjusting hyperparameters in reinforcement learning of large AI models, which solves the above-mentioned problems existing in the prior art. It can adaptively adjust the hyperparameters of large AI models during training to effectively improve the performance of large AI models.

[0004] Firstly, a method for dynamically adjusting hyperparameters in large AI model reinforcement learning is provided, which may include: The first performance evaluation value of the AI ​​large model under a first time window and the second performance evaluation value under a second time window are obtained during the training process; wherein, any time window includes multiple training rounds of the training process; the performance evaluation value of the AI ​​large model under any time window is determined based on the performance evaluation values ​​of the AI ​​large model under different training rounds within the corresponding time window; the second time window is later than the first time window; Based on the difference between the first performance evaluation value and the second performance evaluation value, the performance change of the AI ​​large model from the first time window to the second time window is determined; The comprehensive performance score of the AI ​​large model is calculated based on the performance change, the first performance evaluation value, and the second performance evaluation value. Based on the comprehensive performance score and the configured performance threshold, the adjustment direction of the hyperparameters of the AI ​​large model is determined; The hyperparameters of the large AI model are adjusted according to the configured adjustment factor and the adjustment direction.

[0005] In an optional implementation, the performance evaluation values ​​include reward values ​​and loss values; The method for obtaining the reward value includes: For any given time window, obtain multiple candidate outputs of the AI ​​model under different training rounds within that time window; For any training epoch, obtain the reward scores for different candidate outputs in that training epoch; Based on the reward scores corresponding to different candidate outputs, the relative reward value of the AI ​​large model in the training round is determined; The reward value of the AI ​​model under the stated time window is obtained based on the relative reward value of the AI ​​model in each training round.

[0006] In an optional implementation, determining the relative reward value of the AI ​​large model in the training round based on the reward scores corresponding to different candidate outputs includes: The average of the reward scores corresponding to each candidate output is taken as the average reward score of the AI ​​model in the training round. The standard deviation of the reward score corresponding to each candidate output is used as the reward standard deviation of the AI ​​large model in the training round. For any candidate output, the difference between the reward score corresponding to the candidate output and the average reward score is taken as the target difference. The ratio of the target difference to the reward standard deviation is used as the relative reward value of the candidate output; Based on the relative reward values ​​of each candidate output in the training round, the relative reward value of the AI ​​large model in the training round is determined.

[0007] In an optional implementation, the performance variation includes: a change in reward and a change in loss; Based on the difference between the first performance evaluation value and the second performance evaluation value, the performance change of the AI ​​large model from the first time window to the second time window is determined, including: The difference between the second reward value and the first reward value is used as the change in reward. The difference between the second loss value and the first loss value is taken as the change in loss. Based on the absolute values ​​of the changes in reward and loss, the performance change of the AI ​​model from the first time window to the second time window is determined.

[0008] In an optional implementation, the comprehensive performance score of the large AI model is calculated based on the performance change, the first performance evaluation value, and the second performance evaluation value, including: Calculate the weight of the reward change based on the reward change and the performance change; Calculate the weight of the change in loss based on the change in loss and the change in performance; The change in reward is weighted using the weights of the change in reward to obtain the weighted change in reward. The change in loss is weighted using the weights of the change in loss to obtain the weighted change in loss; The difference between the weighted change in reward and the weighted change in loss is used as the overall performance score of the AI ​​model.

[0009] In an optional implementation, the hyperparameters include: a clipping parameter and a KL divergence coefficient; The adjustment directions include: increasing and decreasing; Based on the comprehensive performance score and the configured performance threshold, the adjustment direction of the hyperparameters of the large AI model is determined, including: If the overall performance score is less than the performance threshold, the adjustment direction of the trimming parameter is determined to be increased, and the adjustment direction of the KL divergence coefficient is determined to be decreased. If the overall performance score is not less than the performance threshold, then the adjustment direction of the trimming parameter is determined to be decreasing, and the adjustment direction of the KL divergence coefficient is determined to be increasing.

[0010] In an optional implementation, the adjustment factor includes: a growth coefficient and a decay ratio coefficient; The hyperparameters of the large AI model are adjusted according to the configured adjustment factor and the adjustment direction, including: When the overall performance score is less than the performance threshold, the amount of increase in the trimming parameter is determined according to the growth coefficient; and the amount of decrease in the KL divergence coefficient is determined according to the decay ratio coefficient. When the overall performance score is not less than the performance threshold, the amount of reduction of the trimming parameter is determined according to the attenuation ratio coefficient; and the amount of increase of the KL divergence coefficient is determined according to the growth coefficient.

[0011] Secondly, a hyperparameter dynamic adjustment device for large AI models is provided, which may include: An acquisition unit is used to acquire a first performance evaluation value of the AI ​​large model under a first time window and a second performance evaluation value under a second time window during the training process; wherein, any time window includes multiple training rounds of the training process; the performance evaluation value of the AI ​​large model under any time window is determined based on the performance evaluation values ​​of the AI ​​large model under different training rounds within the corresponding time window; the second time window is later than the first time window; The first determining unit is used to determine the performance change of the AI ​​large model from the first time window to the second time window based on the difference between the first performance evaluation value and the second performance evaluation value. The calculation unit is used to calculate the comprehensive performance score of the AI ​​large model based on the performance change, the first performance evaluation value, and the second performance evaluation value. The second determining unit is used to determine the adjustment direction of the hyperparameters of the AI ​​large model based on the comprehensive performance score and the configured performance threshold. The adjustment unit is used to adjust the hyperparameters of the AI ​​large model according to the configured adjustment factor and the adjustment direction.

[0012] Thirdly, an electronic device is provided, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0013] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the methods described in the first aspect above.

[0014] This application provides a novel adaptive parameter tuning mechanism for dynamic hyperparameter tuning during reinforcement learning training. It adaptively adjusts the exploration level ε (epsilon) and the coefficient β (β_KL) of the KL divergence penalty term, while dynamically balancing the weights (alpha and beta) of reward enhancement and validation loss reduction. This effectively improves the convergence and generalization ability of large AI models. Furthermore, the dynamic tuning strategy can automate the existing manual tuning process of reinforcement learning algorithms to some extent, reducing the need for manual intervention.

[0015] This application dynamically adjusts the pruning parameters to control the magnitude of AI large model updates; dynamically adjusting the KL divergence coefficient can help the AI ​​large model remain stable during the training phase; the dynamic adjustment mechanism of this application allows the AI ​​large model to self-adjust based on the learning progress and effects observed on a specific task, enabling the AI ​​large model algorithm to more intelligently adapt to changes in the training process, thereby improving learning efficiency, stability and final performance.

[0016] This application uses reward and loss values ​​to measure the performance of large AI models. It dynamically balances the influence of reward and loss values ​​through weighted ratios (alpha and beta), rather than solely relying on reward values. A sliding time window is used to calculate reward and loss values, avoiding the lag of global averaging and enabling faster response to recent performance changes. The adjustment direction of the hyperparameters of the large AI model is determined based on the comprehensive performance score, with the two hyperparameters adjusted in opposite directions (one increasing, one decreasing), forming a balance mechanism between exploration and constraints.

[0017] This application achieves adaptive control of pruning parameters and KL divergence coefficients through sliding window statistics, dual-objective weight allocation, and parameter linkage adjustment. It dynamically combines traditionally independently adjustable parameters through performance feedback, improving training stability while maintaining lightweight design. This further enhances the performance and generalization ability of large AI models, and can make a small contribution to the development of reinforcement learning in the field of artificial intelligence. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a method for dynamically adjusting hyperparameters in large AI model reinforcement learning, provided in an embodiment of this application; Figure 2 A schematic flowchart illustrating the process of determining the relative advantage value of each candidate output, provided for an embodiment of this application; Figure 3 A schematic diagram illustrating the conversion between reward points and relative advantages, provided for an embodiment of this application; Figure 4 A flowchart illustrating another method for dynamically adjusting hyperparameters in reinforcement learning of large AI models provided in this application embodiment; Figure 5 A flowchart illustrating an AI large model update method provided in an embodiment of this application; Figure 6 A schematic diagram of a device for dynamically adjusting hyperparameters in large AI model reinforcement learning, provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by those skilled in the art. The words "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are only used to distinguish different components. The words "comprising" or "including," etc., mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, but do not exclude other elements or objects. The words "connected," "coupled," or "connected," etc., are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," "right," etc., are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0021] For ease of understanding, the terms used in the embodiments of this application are explained below: GRPO: Group-based Relative Policy Optimization: By constructing groups from the model output and calculating the relative rewards within the groups to estimate the baseline, it avoids the need for a comment model of the same size as the policy model in traditional policy optimization algorithms, reducing the computational cost of reinforcement learning training and ensuring that the model can learn effective policies.

[0022] The method for dynamically adjusting hyperparameters of AI large-scale reinforcement learning provided in this application embodiment is applied in the training process of AI large-scale models; the AI ​​large-scale model is a policy model, i.e., a large-scale model LLM, which is used to output the probability distribution of word units based on the questions raised by users; the AI ​​large-scale model needs to be pre-trained or supervised fine-tuned before actual application.

[0023] The method for dynamically adjusting hyperparameters of large AI model reinforcement learning provided in this application can be applied to servers. The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0024] Figure 1 This is a flowchart illustrating a method for dynamically adjusting hyperparameters in large AI model reinforcement learning, provided as an embodiment of this application. Figure 1 As shown, the method may include: Step S110: Obtain the first performance evaluation value of the AI ​​large model under the first time window and the second performance evaluation value under the second time window during the training process.

[0025] The training process of the AI ​​large model includes multiple training rounds; each training round corresponds to a performance evaluation value; the performance evaluation values ​​of each training round of the AI ​​large model are stored in the form of a queue data structure to obtain a performance evaluation dataset; the performance evaluation dataset only retains the most recent data; when the number of performance evaluation values ​​in the performance evaluation dataset reaches the user-configured sliding window size, the performance evaluation values ​​in the performance evaluation dataset are divided into two parts according to the order in which they entered the performance evaluation dataset; the second part is the second performance evaluation value of the AI ​​large model in the current time window (i.e., the second time window); the second performance evaluation value is used to represent the performance evaluation result of the current AI large model; the first part is the first performance evaluation value of the AI ​​large model in the previous time window (i.e., the first time window); the first performance evaluation value is used to represent the performance evaluation result of the current AI large model in the previous time window (i.e., the previous version of the AI ​​large model); the number of the first performance evaluation value and the number of the second performance evaluation value are the same; when the number of performance evaluation values ​​in the performance evaluation dataset does not reach the user-configured sliding window size, the hyperparameters of the AI ​​large model are not adjusted. Specifically, the performance evaluation values ​​include reward values ​​and loss values; the first performance evaluation value includes: the first reward value and the first loss value; the second performance evaluation value includes: the second reward value and the second loss value.

[0026] For example, if the user-configured sliding window size is 100, then when there are 100 performance evaluation values ​​in the performance evaluation dataset, the performance evaluation dataset is divided into two parts. The first 50 performance evaluation values ​​in the performance evaluation dataset are taken as the first performance evaluation value under the first time window; the last 50 performance evaluation values ​​in the performance evaluation dataset are taken as the second performance evaluation value under the second time window. In practice, the methods for obtaining reward values ​​include: (1) For any given time window, obtain multiple candidate outputs of the AI ​​large model under different training rounds within the time window; In practical applications and at any training epoch, the large AI model generates multiple candidate outputs based on the user's input and the configured response strategy. , , ,……, The number of candidate outputs generated by the AI ​​large model for a given input is a group hyperparameter that can affect training performance and stability. A larger group size can better reduce variance but increases computational cost; preferably, the number of candidate outputs can be 64. Specifically, the response strategy includes parameters such as cue word templates, temperature, whether to enable sampling strategies, and maximum text length. The cue word template is used to instruct the AI ​​large model to follow specified instructions, generate an inference process from the input content to the candidate output, and finally output the candidate output. The cue word template can be: "This is a dialogue between the user and the AI ​​large model. The user asks a question, and the AI ​​large model solves the problem. The AI ​​large model first thinks about the reasoning process in its mind, and then provides the answer to the user. The reasoning process and the answer are respectively contained in..." <reasoning>< / reasoning> and <answer>< / answer> In the tag, that is <reasoning> reasoninghere< / reasoning> <answer> Answer here< / answer> Temperature is a parameter used to adjust the sampling probability distribution, and its value typically ranges from 0 to positive infinity. When the temperature is less than 1, the probability distribution becomes steeper, and the AI ​​model tends to select higher-probability markers, resulting in more stable and deterministic text. When the temperature is greater than 1, the probability distribution becomes flatter, increasing the likelihood that the AI ​​model will select lower-probability markers, resulting in more random and creative text. Preferably, a temperature of 0.9 makes the text generated by the AI ​​model relatively stable, but still has a certain degree of randomness. When the sampling strategy is "true," the AI ​​model generates text by randomly sampling according to the probability distribution of the markers. When the sampling strategy is "false," the AI ​​model uses a greedy search, that is, it selects the marker with the highest probability at each step without random sampling.

[0027] (2) For any training round, obtain the reward score for different candidate outputs in the training round; In this system, the reward scores for different candidate outputs in any training epoch are obtained based on a configured reward function. This reward function includes three types: based on reward rules, based on reward models, or based on human feedback evaluation, determined according to the target application scenario of the large AI model. Figure 2 As shown, the reward function assigns a reward score to each candidate output based on its quality. , , }, where G represents the number of candidate outputs; the first i candidate outputs Reward points The following formula is given: = R( , q), where R represents the reward function; q represents the input content corresponding to the candidate output; In some embodiments of this application, the reward serves as the source of the training signal, determining the optimization direction of reinforcement learning. To make training more effective, rule-based rewards can be employed. For example, in mathematical reasoning, the reward for a correct answer might be 1, and the reward for an incorrect answer might be 0. For code generation, the reward can be based on whether the code compiles successfully and passes test cases. For general language tasks, the reward can be based on metrics such as fluency, coherence, and relevance. In some other embodiments of this application, the reward function mainly includes three types: accuracy reward function, format reward function, and language consistency reward function. The accuracy reward function (Accuracy Rewards) is used to evaluate whether the answer is correct; that is, it compares the answer predicted by the AI ​​model with the actual situation or the standard answer. If they match, a higher reward score is given; otherwise, a lower reward score is given. For example, for mathematical problems with a definite result, the AI ​​model needs to provide the final answer in a specified format to achieve reliable rule-based correctness verification. The format reward function (Format Rewards) is used to place the model's response thought process within a specific context. <reasoning>< / reasoning> In the tags, the answer is placed <answer>< / answer> The tag template constrains the response content to a specified structure and format, ensuring accurate parsing of the thought process and results and awarding bonus points. Higher bonus points are awarded for compliance with the rules, and lower bonus points for non-compliance. The language consistency bonus function encourages the model to use a single target language for reasoning responses as much as possible, avoiding language mixing and ensuring consistent language style in the model's output. Regular expressions are typically used to determine if the output falls within the Chinese encoding range. Under the bonus function, the AI ​​large-scale model will meticulously evaluate the quality of its reasoning chain and guide the model to generate more reasonable and accurate reasoning processes through the bonus mechanism, allowing the AI ​​large-scale model to naturally explore, verify, backtrack, reflect, and summarize behavioral paradigms. In practical applications, users can use different types of bonus functions according to actual needs or target application scenarios, making it adaptable to a wide range of tasks and evaluation criteria.

[0028] (3) Determine the relative reward value of the AI ​​large model in each training round based on the reward scores corresponding to different candidate outputs; specifically, this includes: taking the average of the reward scores corresponding to each candidate output as the average reward score of the AI ​​large model in that training round; wherein, the average reward score The calculation formula is: The standard deviation of the reward scores corresponding to each candidate output is used as the reward standard deviation of the AI ​​model in this training round; where, the reward standard deviation... The calculation formula is: For any candidate output, the difference between the reward score corresponding to that candidate output and the average reward score is taken as the target difference; where the target difference characterizes the degree of deviation of each candidate output from the average level; the ratio of the target difference to the reward standard deviation is taken as the relative reward value of the candidate output; where the relative reward value of the i-th candidate output is... The calculation formula is as follows: Based on the relative reward values ​​of each candidate output in each training round, the relative reward value of the AI ​​large model in each training round is determined; wherein, the average of the relative reward values ​​of all candidate outputs in any training round is taken as the relative reward value of the AI ​​large model in that training round.

[0029] In practice, determining the relative reward value of the AI ​​model in each training epoch based on the reward scores corresponding to different candidate outputs is to normalize the reward scores, making the relative reward values ​​of different training epochs comparable, thereby achieving reward scaling. For example, if the reward score of a candidate output is much higher than the average reward score, then the relative reward value of that candidate output is a large positive number, and the AI ​​model will tend to strengthen the strategy of generating this output; conversely, if the reward score of a candidate output is much lower than the average reward score, then the relative reward value of that candidate output is a small negative number, and the AI ​​model will suppress the strategy of generating this output. As shown in Table 1, assuming the reward scores of different candidate outputs are as follows: Table 1

[0030] like Figure 3 As shown, reward scores of 7, 9, and 10 are higher than the average reward score of 6.6, and their relative reward values ​​are all positive. Instead of directly evaluating the absolute quality of a candidate output, it compares its performance with that of other candidate outputs in the same training round to obtain a relative reward value (i.e., ...). Figure 3The advantage of a candidate output is greater than 0 if its reward score is higher than the average reward score of the group, indicating that the candidate output is relatively good; conversely, if its reward score is lower than the average reward score, its advantage is less than 0, indicating that the candidate output is relatively poor. Ultimately, outputs with reward scores higher than the average reward score are encouraged, while outputs with reward scores lower than the average reward score are suppressed. This calculation yields a stable and discriminative reward signal to guide the updating of large AI models. It is worth noting that GRPO effectively avoids dependence on the value function (Critic model) in traditional reinforcement learning by calculating relative rewards within groups, thereby reducing memory consumption and computational burden by approximately 50%.

[0031] (4) Based on the relative reward value of the AI ​​model in each training round, obtain the reward value of the AI ​​model in the time window; wherein, the average of the relative reward values ​​in all training rounds of the time window is taken as the reward value of the AI ​​model in the time window.

[0032] In specific implementation, the methods for obtaining the loss value include: After each training epoch, the AI ​​model is evaluated using a validation set. For any given training epoch, the AI ​​model that has completed the training for that epoch generates output from the input content in the validation set. The difference between the generated output and the true label corresponding to the input content in the validation set is calculated using a loss function to obtain one or more loss values. The average of the multiple loss values ​​is calculated as the loss value for that training epoch.

[0033] Step S220: Determine the performance change of the AI ​​large model from the first time window to the second time window based on the difference between the first performance evaluation value and the second performance evaluation value.

[0034] Among them, performance change includes reward change and loss change; performance change is used to reflect the drasticness of the change.

[0035] Specifically, based on the difference between the first and second performance evaluation values, the performance change of the large AI model from the first time window to the second time window is determined, including: The difference between the second reward value and the first reward value is taken as the reward change; where the reward change... The calculation formula is: N represents the sum of the window sizes of the first and second time windows; the sum of the window sizes of the first and second time windows is the number of training rounds contained in the first and second time windows, and also the number of the first reward value, the second reward value, the first loss value, and the second loss value. This represents the relative reward value of the large AI model in each training round within the second time window; This represents the relative reward value of the AI ​​large model in each training round within the first time window; This represents the second reward value of the large AI model within the second time window; This represents the first reward value of the large AI model within the first time window; if A value greater than 0 indicates that the reward is increasing, the reward evaluation is effective, and the performance of the large AI model is better. The difference between the second loss value and the first loss value is taken as the change in loss; where, the change in loss The calculation formula is: ; This represents the loss value of the large AI model in each training round within the second time window; This represents the loss value of the large AI model in each training round within the first time window; This represents the second loss value of the large AI model within the second time window; This represents the first loss value of the large AI model within the first time window; if A value less than 0 indicates that the validation loss is decreasing, generalization is successful, and the performance of the large AI model is improving. Based on the absolute values ​​of the changes in reward and loss, the performance change of the large AI model from the first time window to the second time window is determined; among them, the performance change... The calculation formula is as follows: ; This represents the absolute value of the change in reward. This represents the absolute value of the change in loss; determining the performance change based on the absolute values ​​of the change in reward and the change in loss is to simultaneously determine whether a change has occurred and the magnitude of the change. Represents a constant; It's a very small number, plus To avoid the error of dividing by 0.

[0036] Step S230: Calculate the comprehensive performance score of the AI ​​large model based on the performance change, the first performance evaluation value, and the second performance evaluation value.

[0037] Specifically, based on the performance change, the first performance evaluation value, and the second performance evaluation value, the comprehensive performance score of the large AI model is calculated, including: Calculate the weight of the reward change based on the reward change and the performance change; where the weight of the reward change... The calculation formula is: ,in, The function is used to limit alpha to between the minimum and maximum values ​​of alpha_bounds, ensuring that alpha fluctuates within a reasonable range and preventing it from getting out of control; This indicates the upper and lower bounds of the value range of alpha, which can be between 0.1 and 0.5. The upper and lower bounds of the alpha value range are configured by the user according to actual needs. Multiplying by... The goal is to set a target value for the total weight so that alpha + beta ≈ 2.0, making the adjustment process more sensitive and responsive, while keeping the sum of alpha and beta controllable. The weight of the change in loss is calculated based on the change in loss and the change in performance; among which, the weight of the change in reward is... The calculation formula is: ,in, The function is used to... Limited to Between the minimum and maximum values, ensure that beta fluctuates within a reasonable range to prevent it from getting out of control; express The upper and lower bounds of the range of values, The upper and lower bounds of its value range can be 0.1-0.5; The upper and lower bounds of the value range are configured by the user according to actual needs; The change in reward is weighted by the weight of the change in reward to obtain the weighted change in reward; the change in loss is weighted by the weight of the change in loss to obtain the weighted change in loss. The difference between the weighted change in reward and the weighted change in loss is used as the overall performance score of the AI ​​model; the formula for calculating the overall performance score perf_score is as follows: perf_score = The comprehensive performance score takes into account both the change in reward and the change in loss, as well as their respective weights. The two weights, alpha (α) and beta (β), indicate the relative importance of evaluating reward and verifying loss in performance evaluation. Specifically, by dividing the absolute values ​​of the change in reward and the absolute values ​​of the change in loss by the change in performance, it is determined which of the current reward and loss fluctuates more, thus obtaining their weights. The factor with the greater change in alpha and beta has a larger weight, which means that it needs more attention.

[0038] Step S240: Determine the adjustment direction of the hyperparameters of the AI ​​large model based on the comprehensive performance score and the configured performance threshold; adjust the hyperparameters of the AI ​​large model according to the configured adjustment factors and adjustment direction.

[0039] The hyperparameters include: the clipping parameter ε (Epsilon) and the KL divergence coefficient β (β... KL The pruning parameter ε limits the update rate of the large AI model. The value of ε determines the maximum change the large AI model can make in each update, typically between 0.1 and 0.3. A smaller ε value makes the large AI model update more conservative and the training process more stable, but it leads to slower convergence because the large AI model can only make small adjustments each time. Conversely, a larger ε value allows for more aggressive updates, speeding up convergence, but it also increases the risk of training instability, as the large AI model may make large jumps in a single update, leading to performance degradation. Therefore, choosing an appropriate ε value requires a trade-off between training stability and convergence speed. The KL divergence coefficient is the KL... The coefficient of the divergence penalty term controls the weight of the KL divergence penalty term in the total loss function. The value of β determines the degree of deviation of the new AI model from the reference AI model. A larger β value makes the AI ​​model update closer to the reference policy, and the AI ​​model can only make small adjustments each time, thus improving training stability, but also limiting the AI ​​model's exploration ability. Conversely, a smaller β value allows the new AI model to deviate more freely from the reference policy, thus discovering a better AI model, but also increases the risk of training instability. Therefore, the choice of β also needs to be balanced between AI model exploration and training stability, and usually needs to be adjusted according to the specific task and characteristics of the AI ​​model. Users can configure the minimum and maximum values ​​of the pruning parameters and the KL divergence coefficient according to actual needs, which are used to limit the exploration range of ε and the range of the KL divergence coefficient, respectively. Preferably, the minimum and maximum values ​​of the pruning parameters can be 0.05 and 0.2, and the minimum and maximum values ​​of the KL divergence coefficient can be 0.001 and 0.05, respectively. At the same time, β is pre-configured. KL The base value can be 0.01; adjustment directions include increasing and decreasing. Adjustment factors are used to adjust the clipping parameters and KL divergence coefficient, including growth coefficients and decay ratio coefficients, controlling the adjustment magnitude. If the growth coefficient is greater than 1, increase ε or β. KL If the attenuation ratio is less than 1, decrease ε or β. KL For example, 1.05 represents an increase of 5%, while 0.95 represents a decrease of 5%.

[0040] Specifically, based on the comprehensive performance score and the configured performance thresholds, the direction for adjusting the hyperparameters of the large AI model is determined, including: If the overall performance score is less than the performance threshold, the adjustment direction of the trimming parameter will be determined to increase, and the adjustment direction of the KL divergence coefficient will be determined to decrease. If the overall performance score is not less than the performance threshold, then the adjustment direction of the trimming parameter will be determined to decrease, and the adjustment direction of the KL divergence coefficient will be determined to increase.

[0041] If the overall performance score is not less than the performance threshold, it indicates that the generalization of the AI ​​model is good and the performance has improved, so "tighten ε / increase regularization" can be used; if the overall performance score is less than the performance threshold, it indicates that the AI ​​model is not good and the performance has decreased, so "increase exploration / relax constraints" should be used, as shown in Table 2.

[0042] Table 2

[0043] In Table 2, Scenario A is the most ideal (reward increases, loss decreases), showing significant improvement in the AI ​​model. The change in reward increases while the change in loss decreases, resulting in a high perf_score and excellent performance. Scenario B shows progress at a cost (reward increases, loss increases). Although the reward increases, the loss also worsens, indicating that the AI ​​model may be overfitting or sacrificing generalization ability, resulting in only a slight performance improvement. Scenario C is structural annealing (reward decreases, loss decreases). Although the change in reward decreases, the decrease in loss is more significant, resulting in a positive overall score. The AI ​​model may be in a short-term "robust pullback" phase. Scenario D is the least ideal (reward decreases, loss increases), with both reward and loss changes deteriorating simultaneously. The AI ​​model's performance significantly deteriorates, and the scheduler should increase exploration and reduce KL. Limitations: Scenario E increases the weight of rewards and tolerates mild losses. At this time, more attention is paid to rewards (such as in the early stage). Even if the loss increases slightly, the overall benefits are considered to outweigh the costs. Scenario F increases the weight of losses and strictly limits overfitting. Although the large AI model increases the amount of reward change, it is very sensitive to generalization (high β). Even a slight increase in loss is strongly penalized, and the overall model is considered to be degenerate.

[0044] Specifically, such as Figure 4 As shown, the hyperparameters of the large AI model are adjusted according to the configured adjustment factors and adjustment directions, including: When the overall performance score is less than the performance threshold, the amount of increase in the pruning parameter is determined based on the growth coefficient; the amount of decrease in the KL divergence coefficient is determined based on the decay ratio coefficient. The performance threshold is used to measure the performance of the large AI model and is the minimum threshold for performance improvement. If the overall performance score is lower than this performance threshold, the performance of the large AI model is considered not to have improved. The performance threshold can be 0. When the overall performance score is less than the performance threshold, the ε parameter is increased to improve exploration, and the β parameter is decreased to relax the KL constraint, thus increasing the update range of the large AI model. When the overall performance score is not less than the performance threshold, the amount of reduction of the pruning parameter is determined according to the decay ratio coefficient; the amount of increase of the KL divergence coefficient is determined according to the growth coefficient. Among them, when the overall performance score is not less than the performance threshold, it indicates that the performance of the AI ​​large model has been improved. The ε parameter is reduced to reduce exploration, and the β parameter is increased to tighten the KL constraint and limit the policy update range.

[0045] The formula is as follows:

[0046]

[0047] in, This indicates the overall performance score; Indicates the performance threshold; and They represent The maximum and minimum values ​​are obtained from user-defined configurations; and These represent the maximum and minimum values ​​of β, respectively, which are obtained through user-defined configuration. Indicates the growth coefficient; This represents the attenuation ratio coefficient.

[0048] This application calculates the dynamic weight of reward and loss values ​​based on the changes in reward and loss values ​​during training, adjusts the weights alpha / beta in the loss function, and then automatically adjusts the coefficients β (β_kl) of the exploration ε (epsilon) and KL divergence penalty terms, thereby effectively improving policy convergence and generalization ability.

[0049] In some embodiments of this application, after each round of training, the loss value is calculated according to the objective function of the configured AI model, and the AI ​​model is updated and optimized through backpropagation; the above process is repeated until the AI ​​model converges, and a trained AI model is obtained.

[0050] The objective function of the large AI model consists of two parts: the first part is a pruned policy gradient term, used to maximize the advantage-weighted policy probability ratio, limiting the policy update magnitude through pruning; the second part is a KL divergence penalty term, used to constrain the current policy from deviating too far from the reference policy, ensuring training stability. The learning objective of the large AI model is to optimize the parameters of the policy model by maximizing the expected cumulative reward. (Weights and biases); The objective function for a large AI model is as follows:

[0051] in, Represents a large AI model; This represents the reference model, typically the initial SFT model, used to calculate the KL divergence to prevent the policy model from deviating excessively from the initial model; it is also used to penalize models with large deviations. It is usually the same as the old AI large model. This represents the expectation of the problem distribution P(Q); in practical applications, this is usually achieved by sampling over a batch containing multiple problems. The relative advantage is calculated by sampling a set of candidate outputs and normalizing the reward of each candidate output within the set. It refers to the number of candidate outputs when sampling for each problem, also known as the group size; For input content Output The average of the candidate outputs is taken. Is this the current state of large AI models in a given problem? In the case of generating the first candidate outputs The probability of; It is the old AI big model (the parameters of the AI ​​big model in previous iterations) in a given problem In the case of generating the first candidate outputs The probability of using the old AI model is to limit the extent of updates to the AI ​​model. It is the current large-scale AI model Compared with reference AI large model In the given problem The KL divergence under certain conditions is used to prevent excessive policy changes. (Old AI large model) These are models frozen prior to the current time, intended for use with new large AI models. The comparison is made to prevent drastic changes in the model; while the new AI big model is the AI ​​big model that has just been updated through reinforcement learning at the current time (the AI ​​big model after the weights and bias parameters have been updated).

[0052] In some embodiments of this application, such as Figure 5 As shown, the methods for updating large AI models include: (1) Using an AI large model, a set of candidate outputs is generated based on the configured prompts and the user's input content (i.e., Figure 5 (Results in the middle) (2) Calculate the reward score for each candidate output, and obtain the relative advantage (i.e., relative reward value) of each candidate output by normalizing the reward score of each candidate output. (3) Use a pruning strategy to optimize the objective; Specifically, the update scope of the AI ​​big model is determined by the new AI big model (i.e., the current AI big model). And the old AI big model (i.e., the previous version or the previous AI big model of the current AI big model) Enter content Generate actions (i.e., outputs) under certain circumstances It is measured by the probability ratio of )

[0053] in, This represents the ratio of the strategies of the new AI model to those of the old AI model in the unpruned target items, measuring the performance of the new AI model. Generate Actions The probability compared to the old AI large model The change in this ratio, if greater than 1, means that the new AI model is significantly better than the old AI model. Assign a higher probability; conversely, if the ratio is less than 1, it indicates that the new AI model is... Assigning a lower probability. By comparing the probability differences in outputs produced by the old and new AI models under the same input, we can effectively evaluate whether the policy update direction meets expectations and provide a basis for subsequent policy adjustments; the policy gradient term in the objective function utilizes this ratio. With advantage function Multiplication. To prevent excessive policy updates that could lead to training instability, large AI models trim this ratio to […]. Within the range of […]. The purpose of pruning is to ensure that the new large AI model does not undergo overly aggressive and drastic updates, improve the stability of the training process, and prevent model performance degradation or crashes due to excessive policy updates. Without pruning, the new large AI model would deviate significantly from the old large AI model. To prevent this, a hyperparameter ε is needed to limit the magnitude of policy updates. A smaller ε will make policy updates more conservative, while a larger ε will allow for more significant policy updates.

[0054]

[0055]

[0056] in, The loss is represented by the policy gradient term, which is the minimum of the unpruned ratio multiplied by the advantage and the pruned ratio multiplied by the advantage. Minimize losses for the agent.

[0057] (4) KL divergence regularization; In practice, while pruning largely ensures reasonable policy updates, unsatisfactory results can still occur at times. Therefore, to prevent this, a KL divergence penalty term (KL divergence is a measure of the difference between two probability distributions) will be added to assess the impact of pruning on a given problem. The degree of change under certain conditions, current large AI models Compared to Referencing the degree of change in large AI models It's a policy ratio; this method is called KL divergence regularization, which encourages new large AI models to improve performance while simultaneously making the current (updated) large AI model more efficient by adding a divergence penalty constraint to the objective function. Compared with reference AI large model The differences between them should not deviate too much, to avoid The computation is made more complex, thereby optimizing the objective function, maintaining training stability, and avoiding policy collapse.

[0058]

[0059]

[0060] in, This is the coefficient of the KL divergence penalty term, which controls the strength of the penalty and is used to balance the objective and scale the KL divergence penalty. It is weighted in the total loss function by the coefficient β. A larger β makes the AI ​​model update closer to the reference policy, with smaller policy changes, while a smaller β makes the AI ​​model update deviate more from the reference policy, with larger policy changes. When the value is large, the penalty increases, making the AI ​​model updates more conservative; conversely, when the value is small... When the size is smaller, the penalty is reduced, allowing for more aggressive updates to large AI models. In this way, the algorithm limits the magnitude of change in the large AI model while pursuing greater rewards (measured by the advantage function). This mechanism helps the model retain existing knowledge while learning new information, avoiding "catastrophic forgetting."

[0061] (5) Adaptive adjustment of hyperparameters is performed using the method of dynamic adjustment of hyperparameters in AI large model reinforcement learning as described in the embodiments of this application; the AI ​​large model is updated and optimized according to the objective function containing the adaptively adjusted hyperparameters.

[0062] In some other embodiments of this application, in order to avoid the instability of reinforcement learning in the early stage of training, and to accelerate the convergence of reinforcement learning training and shorten the training time, a cold start method or a change in the number of training rounds can be used to make the training more stable.

[0063] Corresponding to the above method, this application also provides a device for dynamically adjusting hyperparameters of large AI model reinforcement learning, such as... Figure 6 As shown, the device for dynamically adjusting the hyperparameters of this large AI model reinforcement learning includes: The acquisition unit 610 is used to acquire the first performance evaluation value of the AI ​​large model under the first time window and the second performance evaluation value under the second time window during the training process; wherein, any time window includes multiple training rounds of the training process; the performance evaluation value of the AI ​​large model under any time window is determined based on the performance evaluation value of the AI ​​large model under different training rounds within the corresponding time window; the second time window is later than the first time window; The first determining unit 620 is used to determine the performance change of the AI ​​large model from the first time window to the second time window based on the difference between the first performance evaluation value and the second performance evaluation value. The computing unit 630 is used to calculate the comprehensive performance score of the large AI model based on the performance change, the first performance evaluation value, and the second performance evaluation value. The second determining unit 640 is used to determine the adjustment direction of the hyperparameters of the AI ​​large model based on the comprehensive performance score and the configured performance threshold. The adjustment unit 650 is used to adjust the hyperparameters of the large AI model according to the configured adjustment factor and adjustment direction.

[0064] The functions of each functional unit in the AI ​​large model reinforcement learning hyperparameter dynamic adjustment device provided in the above embodiments of this application can be implemented through the above methods and steps. Therefore, the specific working process and beneficial effects of each unit in the AI ​​large model reinforcement learning hyperparameter dynamic adjustment device provided in the embodiments of this application will not be repeated here.

[0065] This application also provides an electronic device, such as... Figure 7 As shown, it includes a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740.

[0066] Memory 730 is used to store computer programs; When the processor 710 executes the program stored in the memory 730, it performs the following steps: Obtain the first performance evaluation value of the AI ​​large model under the first time window and the second performance evaluation value under the second time window during the training process; wherein, any time window includes multiple training rounds of the training process; the performance evaluation value of the AI ​​large model under any time window is determined based on the performance evaluation values ​​of the AI ​​large model under different training rounds within the corresponding time window; the second time window is later than the first time window; The performance change of the AI ​​model from the first time window to the second time window is determined based on the difference between the first performance evaluation value and the second performance evaluation value. The comprehensive performance score of the large AI model is calculated based on the performance change, the first performance evaluation value, and the second performance evaluation value. Based on the comprehensive performance score and the configured performance threshold, determine the direction for adjusting the hyperparameters of the large AI model; The hyperparameters of the large AI model are adjusted according to the configured adjustment factors and adjustment directions.

[0067] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0068] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0069] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0070] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0071] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments for solving the problem can be found in [reference needed]. Figure 1The steps in the illustrated embodiments are used to implement the electronic device. Therefore, the specific working process and beneficial effects of the electronic device provided in this application will not be repeated here.

[0072] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform the method for dynamically adjusting hyperparameters of AI large-scale model reinforcement learning as described in any of the above embodiments.

[0073] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the method for dynamic adjustment of hyperparameters of AI large-scale reinforcement learning as described in any of the above embodiments.

[0074] Those skilled in the art will understand that the embodiments in this application can be provided as methods, systems, or computer program products. Therefore, the embodiments in this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments in this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0079] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims in this application and their equivalents, then this application also intends to include these modifications and variations.

Claims

1. A method for dynamically adjusting hyperparameters in large AI model reinforcement learning, characterized in that, The method is applied on a server and is used in the training process of a large AI model. The method includes: The method involves obtaining a first performance evaluation value for a large AI model within a first time window and a second performance evaluation value within a second time window during training. Each time window includes multiple training epochs. The performance evaluation value of the large AI model within any time window is determined based on its performance evaluation values ​​across different training epochs within that time window. The second time window is later than the first time window. The large AI model is used to output the probability distribution of word units based on user-generated questions. The performance evaluation value includes a reward value and a loss value. The performance change of the large AI model from the first time window to the second time window is determined based on the difference between the first and second performance evaluation values. The method for obtaining the reward value is described below. This includes: for any given time window, acquiring multiple candidate outputs of the AI ​​model under different training epochs within that time window; in practical applications and in any training epoch, the AI ​​model generates multiple candidate outputs based on user input and a configured response strategy; the response strategy includes: prompt word templates, temperature, whether to enable a sampling strategy, and maximum text length; the temperature is a parameter used to adjust the sampling probability distribution; for any given training epoch, acquiring reward scores for different candidate outputs under that training epoch; determining the relative reward value of the AI ​​model in that training epoch based on the reward scores corresponding to different candidate outputs; and obtaining the reward value of the AI ​​model under the given time window based on the relative reward value of the AI ​​model in each training epoch. The comprehensive performance score of the AI ​​large model is calculated based on the performance change, the first performance evaluation value, and the second performance evaluation value. Based on the comprehensive performance score and the configured performance threshold, the adjustment direction of the hyperparameters of the AI ​​large model is determined; the hyperparameters include: pruning parameters and KL divergence coefficient; The hyperparameters of the large AI model are adjusted according to the configured adjustment factor and the adjustment direction.

2. The method as described in claim 1, characterized in that, Determining the relative reward value of the AI ​​large model in the training round based on the reward scores corresponding to different candidate outputs includes: The average of the reward scores corresponding to each candidate output is taken as the average reward score of the AI ​​model in the training round. The standard deviation of the reward score corresponding to each candidate output is used as the reward standard deviation of the AI ​​large model in the training round. For any candidate output, the difference between the reward score corresponding to the candidate output and the average reward score is taken as the target difference. The ratio of the target difference to the reward standard deviation is used as the relative reward value of the candidate output; Based on the relative reward values ​​of each candidate output in the training round, the relative reward value of the AI ​​large model in the training round is determined.

3. The method as described in claim 1, characterized in that, The performance changes include: changes in reward and changes in loss; Based on the difference between the first performance evaluation value and the second performance evaluation value, the performance change of the AI ​​large model from the first time window to the second time window is determined, including: The difference between the second reward value and the first reward value is used as the change in reward. The difference between the second loss value and the first loss value is taken as the change in loss. Based on the absolute values ​​of the changes in reward and loss, the performance change of the AI ​​model from the first time window to the second time window is determined.

4. The method as described in claim 3, characterized in that, Based on the performance change, the first performance evaluation value, and the second performance evaluation value, the comprehensive performance score of the AI ​​large model is calculated, including: Calculate the weight of the reward change based on the reward change and the performance change; Calculate the weight of the change in loss based on the change in loss and the change in performance; The change in reward is weighted using the weights of the change in reward to obtain the weighted change in reward. The change in loss is weighted using the weights of the change in loss to obtain the weighted change in loss; The difference between the weighted change in reward and the weighted change in loss is used as the overall performance score of the AI ​​model.

5. The method as described in claim 1, characterized in that, The adjustment directions include: increasing and decreasing; Based on the comprehensive performance score and the configured performance threshold, the adjustment direction of the hyperparameters of the large AI model is determined, including: If the overall performance score is less than the performance threshold, the adjustment direction of the trimming parameter is determined to be increased, and the adjustment direction of the KL divergence coefficient is determined to be decreased. If the overall performance score is not less than the performance threshold, then the adjustment direction of the trimming parameter is determined to be decreasing, and the adjustment direction of the KL divergence coefficient is determined to be increasing.

6. The method as described in claim 5, characterized in that, The adjustment factors include: growth coefficient and decay ratio coefficient; The hyperparameters of the large AI model are adjusted according to the configured adjustment factor and the adjustment direction, including: When the overall performance score is less than the performance threshold, the amount of increase in the trimming parameter is determined according to the growth coefficient; and the amount of decrease in the KL divergence coefficient is determined according to the decay ratio coefficient. When the overall performance score is not less than the performance threshold, the amount of reduction of the trimming parameter is determined according to the attenuation ratio coefficient; and the amount of increase of the KL divergence coefficient is determined according to the growth coefficient.

7. An apparatus for dynamically adjusting hyperparameters in large-scale AI reinforcement learning as described in any one of claims 1-6, characterized in that, The device includes: An acquisition unit is used to acquire a first performance evaluation value of the AI ​​large model under a first time window and a second performance evaluation value under a second time window during the training process; wherein, any time window includes multiple training rounds of the training process; the performance evaluation value of the AI ​​large model under any time window is determined based on the performance evaluation values ​​of the AI ​​large model under different training rounds within the corresponding time window; the second time window is later than the first time window; The first determining unit is used to determine the performance change of the AI ​​large model from the first time window to the second time window based on the difference between the first performance evaluation value and the second performance evaluation value. The calculation unit is used to calculate the comprehensive performance score of the AI ​​large model based on the performance change, the first performance evaluation value, and the second performance evaluation value. The second determining unit is used to determine the adjustment direction of the hyperparameters of the AI ​​large model based on the comprehensive performance score and the configured performance threshold. The adjustment unit is used to adjust the hyperparameters of the AI ​​large model according to the configured adjustment factor and the adjustment direction.

8. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Citation Information

Patent Citations

  • AI video large model training method based on 5G network slicing

    CN119364421A

  • Federal learning equipment scheduling optimization method and device based on double-layer reinforcement learning

    CN120068994A