Large language model optimization method based on reinforcement learning fine tuning

Abstract

The invention relates to the technical field of large-scale language models, and discloses a large-scale language model optimization method based on reinforcement learning fine tuning, which comprises the following steps: collecting advantage values obtained by interaction of a target model, and using an advantage matrix with the size of T * B; performing segmentation processing on the advantage matrix from a time dimension according to a preset window to form m fragments; source distribution of an advantage matrix is determined according to the m fragments, and a cost matrix is constructed; based on the source distribution and the cost matrix, determining a transmission matrix with the size of m * m; determining the net energy carrying amount of the pth fragment based on the transmission matrix; according to the net energy carrying amount of the pth fragment, the advantage matrix and a preset window, updating the advantage matrix to obtain an updated advantage matrix; and calculating a loss value based on the updated advantage matrix, executing back propagation by using the loss value, and updating the target model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language models, more particularly, it relates to a large language model optimization method based on reinforcement learning fine-tuning. BACKGROUND

[0002] Large language models widely apply reinforcement learning fine-tuning technology (for example, RLHF, RLAIF) in completing instruction alignment, human preference alignment and other tasks. The learning fine-tuning technology corrects the generation strategy of the language model in the later period by introducing a reward signal. In actual tasks, the reward signal does not exist uniformly in each generation step. For example, in dialog systems, summary generation, code generation and other tasks, user feedback or reward evaluation often only occurs at the end of generation or at some key moments, while the reward signal at most time steps is zero or invalid value, which is called reward sparsity.

[0003] The strategy update of reinforcement learning mainly depends on the estimation of the advantage function. In actual implementation, the advantage function is often obtained through time difference (TD) estimation or generalized advantage estimation (GAE), and is finally used to guide the gradient update of the strategy. In the environment of reward sparsity, since the rewards of most time steps are zero, only a few time segments have non-zero advantage values in the process of advantage function estimation, and the advantage values of most time steps are close to zero. The uneven distribution of such advantage vectors directly leads to the concentration of the energy of the advantage covariance matrix or the related gradient update matrix on a few time segments when they are used to construct the advantage covariance matrix or the related gradient update matrix.

[0004] When the training trajectory is divided into multiple time windows, the number and intensity of the reward signals corresponding to each window present an extremely uneven distribution. This makes the high-dimensional matrix used to calculate the strategy gradient exhibit the following trends in mathematical properties:

[0005] 1. Most of the matrix components have almost no energy, and the effective rank of the matrix rapidly decreases;

[0006] 2. A few components or sub-blocks dominate the overall numerical range of the matrix due to the accumulation of a small amount of high-amplitude rewards;

[0007] 3. The eigenvalues of the matrix present a huge amplitude difference, leading to a sharp deterioration of the numerical condition number.

[0008] This uneven phenomenon of the advantage function matrix caused by reward sparsity further leads to the following serious consequences:

[0009] 1. At the beginning of training, the direction of strategy update is dominated by a few sparse rewards, and the update step is amplified in some local time periods, which further leads to overfitting of the local strategy;

[0010] 2. After task switching, reward position change or sampling distribution drift, the model update no longer has sufficient smooth transition ability, resulting in obvious oscillation phenomenon in the training process;

[0011] 3. Due to the decrease of the effective rank of the matrix, the policy update of the model on the non-reward area gradually fails, and the overall training convergence speed decreases significantly.

[0012] In summary, the specific formation process of the credit signal imbalance can be summarized as follows: first, the reward sparsity causes the advantage function value to present an unbalanced distribution in the time dimension; second, the sum of squares of the advantage function focuses on a limited time segment, so that the high-dimensional matrix constructed based on the advantage is highly skewed in energy distribution; third, since the policy update of reinforcement learning is highly dependent on the gradient direction provided by the advantage matrix, the imbalance of the matrix energy distribution directly leads to the deviation of the gradient direction from the global optimum during training, causing training oscillation and performance instability; finally, in high-dimensional space, this problem continues to worsen as the training scale expands and the time steps increase, seriously affecting the stable training and alignment effect of large models. SUMMARY

[0013] The application provides a large language model optimization method based on reinforcement learning fine-tuning, which solves the technical problem of convergence difficulty of large language model reinforcement learning caused by energy imbalance of advantage matrix under sparse reward in related technologies.

[0014] The application provides a large language model optimization method based on reinforcement learning fine-tuning, which includes:

[0015] S1, collecting the advantage values obtained by the target model interaction to form an advantage matrix with a size of TxB; wherein T represents the number of time dimensions, and B represents the number of batch dimensions;

[0016] S2, according to the preset window, the advantage matrix is segmented from the time dimension to form m segments;

[0017] S3, according to the m segments, the source distribution of the advantage matrix is determined respectively, and a cost matrix is constructed;

[0018] S4, based on the source distribution and the cost matrix, a transmission matrix with a size of mxm is determined;

[0019] S5, based on the transmission matrix, the net energy carrying capacity of the pth segment is determined;

[0020] S6, according to the net energy carrying capacity of the pth segment, the advantage matrix and the preset window, the advantage matrix is updated to obtain an updated advantage matrix;

[0021] S7, based on the updated advantage matrix, the loss value is calculated, the loss value is used for back propagation, and the target model is updated.

[0022] Further, the advantage matrix is segmented in time dimension according to the preset window to form m segments, including:

[0023] The number m of segments is determined according to the length W of the preset window,

[0024] The advantage matrix is sequentially intercepted in time dimension based on the number m of segments to form m segments;

[0025] When the time length of the mth segment is less than the length W of the preset window, the mth segment is padded with zero to complete.

[0026] Further, the source distribution of the advantage matrix is determined according to the m segments, including:

[0027] The square sum of the elements in the pth segment is calculated to obtain the energy e p of the pth segment;

[0028] Based on the median of the energies of the m segments, the m segments are divided into a high-energy group and a low-energy group;

[0029] The maximum energy e max in the high-energy group and the minimum energy e min in the low-energy group are selected, respectively;

[0030] If the pth segment is in the high-energy group, the corresponding updated energy is:

[0031] If the pth segment is in the low-energy group, the corresponding updated energy is:

[0032] The energy sum value of the updated energies of the m segments is calculated;

[0033] The ratio of the updated energy of the pth segment to the energy sum value is taken as the source distribution S p , p∈{1,…,m}.

[0034] Further, a cost matrix is constructed, including:

[0035] A zero matrix C of size m×m is initialized;

[0036] The element in the pth row and qth column of the zero matrix C is assigned a value: C(p,q)=|p-q|W;

[0037] If the check zero matrix C satisfies C(p,q)=C(q,p) and C(p,p)=0, the check passes, and the zero matrix C is taken as the cost matrix.

[0038] Further, based on the source distribution and the cost matrix, a transmission matrix of size m x m is determined, including:

[0039] An entropy regularization coefficient ε is set, and a row dual variable u and a column dual variable v are initialized; wherein the row dual variable u and the column dual variable v are both m-dimensional all-1 vectors;

[0040] Based on the cost matrix and a convergence threshold τ, an exponential matrix K of size m x m is constructed; wherein the element of the pth row and the qth column of the exponential matrix is

[0041] The pth element u p of the row dual variable u is updated as follows:

[0042] The qth element v p of the column dual variable v is updated as follows: Wherein T q represents the target distribution,

[0043] After repeating the update of the row dual variable u and the column dual variable v for a preset number of times, updated row dual variable u(new) and column dual variable v(new) are obtained respectively;

[0044] Based on the updated row dual variable u(new) and the column dual variable v(new), combined with the exponential matrix K, a transmission matrix Π of size m x m is synthesized; wherein the element of the pth row and the qth column of the transmission matrix is Π(p,q)=u(new) p ·K(p,q)·v(new) q .

[0045] Further, based on the transmission matrix, the net energy transport amount of the pth slice is determined, including:

[0046] The diagonal element (p,p) of the transmission matrix is determined, and the element of the pth row and the pth column of the transmission matrix is taken as the self-loop component;

[0047] The self-loop component is removed from the pth column of the transmission matrix, and the sum of the remaining elements of the pth column is calculated as the original total inflow;

[0048] The self-loop component is removed from the pth row of the transmission matrix, and the sum of the remaining elements of the pth row is calculated as the original total outflow;

[0049] The maximum value among the remaining elements of the pth column is determined as the maximum inflow component;

[0050] The maximum value among the remaining elements of the pth row is determined as the maximum outflow component;

[0051] The first corrected weight is obtained by calculating the ratio of the maximum inflow component to the sum of the maximum inflow component and the maximum outflow component. The product of the first corrected weight and the original total inflow is taken as the total inflow.

[0052] The second corrected weight is obtained by calculating the ratio of the maximum outflow component to the sum of the maximum inflow component and the maximum outflow component. The product of the second corrected weight and the original total outflow is taken as the total outflow.

[0053] The difference between the total inflow and the total outflow is used as the net energy transport amount of the p-th segment of the dominance matrix.

[0054] Furthermore, based on the net energy transport volume of the p-th segment, the dominance matrix, and the preset window, the dominance matrix is ​​updated to obtain the updated dominance matrix, including:

[0055] The ratio of the net energy transport amount of the p-th segment to the preset window length W is used as the window increment α of the p-th segment. k ;

[0056] Each element in the p-th slice is compared with its corresponding window increment α. k The p-th slice is added together to form the updated dominance matrix A. new .

[0057] Furthermore, the loss value is calculated based on the updated advantage matrix, and backpropagation is performed using the loss value to update the target model, including:

[0058] Obtain the current policy π of the target model θ and reference strategy π ref And construct a probability ratio matrix R of size T×B;

[0059] After trimming the probability ratio matrix, multiply it with the updated advantage matrix as follows:

[0060] M(t,b)=clip(R(t,b),(1-∈ clip ),(1+∈ clip ))×A new (t,b)

[0061] Where M(t,b) represents the element in the t-th row and b-th column of the clipping product matrix, ∈ clip A represents the clipping threshold. new (t,b) represents updating the element in the t-th row and b-th column of the dominance matrix, and clip represents the over-limit clipping function;

[0062] Calculate the policy loss using the pruning product matrix M. And superimpose regular terms The total loss L is obtained total =L policy+L KL ;

[0063] with the total loss L total Performing back propagation, updating the hyperparameters of the target model according to the preset learning rate η.

[0064] The beneficial effects of the present application are that by introducing the mechanism of slice source distribution, cost matrix and transmission matrix, the dynamic balance adjustment of the advantage matrix in the time dimension is realized, the energy imbalance problem of the advantage matrix caused by sparse reward is effectively solved, and the training stability and convergence speed in the fine-tuning process of large language model reinforcement learning are significantly improved; the net energy carrying capacity is used to smooth and redistribute the advantage signals of different time windows, which takes into account local optimization and global convergence, effectively suppresses gradient shock and update offset in the training process, and ensures that the model alignment effect is better and the training process is more stable. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 is a flowchart of the present application. DETAILED DESCRIPTION

[0066] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is merely meant to provide a better understanding of the subject matter described herein and can be changed in function and arrangement without departing from the scope of the present specification. Various processes or components can be omitted, substituted, or added according to desired implementations. Additionally, features described with respect to some examples can be combined in other examples.

[0067] As Figure 1 shown, a large language model optimization method based on reinforcement learning fine-tuning includes:

[0068] S1, collecting advantage values obtained by target model interaction to form an advantage matrix with a size of TxB; wherein T represents the number of time dimensions, and B represents the number of batch dimensions;

[0069] S2, according to the preset window, the advantage matrix is segmented from the time dimension to form m slices;

[0070] S3, according to the m slices, respectively determine the source distribution of the advantage matrix, and construct the cost matrix;

[0071] S4, based on the source distribution and the cost matrix, determine the transmission matrix with a size of mxm;

[0072] S5, based on the transmission matrix, determine the net energy carrying capacity of the pth slice;

[0073] S6, updating the advantage matrix according to the net energy carrying capacity of the pth slice, the advantage matrix and the preset window to obtain an updated advantage matrix;

[0074] S7, calculating a loss value based on the updated advantage matrix, performing back propagation with the loss value, and updating the target model.

[0075] It should be noted that the advantage matrix is a core data structure for quantifying the improvement direction of the policy in reinforcement learning fine-tuning, and its construction is directly related to the interaction process of the target model. The elements of the advantage matrix are generated based on advantage values. The advantage value is a quantitative indicator in reinforcement learning for measuring the advantage of an action taken at a certain time over the average level action. If the advantage value at a certain time is positive, it means that the action at that time brings higher returns than the average level, and the policy should be strengthened in that direction. If it is negative, it means that the action returns are lower than the average level, and the policy needs to be adjusted. When the target model interacts with the environment (or task), it will generate corresponding advantage values at different time steps and different sample batches. These values form the advantage matrix when arranged according to rules. The size of the advantage matrix is T x B. Wherein, T represents the number of time dimensions, corresponding to the time steps in the model interaction process (for example, in the dialog generation task, T can be understood as the token sequence length of the generated text); B represents the number of batch dimensions, corresponding to the number of samples processed in parallel (for example, B dialog samples are input at the same time). The element in the tth row and bth column of the matrix, that is, the advantage value corresponding to the tth time step in the bth sample batch. Due to the sparsity of rewards (most time steps have no effective rewards), most elements in the advantage matrix have values close to zero, and only a few elements corresponding to the time steps with effective rewards have non-zero values (non-zero values are the key basis for policy update). This feature also makes the advantage matrix a direct object of subsequent energy balancing processing, that is, through subsequent slicing, energy transmission and other operations, the energy distribution of the advantage matrix can be balanced, and the policy update can be prevented from being dominated by a few non-zero values.

[0076] In an embodiment of the present application, the advantage matrix is segmented in the time dimension according to the preset window to form m slices, including:

[0077] The number of slices m is determined according to the length W of the preset window,

[0078] The advantage matrix is sequentially intercepted in the time dimension based on the number of slices m to form m slices;

[0079] Wherein, when the time length of the mth slice is less than the length W of the preset window, the mth slice is padded with zeros to complete the processing.

[0080] It should be noted that the time dimension length of the advantage matrix is T, the length of the preset window is W, and the number of fragments m needs to be determined by calculation. The specific calculation method is to take the ratio of T and W to the upper integer, that is, m is equal to the upper integer of the result of T divided by W. The core of this calculation method is to ensure that all data in the time dimension can be completely covered. For example, if the time dimension length T is 100, the preset window length W is 30, the result of T divided by W is about 3.33, and m is 4 after being rounded up, which means that 4 fragments are needed to cover all 100 time steps of data.

[0081] After determining the value of m, data is sequentially intercepted from the starting position of the time dimension of the advantage matrix according to the preset window length W to form m fragments. The interception process strictly follows the time sequence, the first fragment corresponds to the data from the starting position to the Wth time step in the time dimension, the second fragment corresponds to the data from the W+1th time step to the 2Wth time step, and so on, until the mth fragment is intercepted. This sequential interception method can ensure the correspondence between the fragments and the original time sequence, and ensure that subsequent energy calculation and other operations can reflect the characteristics in the time dimension.

[0082] When the mth fragment is intercepted, if the number of remaining time steps is less than the preset window length W, resulting in a time length of the fragment being less than W, zero padding processing needs to be performed on the fragment. Zero padding means supplementing data with a value of zero to the end of the fragment until its time length reaches W. Thus, the time lengths of all fragments are unified to W, avoiding interference with subsequent energy calculation, transmission matrix construction and other steps due to inconsistent fragment lengths. For example, if the mth fragment originally has 20 time steps left and the preset window length W is 30, 10 zero values need to be supplemented to make the time length of the fragment reach 30.

[0083] In an embodiment of the present application, the source distribution of the advantage matrix is determined according to the m fragments, comprising:

[0084] The sum of squares of the elements in the pth fragment is calculated to obtain the energy e p of the pth fragment;

[0085] Based on the median of the energies of the m fragments, the m fragments are divided into a high-energy group and a low-energy group;

[0086] The maximum energy e max in the high-energy group and the minimum energy e min in the low-energy group are selected respectively;

[0087] If the pth fragment is in the high-energy group, the corresponding updated energy is:

[0088] If the pth fragment is in the low-energy group, the corresponding updated energy is:

[0089] Calculate the sum of the update energies for m partitions;

[0090] The ratio of the update energy of the p-th partition to the sum of its energy values ​​is used as the source distribution S. p , p∈{1,…,m}.

[0091] It should be noted that for the p-th partition, the sum of squares of all elements within it is calculated, and the result is the energy e of that partition. p The reason for using the sum of squares to calculate energy is that it avoids the cancellation of positive and negative values, accurately reflects the overall amplitude level of elements within a slice, and thus objectively measures the dominant signal energy contained in that slice. First, the median of the energy of m slices is calculated. Then, using this median as a boundary, all slices are divided into two groups: slices with energy greater than or equal to the median are assigned to the high-energy group, and slices with energy less than the median are assigned to the low-energy group. Using the median as the dividing standard ensures that the number of slices in the two groups is roughly balanced, providing a symmetrical operational basis for subsequent energy adjustments. The maximum energy value e is selected from the high-energy group. max Select the minimum energy value e from the low energy group. min Using this as a benchmark, the energy difference between the high-energy group and the low-energy group is quantified, providing a clear proportional basis for subsequent scaling adjustments. For the p-th segment of the high-energy group, its updated energy e... p (new) equals the original energy e p Multiply by e min With e max The ratio of [the energy level] to [the energy level]. This reduces the energy of the high-energy fragments, preventing them from excessively dominating subsequent transmission processes. For the p-th fragment of the low-energy group, its update energy e [is...]. p (new) equals the original energy e p Multiply by e max With e min The ratio of the energy of the low-energy fragments is used to amplify their energy and increase their weight in subsequent transmissions, achieving a preliminary balance in energy distribution. The updated energies of the m fragments are summed to obtain the sum of their energies. This sum is the basis for subsequent normalization, ensuring that the final source distribution meets the basic requirement of a probability distribution (the sum of all components equals 1). The updated energy e of the p-th fragment is... p The ratio of (new) to the sum of energy values ​​is used as the source distribution S of this partition. p (The value of p ranges from 1 to m). The source distribution is essentially the normalized proportion of the energy after each piece is updated. Its function is to transform the energy characteristics of the pieces into a probability distribution form that meets the requirements of subsequent optimal transmission calculation, and to provide input basis for the construction of the transmission matrix.

[0092] In one embodiment of the present invention, constructing the cost matrix includes:

[0093] initialize a zero matrix C with size m*m;

[0094] assign a value to an element in the pth row and qth column of the zero matrix C, C(p,q)=|p-q|W;

[0095] if the zero matrix C satisfies C(p,q)=C(q,p) and C(p,p)=0, the check is passed, and the zero matrix C is used as the cost matrix.

[0096] It should be noted that a zero matrix C with size m*m is first created, where m is the number of fragments determined previously. The initialization of the zero matrix is to provide a basic framework, and specific values will be filled in the subsequent steps according to the relationship between the fragments. The element in the pth row and qth column of the zero matrix C is assigned a value, and the specific assignment rule is that the element is equal to the absolute value of the difference between p and q multiplied by the preset window length W. The core of the assignment logic is based on the time correlation of the fragments, that is, p and q represent the indexes of two fragments, and the absolute value of the difference between them reflects the distance between the two fragments in the time sequence. The closer the distance, the closer the two fragments in time; multiplying the window length W converts the index distance into the actual time length distance. This assignment method makes the cost of transporting energy between adjacent fragments in time lower, and the cost of transporting energy between fragments far apart in time higher, which conforms to the logic that energy in the time dimension should flow preferentially between adjacent fragments, avoiding the destruction of time correlation caused by long-distance transportation. After completing the element assignment, the zero matrix C needs to be checked, which needs to satisfy two conditions. The first condition is that the element in the pth row and qth column of the zero matrix C is equal to the element in the qth row and pth column, so as to ensure that the cost matrix has symmetry, that is, the cost of transporting energy from fragment p to fragment q is the same as the cost of transporting energy from fragment q to fragment p, which conforms to the basic logic of consistent cost in both directions. The second condition is that the element in the pth row and pth column of the zero matrix C is equal to 0, which ensures that the energy transportation of the fragment itself (i.e., the energy remains in the fragment itself) has no cost, which conforms to the actual rule that there is no transportation distance from itself to itself. Only when both conditions are met, the zero matrix C is confirmed to be valid and can be used as a cost matrix for subsequent steps.

[0097] In an embodiment of the present application, based on the source distribution and the cost matrix, a transmission matrix with size m*m is determined, comprising:

[0098] Set the entropy regularization coefficient ε, and initialize the row dual variable u and the column dual variable v; wherein the row dual variable u and the column dual variable v are both m-dimensional all-1 vectors;

[0099] Based on the cost matrix and the convergence threshold τ, an exponential matrix K with size m*m is constructed; wherein the element in the pth row and qth column of the exponential matrix

[0100] the pth element u p is updated as follows:

[0101] the qth element v p is updated as follows: where T q denotes the target distribution,

[0102] After the preset number of iterations of updating the row dual variable u and the column dual variable v, the updated row dual variable u(new) and the column dual variable v(new) are obtained, respectively;

[0103] Based on the updated row dual variable u(new) and the column dual variable v(new), the transmission matrix Π of size m x m is synthesized in combination with the exponential matrix K, wherein the element Π(p, q) of the pth row and the qth column of the transmission matrix is equal to u(new) p ·K(p, q)·v(new) q .

[0104] It should be noted that first, the entropy regularization coefficient ε is set, and the role of the entropy regularization coefficient ε is to avoid the excessive concentration of the energy of the transmission matrix in a small number of fragment pairs by introducing an entropy regularization term, and to improve the smoothness of energy transmission; at the same time, the row dual variable u and the column dual variable v are initialized, both of which are m-dimensional all-1 vectors. The initialization of the dual variable provides a starting point for subsequent iterative optimization, and the all-1 vector ensures the balance of the weights of each fragment in the initial state. Based on the constructed cost matrix C and the entropy regularization coefficient ε, the exponential matrix K of size m x m is constructed, wherein the element of the pth row and the qth column is equal to the negative C of the pth row and the qth column of the natural exponential function e divided by the pth power of ε. The core of the exponential matrix K is to convert the cost into a probability correlation, that is, the smaller the element value in the cost matrix (i.e., the closer the time distance between two fragments), the larger the corresponding element value in the exponential matrix, which means that the energy transmission between the two fragments is more likely to be preferentially selected, providing a basis for the calculation of the subsequent transmission probability. The pth element of the row dual variable u is updated, and the update rule is that the element is equal to the source distribution value S p of the pth fragment in the source distribution divided by the sum of the products of all elements of the pth row in the exponential matrix K and the corresponding elements in the column dual variable v. Thus, the row dual variable is adapted to the source distribution, that is, by adjusting the element value of u through the ratio of the source distribution value to the current transmission correlation sum, the subsequent transmission is more in line with the initial distribution of the energy of the source distribution. The qth element of the column dual variable v is updated, and the update rule is that the element is equal to the target distribution value T q of the qth fragment in the target distribution divided by the sum of the products of all elements of the qth column in the exponential matrix K and the corresponding elements in the row dual variable u. The target distribution T q is That is, uniform distribution, representing the ideal balanced state of energy transmission. Thus, let the column dual variable adapt to the target distribution, and adjust the element value of v by the ratio of the target distribution value to the sum of the current transmission, to guide the transmission to optimize in the direction of uniform energy distribution. For the row dual variable u and the column dual variable v, repeat the update according to the above rules for a predetermined number of times. The effect of multiple iterations is to gradually converge the dual variables to the optimal state, that is, by constantly adjusting u and v, their matching degrees with the source distribution and the target distribution are gradually increased, providing accurate weight parameters for the synthesis of the subsequent transmission matrix. Based on the updated row dual variable u(new) and the column dual variable v(new), combined with the exponential matrix K, a transmission matrix Π of size m×m is synthesized. The element of the pth row and qth column of the transmission matrix is equal to the product of the pth element of u(new) and the element of the pth row and qth column of the exponential matrix K and the qth element of v(new). Thus, the weight of the dual variable is combined with the transmission between the fragments, and the final transmission matrix element value quantifies the proportion of energy transmission from the pth fragment to the qth fragment, which not only meets the conversion requirements from the source distribution to the target distribution, but also follows the energy transmission cost rules defined by the cost matrix (the closer the distance between the fragment pairs, the more reasonable the transmission proportion will be). The transmission matrix can realize the optimal energy transmission from the source distribution (initial energy distribution) to the target distribution (uniform energy distribution), that is, to minimize the transmission cost while completing the balanced allocation of energy.

[0105] In an embodiment of the present application, the net energy carrying amount of the pth fragment is determined based on the transmission matrix, comprising:

[0106] The diagonal element (p, p) of the transmission matrix is determined, and the element of the pth row and pth column of the transmission matrix is taken as the self-loop component;

[0107] The self-loop component is removed from the pth column of the transmission matrix, and the sum of the remaining elements of the pth column is calculated as the original inflow total;

[0108] The self-loop component is removed from the pth row of the transmission matrix, and the sum of the remaining elements of the pth row is calculated as the original outflow total;

[0109] The maximum value in the remaining elements of the pth column is determined as the maximum inflow component;

[0110] The maximum value in the remaining elements of the pth row is determined as the maximum outflow component;

[0111] The maximum inflow component is calculated, and the ratio of the maximum inflow component to the sum of the maximum inflow component and the maximum outflow component is obtained as the first correction weight, and the product of the first correction weight and the original inflow total is taken as the inflow total;

[0112] The ratio of the maximum outflow component to the sum of the maximum inflow component and the maximum outflow component obtains a second correction weight, and the product of the second correction weight and the original outflow total quantity is taken as the outflow total quantity;

[0113] The difference between the inflow total quantity and the outflow total quantity is taken as the net energy transport quantity of the pth slice of the advantage matrix.

[0114] It should be noted that the transmission matrix is a matrix of size m x m, and the element Π(p, q) represents the proportion of energy transmitted from the pth slice to the qth slice. The element Π(p, p) in the pth row and pth column is the self-loop component, which represents the energy retained by the pth slice itself (not transmitted to other slices), so when calculating the energy transmitted to or received from the outside, the component needs to be excluded first. The original inflow total quantity is the sum of the energy received by the pth slice from all other slices. The specific operation is to take the pth column of the transmission matrix (the elements of this column correspond to the transmission proportion of all slices to the pth slice), remove the self-loop component Π(p, p), and then sum the remaining elements to obtain the original inflow total quantity. The original outflow total quantity is the sum of the energy output by the pth slice to all other slices. The specific operation is to take the pth row of the transmission matrix (the elements of this row correspond to the transmission proportion of the pth slice to all slices), remove the self-loop component Π(p, p), and then sum the remaining elements to obtain the original outflow total quantity. The maximum inflow component is the main source of energy received by the pth slice. Among the remaining elements after removing the self-loop component from the pth column of the transmission matrix, the element with the largest value is selected, which corresponds to the transmission proportion of the slice that transmits the most energy to the pth slice.

[0115] It should be noted that in order to enhance the influence of the main transmission path on energy flow (i.e., to make energy flow mainly through the most active transmission direction), the original inflow and outflow total quantities need to be modified using the maximum inflow component and the maximum outflow component: the first correction weight is the ratio of the maximum inflow component to the sum of the maximum inflow component and the maximum outflow component, which reflects the contribution proportion of the main inflow path; the first correction weight is multiplied by the original inflow total quantity to obtain the modified inflow total quantity. The second correction weight is the ratio of the maximum outflow component to the sum of the maximum inflow component and the maximum outflow component, which reflects the contribution proportion of the main outflow path; the second correction weight is multiplied by the original outflow total quantity to obtain the modified outflow total quantity.

[0116] It should be noted that the net energy transport quantity is the final energy increase or decrease result of the pth slice, which is obtained by subtracting the modified outflow total quantity from the modified inflow total quantity. If the result is positive, it means that the slice obtains energy from other slices; if it is negative, it means that the slice outputs energy to other slices. This result is directly used for subsequent energy adjustment of the advantage matrix, realizing smooth transfer of energy from high-energy slices to low-energy slices.

[0117] In one embodiment of the present invention, the dominance matrix is ​​updated based on the net energy transport volume of the p-th slice, the dominance matrix, and a preset window to obtain an updated dominance matrix, including:

[0118] The ratio of the net energy transport amount of the p-th segment to the preset window length W is used as the window increment α of the p-th segment. k ;

[0119] Each element in the p-th slice is compared with its corresponding window increment α. k The p-th slice is added together to form the updated dominance matrix A. new .

[0120] It should be noted that the net energy transfer amount of the p-th slice reflects the final total energy increase or decrease of that slice, while the preset window length W is the number of elements contained in that slice (each slice is divided by length W). To ensure that the energy change is evenly distributed among all elements within a slice, the net energy transfer amount needs to be evenly distributed to each element, i.e., the window increment is equal to the ratio of the net energy transfer amount of the p-th slice to the preset window length W. This transforms the total energy change at the slice level into an average energy change at the element level, avoiding energy concentration in a few elements within a slice and ensuring the smoothness of energy adjustment. For the p-th slice, each element it contains is added to the window increment obtained in the first step. Since the window increment is the energy change averaged across each element, the dominance values ​​of all elements within the slice are adjusted synchronously. If the net energy transfer amount is positive (the slice gains energy), the dominance values ​​of all elements within the slice increase; if it is negative (the slice outputs energy), the dominance values ​​of all elements decrease. After all slices have completed the above update, the original dominance matrix is ​​replaced by the updated dominance matrix. By transforming the energy transfer results between slices into specific adjustments to the elements of the dominance matrix, uniform distribution ensures that energy changes do not disrupt the temporal correlation within slices, while making the energy distribution of the adjusted dominance matrix more balanced, providing a more stable gradient direction for subsequent model updates.

[0121] In one embodiment of the present invention, calculating a loss value based on the updated advantage matrix, performing backpropagation with the loss value, and updating the target model includes:

[0122] Obtain the current policy π of the target model θ and reference strategy π ref And construct a probability ratio matrix R of size T×B;

[0123] After trimming the probability ratio matrix, multiply it with the updated advantage matrix as follows:

[0124] M(t,b)=clip(R(t,b),(1-∈ clip ),(1+∈ clip ))×Anew (t,b)

[0125] wherein M(t,b) denotes the element in the t-th row and b-th column of the clipped product matrix, and clip denotes the clipping threshold, and A new (t,b) denotes the element in the t-th row and b-th column of the updated advantage matrix, and clip denotes a clipping function;

[0126] The strategy loss is calculated based on the clipped product matrix M and a regularization term is added to obtain the total loss L total = L policy + L KL ;

[0127] Backpropagation is performed based on the total loss L total and the hyperparameters of the target model are updated according to a preset learning rate η.

[0128] It should be noted that the current strategy of the target model is the strategy of the target model in the current training stage, which is the object to be optimized, and its output will change with the adjustment of the model parameters. The reference strategy is a benchmark strategy used to measure the change amplitude of the current strategy, and is usually selected as the strategy of the last training stage of the model or the initial strategy, and its output remains fixed in the current optimization step. Both of them output in the form of probability distribution, that is, for a given input (such as dialogue history or task instruction), the probability of the corresponding action (such as generated token) will be output in each time step and each sample batch. The size of the probability ratio matrix R is T x B, wherein the element in the t-th row and b-th column represents the output probability ratio of the current strategy to the corresponding probability of the reference strategy in the b-th batch and the t-th time step. The role of the probability ratio matrix is to quantify the deviation of the current strategy from the reference strategy, and the value greater than 1 indicates that the output probability of the current strategy in that time step and batch is higher, and the value less than 1 indicates that it is lower.

[0129] First, the probability ratio matrix R is clipped, and each element in the matrix is limited to the range of 1 minus the clipping threshold to 1 plus the clipping threshold through the hyper-limit clipping function. Thus, it is avoided that the probability ratio is too large or too small to cause the policy update to be too large, and it is prevented that the training is unstable. The clipped probability ratio matrix is multiplied by the update advantage matrix to obtain the clipping product matrix M. The element in the tth row and bth column is equal to the product of the corresponding position elements of the clipped probability ratio and the update advantage matrix. Thus, the policy deviation degree and the balanced advantage signal are combined to form a specific quantitative index guiding the optimization of the policy, that is, the larger the product is, the more the adjustment direction of the current policy at the position conforms to the optimization demand indicated by the advantage signal. The policy loss is calculated by the clipping product matrix M, and the specific value is the negative value of the mean of all elements in M. This is because the policy benefit guided by the advantage needs to be maximized in reinforcement learning, and after taking the negative value, it can be realized by minimizing the loss. In order to further limit the deviation degree of the current policy and the reference policy and avoid policy mutation, a KL regularization term needs to be added. The KL regularization term is the KL divergence between the current policy and the reference policy multiplied by the KL coefficient. The larger the KL divergence is, the greater the difference between the two policies is, and the regularization term will punish such large difference. The total loss is obtained by adding the policy loss and the KL regularization term, which not only guarantees that the policy is optimized in the direction indicated by the advantage signal, but also constrains the stability of the policy change. The back propagation is performed with the total loss as the optimization target, that is, the gradient of the total loss with respect to each parameter of the model is calculated through the chain rule, and then the parameters are adjusted according to the preset learning rate. The learning rate controls the step size of parameter update, ensuring that each update is effective in approaching the optimal solution and does not cause oscillation due to too large step size. Through this step, the advantage signal after energy balancing is finally converted into the adjustment of the model parameters, realizing the stable optimization of the target model.

[0130] It should be noted that the current policy of the target model is determined by the current parameters of the model, denoted as π θ (where θ is the current parameter set of the target model). During the training process, the target model receives input (such as task instructions, dialogue history, etc.), and at each time step (corresponding to the time dimension T) and each sample batch (corresponding to the batch dimension B), the probability distribution of the possible output actions (such as generated tokens) is predicted based on the current parameters θ, that is, for each possible action, the probability value of being selected is output. These action probability distributions at different time steps and batches collectively constitute the current policy of the target model. The current policy is the action probability distribution rule generated by the target model under the current parameter θ state, and its essence is to infer the input through the current parameters of the model to output the action probability corresponding to each time step and each batch, and then form a quantitative representation of the policy.

[0131] The reference strategy is the strategy adopted by the target model in the training stage before the current optimization step, which remains unchanged in the current update cycle. For example, if the current is the model update stage of the nth iteration, the reference strategy is the strategy of the target model at the end of the (n-1)th iteration. The strategy does not participate in parameter adjustment in the current step, and is only used as a benchmark to construct the probability ratio matrix (quantifying the deviation of the current strategy from the benchmark) and the KL regularization term (constraining the difference between the current strategy and the benchmark), so as to avoid training instability caused by the current strategy due to the excessive update amplitude.

[0132] The above describes the embodiments of the present embodiment, but the present embodiment is not limited to the specific embodiments described above, which are only illustrative but not limiting. Those skilled in the art can make many forms under the inspiration of the present embodiment, which are all within the protection of the present embodiment.

Claims

1. A method for fine-tuning a large language model optimization based on reinforcement learning, characterized in that, Comprise: S1, collect the advantage value obtained by the target model interaction to form an advantage matrix with a size of TxB; wherein T represents the number of time dimensions, and B represents the number of batch dimensions; S2, according to the preset window, the advantage matrix is segmented in time dimension to form m pieces; S3, according to m pieces, the source distribution of the advantage matrix is determined, and the cost matrix is constructed; S4, based on the source distribution and the cost matrix, a transmission matrix with a size of m x m is determined; S5, based on the transmission matrix, the net energy carrying capacity of the pth piece is determined; S6, according to the net energy carrying capacity of the pth piece, the advantage matrix and the preset window, the advantage matrix is updated to obtain the updated advantage matrix; S7, based on the updated advantage matrix, the loss value is calculated, the loss value is used for back propagation, and the target model is updated.

2. The method of claim 1, wherein, According to the preset window, the advantage matrix is segmented in time dimension to form m pieces, comprising: determining the number m of fragments according to the length W of the preset window, Based on the number m of pieces, the advantage matrix is sequentially intercepted in time dimension to form m pieces; Wherein, when the time length of the mth piece is less than the length W of the preset window, the mth piece is filled with zero to fill in.

3. The method of claim 2, wherein, According to m pieces, the source distribution of the advantage matrix is determined, comprising: Compute the sum of squares of elements within the pth tile, resulting in the energy e of the pth tile p ; Based on the median of the energy of m pieces, m pieces are divided into high energy group and low energy group; Select the maximum energy e max in the high-energy group and the minimum energy e min in the low-energy group, respectively. If the pth slice is in the high-energy group, the corresponding update energy is: If the pth slice is a low-energy group, the corresponding update energy is: The energy sum value of the updated energy of m pieces is calculated; The ratio of the update energy of the p-th patch to the energy sum value is taken as the source distribution S p , p ∈ {1,..., m}.

4. The method of claim 3, wherein, The cost matrix is constructed, comprising: Initialize a zero matrix C with a size of m x m; The element of the pth row and the qth column of the zero matrix C is assigned as: C(p,q)=|p-q|W; If the check zero matrix C satisfies C(p,q)=C(q,p) and C(p,p)=0, the check is passed, and the zero matrix C is taken as the cost matrix.

5. The method of claim 4, wherein, Based on the source distribution and the cost matrix, a transmission matrix with a size of m x m is determined, comprising: Set the entropy regularization coefficient ε, and initialize the row dual variable u and the column dual variable v; wherein the row dual variable u and the column dual variable v are both m-dimensional full 1 vectors; An exponential matrix K of size mxm is constructed based on the cost matrix and a convergence threshold τ; wherein the element in the pth row and qth column of the exponential matrix is The p-th element u p The update process is performed: to the qth element of the column pair variable v p The update process is performed: where T q denotes the target distribution, After repeating updating the preset number of times for the row dual variable u and the column dual variable v, the updated row dual variable u(new) and the column dual variable v(new) are obtained respectively; Based on the updated row dual variable u(new) and column dual variable v(new), combined with the index matrix K, to synthesize a transmission matrix Π of size m x m; wherein the element of the pth row and the qth column of the transmission matrix Π(p,q) = u(new) p · K(p,q) · v(new) q .

6. The method of claim 5, wherein, Based on the transmission matrix, the net energy carrying capacity of the pth piece is determined, comprising: Determine the diagonal element (p,p) of the transmission matrix, then take the element of the pth row and the pth column of the transmission matrix as the self-loop component; Remove the self-loop component from the pth column of the transmission matrix, and calculate the sum value of the remaining elements in the pth column as the original total inflow; Remove the self-loop component from the pth row of the transmission matrix, and calculate the sum value of the remaining elements in the pth row as the original total outflow; Determine the maximum value in the remaining elements of the pth column as the maximum inflow component; Determine the maximum value in the remaining elements of the pth row as the maximum outflow component; Calculate the ratio of the maximum inflow component to the sum of the maximum inflow component and the maximum outflow component to obtain the first correction weight, and take the product of the first correction weight and the original total inflow as the total inflow; Calculate the ratio of the maximum outflow component to the sum of the maximum inflow component and the maximum outflow component to obtain the second correction weight, and take the product of the second correction weight and the original total outflow as the total outflow; The difference between the total inflow and the total outflow is taken as a net energy transport amount of the pth slice of the advantage matrix.

7. The method of claim 6, wherein, According to the net energy transport amount of the pth slice, the advantage matrix, and a preset window, the advantage matrix is updated to obtain an updated advantage matrix, including: The ratio of the net energy carrying capacity of the pth segment to the length W of the preset window is taken as the window increment a of the pth segment k ; Each element within the pth tile is added with the corresponding window increment a k to obtain the updated pth tile to form the updated dominance matrix A new .

8. The method of claim 7, wherein, A loss value is calculated based on the updated advantage matrix, and the loss value is used for back propagation to update the target model, including: Obtain the current policy of the target model θ and the reference policy ref and construct a probability ratio matrix R of size T x B; After the probability ratio matrix is clipped, the probability ratio matrix is multiplied by the updated advantage matrix, as follows: M(t,b) = clip(R(t,b), (1 - ε clip ), (1 + ε clip )) x A new (t,b) where M(t, b) denotes the element in the tth row and bth column of the clipped product matrix, and clip denotes the clipping threshold, A new (t, b) denotes the element in the tth row and bth column of the updated dominance matrix, and clip denotes a hyperbolic tangent clipping function. Compute strategy loss with cropped product matrix M and add regularization term to get total loss L total = L policy + L KL ; with total loss L total The back propagation is performed to update the hyperparameters of the target model according to a preset learning rate η.

Images