Aggregation method of importance sampling ratio, electronic equipment, medium and product

Abstract

The invention discloses an importance sampling ratio aggregation method, electronic equipment, a medium and a product, and relates to the technical field of reinforcement learning, and the method comprises the steps: firstly, under the same target input condition, obtaining first probability information of a first sequence outputted by a current task model, and second probability information of a second sequence output by the historical task model. In addition, a currently used target aggregation model is obtained; then, after the first probability information and the second probability information are compared, behavior change information of the current task model compared with the historical task model is determined; under the condition that the target aggregation model is the first aggregation model, the target aggregation model can be directly adopted to aggregate the importance sampling ratio of the at least one lexical element position included in the behavior change information to obtain a target aggregation value so as to calculate a loss value. In this way, in the training process, weak points of the task model can be repaired, and the success rate of task execution of the task model is increased.

Description

Technical Field

[0001] This application relates to the field of reinforcement learning technology, and in particular to methods, electronic devices, media, and products for aggregating importance sampling ratios. Background Technology

[0002] In training task models, the Group Relative Policy Optimization (GPRO) approach is commonly used to calculate the loss value and guide the training process. During loss calculation, for each input condition and its corresponding output sequence, the importance sampling ratio of each word position in the output sequence is typically calculated first. Then, based on the importance sampling ratio of each word position, the aggregate value of the output sequence is calculated. Finally, the loss value can be calculated based on the aggregate values ​​of the output sequences included in each group under the same output condition.

[0003] Currently, the arithmetic mean is generally used to aggregate the importance sampling ratios of each word position to generate the aggregated value of the output sequence. This makes the aggregated value susceptible to the influence of extremely large importance sampling ratios, while weakening the influence of extremely small importance sampling ratios on the aggregated value. This results in lower accuracy of the generated task model, and consequently, a lower success rate in task execution. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, storage medium, and program product for aggregating importance sampling ratios to address the problem of low success rate in task model execution.

[0005] This application provides a method for aggregating importance sampling ratios, including: Under the target input condition, obtain the first probability information of the first sequence output by the current task model, the second probability information of the second sequence output by the historical task model, and the target aggregation model, wherein the current task model is a task model generated by the historical task model after one or more rounds of training. Based on the first probability information and the second probability information, determine the behavior change information of the current task model compared with the historical task model, wherein the behavior change information includes the importance sampling ratio corresponding to at least one word position; When the target aggregation model is the first aggregation model, the target aggregation model is directly adopted to aggregate the importance sampling ratios corresponding to at least one word position to generate a target aggregation value, which is used to calculate the loss value. The first aggregation model is an aggregation model that improves the influence of the importance sampling ratio of candidate word positions on the target aggregation value. The candidate word position is a word position among at least one word position whose importance sampling ratio is less than a first threshold.

[0006] This application also provides an apparatus for aggregating importance sampling ratios, comprising: The acquisition module is used to acquire, under the target input condition, the first probability information of the first sequence output by the current task model, the second probability information of the second sequence output by the historical task model, and the target aggregation model, wherein the current task model is a task model generated by the historical task model after one or more rounds of training. The determination module is used to determine the behavior change information of the current task model compared with the historical task model based on the first probability information and the second probability information, wherein the behavior change information includes the importance sampling ratio corresponding to at least one word position; The aggregation module is used to directly adopt the target aggregation model when the target aggregation model is the first aggregation model, and aggregate the importance sampling ratios corresponding to at least one word position to generate a target aggregation value for calculating the loss value. The first aggregation model is an aggregation model that improves the influence of the importance sampling ratio of candidate word positions on the target aggregation value. The candidate word position is a word position among at least one word position whose importance sampling ratio is less than a first threshold.

[0007] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the aggregation method for any of the above-described importance sampling ratios.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the above-described method for aggregating any of the importance sampling ratios.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aggregation method for any of the above-described importance sampling ratios.

[0010] This application first obtains, for the same target input conditions, the first probability information of the first sequence output by the current task model and the second probability information of the second sequence output by the historical task model. Then, it obtains the currently used target aggregation model. Next, by comparing the first and second probability information, it determines the behavioral change information of the current task model compared to the historical task model. Since candidate word positions are word positions with an importance sampling ratio less than a first threshold, i.e., word positions with extremely high behavioral change, these candidate word positions are weak points of the current task model. The first aggregation model is an aggregation model that can improve the influence of the importance sampling ratio of candidate word positions on the target aggregation value. Therefore, when the target aggregation model is the first aggregation model, it can be directly used to aggregate the importance sampling ratio of at least one word position included in the behavioral change information to obtain the target aggregation value, which is then used to calculate the loss value. This can increase the contribution of the importance sampling ratio of candidate word positions to the target aggregation value. Furthermore, by using the target aggregation value to calculate the loss value, the importance sampling ratio indicated by the candidate word positions can have a greater impact on the training process. In other words, during training, the focus can be placed on word positions that the task model is relatively weak in, so that the weaknesses of the task model can be repaired during training, thereby improving the accuracy of the task model and the success rate of task execution. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a method for aggregating importance sampling ratios provided in an embodiment of this application; Figure 2 A flowchart illustrating a training task model provided in an embodiment of this application; Figure 3 A schematic flowchart of an importance sampling ratio aggregation device provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] In recent years, Large Language Models (LLMs) have achieved breakthroughs in tasks such as general question answering, code generation, mathematical reasoning, and process planning. The training paradigm has also shifted from Supervised Fine-Tuning (SFT) to Reinforcement Learning from Human Feedback (RLHF), optimizing policies to make the model output more aligned with task objectives and safety standards. A Large Language Model is a deep learning model trained on massive amounts of text data, employing a "Transformer" architecture, and capable of understanding and generating human language. In long-chain reasoning scenarios, it often exhibits a vulnerability of "one mistake, the whole system fails." For example, errors in a few key steps (such as intermediate lemmas, boundary conditions, and variable scope switching) will be amplified in subsequent steps, leading to overall failure.

[0017] During the training of the task model (i.e., the large language model mentioned above), it is generally necessary to calculate the importance sampling ratio of each word position in each output sequence. For each output sequence, it is necessary to aggregate the importance sampling ratios of each word position included in the output sequence to obtain the aggregated value. Then, the loss value is calculated using the pre-constructed loss function and the aggregated value of each output sequence.

[0018] However, for the output sequence, some word positions have extremely high importance sampling ratios, while others have extremely low importance sampling ratios. In the aggregation process, directly using the arithmetic mean formula to process all word positions in a uniform manner will lead to a bias problem of "the strong getting stronger and the weak being ignored". This makes the training process increasingly cater to "easy steps", forming a task model that is superficially convergent but essentially fragile. That is, it is more accurate for the positions that the task model can accurately handle, but no further improvement is made for the positions that the task model cannot accurately handle.

[0019] Especially in long-link inference scenarios, word positions with extremely low importance sampling ratios are more likely to appear in the output sequence. If the arithmetic mean formula is used to perform aggregation operations, it will lead to a low success rate of the task model in performing the task.

[0020] In addition, while using the geometric mean formula to calculate the aggregate value can suppress the contribution of extremely large importance sampling ratios to the aggregate value, it still does not adequately emphasize extremely small importance sampling ratios, which will also lead to the problem of low task success rate of the task model.

[0021] To address the aforementioned technical problems, embodiments of this application provide a method for aggregating importance sampling ratios, which can be executed by a computer device (such as a server, computer, etc.). Figure 1 As shown, the specific processing steps of the importance sampling ratio aggregation method may include: Step S101: Under the target input conditions, obtain the first probability information of the first sequence output by the current task model, the second probability information of the second sequence output by the historical task model, and the target aggregation model.

[0022] The current task model can be a task model generated after one or more rounds of training from a historical task model. For example, the task model can be a large language model applied to specific fields such as mathematical reasoning, image analysis, and code generation.

[0023] The first sequence can be the sequence output by the current task model after performing inference operations under the target input conditions. The second sequence can be the sequence output by the historical task model after performing inference operations under the target input conditions. Both the first and second sequences each include at least one token. The number of tokens included in the first and second sequences can be the same or different. All tokens included in the first and second sequences can be the same, or some tokens can be different.

[0024] The first probability information may include the probability value of at least one word position in the first sequence, and the second probability information may include the probability value of at least one word position in the second sequence. Taking the target word position (any word position) as an example, if the word at the target word position is A, the probability value represents the probability that the target word position is A. For example, the probability value can be expressed as... That is, given the input condition is q, and the words preceding the t-th word position are already determined, the word at the current t-th word position is... The probability value.

[0025] The target aggregation model can be a first aggregation model or a second aggregation model, etc. The first aggregation model is an aggregation model that increases the influence of the importance sampling ratio of candidate word positions on the target aggregation value during the aggregation process. Conversely, the second aggregation model is an aggregation model that cannot increase the influence of the importance sampling ratio of candidate word positions on the target aggregation value. For example, the first aggregation model can be a harmonic average formula, a weighted formula, etc., while the second aggregation model can be an arithmetic average formula, a geometric average formula, etc.

[0026] The harmonic mean formula can be expressed as follows: (1) Where i is the group index, j is the sequence index, and HM is the aggregate value of the i-th sequence in the j-th group. Let be the number of tokens in the i-th sequence of the j-th group, and t be the index of the token. The importance sampling ratio of the t-th word in the i-th sequence.

[0027] The formula for the arithmetic mean can be as follows: (2) Where AM is the aggregate value of the i-th sequence in the j-th group.

[0028] The geometric mean formula can be expressed as follows: (3) Wherein, GM is the aggregate value of the i-th sequence in the j-th group.

[0029] Specifically, during the training of a task model for a certain task type, the computer device needs to calculate the loss value in each training round and use the loss value as a reference to update the model parameters in the task model until the loss value meets a preset stopping condition or the number of training iterations reaches a preset threshold, at which point training stops. During the loss value calculation, the computer device needs to analyze the relevant information output by the task model in the current round and the task model in the previous round. Furthermore, during training, the sample data can be divided into multiple groups, each group corresponding to an input condition. The target input condition can be the input condition corresponding to any one of the multiple groups. Under the same input condition, the task model can output multiple sequences. When the task model is the current task model, the first sequence can be any one of the multiple sequences output by the current task model based on the target input condition. When the task model is a historical task model, the second sequence can be any one of the multiple sequences output by the historical task model based on the target input condition.

[0030] In the current training round, the current task model can infer based on the target input conditions and output the first sequence and the first probability information of the first sequence. Similarly, the historical task model outputs the second sequence and the second probability information of the second sequence based on the same target input conditions in the previous training round.

[0031] In addition, the computer device can also acquire a target aggregation model for performing aggregation operations on importance sampling ratios.

[0032] Step S102: Based on the first probability information and the second probability information, determine the behavioral change information of the current task model compared with the historical task model.

[0033] The behavioral change information may include the importance sampling ratio corresponding to at least one lexical position. The importance sampling ratio is a scalar value used to quantify the relative difference in the probability of the current task model and the historical task model performing a specific action under the same conditions.

[0034] Specifically, the computer device can compare the first probability information and the second probability information to determine the change in behavior of the current task model compared to the historical task model when processing the same target input conditions, that is, to determine the behavior change information. Specifically, since the lengths of the first sequence and the second sequence can be the same or different, meaning the number of probability values ​​included in the first probability information and the second probability information are different, when the lengths of the first sequence and the second sequence are different, the computer device can pad the word positions of the shorter sequence and provide the probability value of the padded word positions, making the number of probability values ​​in the first sequence and the second sequence the same and comparable. After comparison, the importance sampling ratio of each word position can be obtained and constructed as behavior change information.

[0035] Step S103: If the target aggregation model is the first aggregation model, the target aggregation model is directly adopted to aggregate the importance sampling ratios corresponding to at least one word position to generate a target aggregation value, which is used to calculate the loss value.

[0036] Here, candidate lexical positions are lexical positions whose importance sampling ratio is less than a first threshold among at least one lexical positions. The first threshold can be preset by an expert or determined based on behavioral change information. For example, lexical positions with an importance sampling ratio less than the first threshold can generally be rare symbols, structural symbols, key values, etc.

[0037] Specifically, a smaller importance sampling ratio indicates a significant change in the current task model's handling of candidate word positions compared to historical task models. During training, the focus should be on positions with lower importance sampling ratios, directing optimization towards addressing the model's weaknesses. The target aggregation model can be a first aggregation model, a second aggregation model, etc. The first aggregation model can amplify the impact of candidate word positions on the training process, while the second aggregation model cannot achieve this effect. Furthermore, the aforementioned target aggregation model can be a pre-selected aggregation model by technicians or an automatically selected aggregation model by the computer based on the actual training situation. Therefore, to ensure that optimization consistently addresses the model's weaknesses under any circumstances, an aggregation method corresponding to the selected aggregation model can be used to aggregate the importance sampling ratios within the sequence.

[0038] Accordingly, when the target aggregation model is the first aggregation model, the computer device can directly adopt the target aggregation model to aggregate the importance sampling ratios corresponding to at least one word position, generating a target aggregation value. For example, the harmonic mean formula mentioned above can be directly used to aggregate the importance sampling ratios corresponding to at least one word position, generating a target aggregation value. Alternatively, when using a weighted formula, a first weight value is assigned to candidate word positions, and a second weight value is assigned to non-candidate word positions, with the first weight value being greater than the second weight value. Then, the importance sampling ratios of each word position are aggregated.

[0039] The importance sampling ratio aggregation method of this application firstly obtains, for the same target input conditions, the first probability information of the first sequence output by the current task model and the second probability information of the second sequence output by the historical task model. Then, the currently used target aggregation model is obtained. Next, after comparing the first and second probability information, the behavioral change information of the current task model compared to the historical task model is determined. Since candidate word positions are word positions with an importance sampling ratio less than a first threshold, i.e., word positions with extremely high behavioral change, these candidate word positions are weak points of the current task model. The first aggregation model is an aggregation model that can improve the influence of the importance sampling ratio of candidate word positions on the target aggregation value. Therefore, when the target aggregation model is the first aggregation model, the target aggregation model can be directly used to aggregate the importance sampling ratio of at least one word position included in the behavioral change information to obtain the target aggregation value, which is then used to calculate the loss value. This approach increases the contribution of the importance sampling ratio of candidate lexical positions to the target aggregation value. Furthermore, using the target aggregation value to calculate the loss value allows the importance sampling ratio indicated by the candidate lexical positions to have a greater impact on the training process. This means that during training, the focus can be placed on lexical positions where the task model is relatively weak, thus addressing weaknesses in the task model and improving its accuracy and success rate. Especially in long-link inference scenarios, extremely small importance sampling ratios appear more frequently in the output sequence. Therefore, performing aggregation operations through the first aggregation model can effectively increase the impact of these small importance sampling ratios on the aggregation value, thereby significantly improving the task model's success rate. Specifically, for image analysis tasks, this improves the accuracy of the analysis results; for code generation tasks, it improves the accuracy of the code generation results.

[0040] For example, when using the harmonic mean formula to calculate the aggregated value, the harmonic mean formula is insensitive to the importance sampling ratio of the largest sample, which can effectively suppress training oscillations caused by a small number of abnormally high ratio samples. Under the same learning rate and batch size, the gradient variance after aggregation using the harmonic mean formula can be reduced by about 25%–40% compared to the calculated average, thus making the training process smoother and the convergence speed faster.

[0041] In some alternative implementations, when the target aggregation model is a second aggregation model, the computer device may also determine the target aggregation value according to the following specific steps: Step 1: Based on the behavior change information, determine that at least one word position contains a word position to be adjusted, and then adjust the importance sampling ratio of the word position to be adjusted based on the preset adjustment method.

[0042] Step 2: After completing the adjustment of the importance sampling ratio of all the word positions to be adjusted, the target aggregation model is used to aggregate the importance sampling ratio corresponding to at least one word position to generate the target aggregation value.

[0043] Specifically, the computer device can compare the importance sampling ratios of word positions in the behavior change information to determine whether the importance sampling ratios in the behavior change information are stable, and then determine whether there are any word positions to be adjusted. Then, if it is determined that there are word positions to be adjusted, the importance sampling ratios of the word positions to be adjusted can be adjusted respectively based on a preset adjustment method, so that the importance sampling ratios within the sequence tend to be stable. After completing the adjustment operations of all word positions to be adjusted, the target aggregation model is then used to aggregate the importance sampling ratios corresponding to all word positions within the sequence to generate a target aggregation value. For example, the target aggregation value can be obtained by using the above formula (2) or formula (3).

[0044] Thus, given that the target aggregation model is the second aggregation model, adjusting the importance sampling ratio of the word positions to be adjusted in advance can increase the contribution of the importance sampling ratio of the candidate word positions to the target aggregation value. Furthermore, using the target aggregation value to calculate the loss value can make the importance sampling ratio indicated by the candidate word positions have a greater impact on the training process, repair the weaknesses of the task model, and improve the stability and success rate of the task model.

[0045] In some alternative implementations, if it is determined that there are no lexical positions to be adjusted, a target aggregation model can be directly used to aggregate the importance sampling ratios corresponding to at least one lexical position to generate a target aggregation value.

[0046] Since there are no lexical positions to be adjusted, it indicates that the current task model's processing changes at different lexical positions are consistent with those of the historical task model, and the changes are neither too large nor too small. Therefore, the aggregation operation is directly performed to obtain the target aggregation value, and the loss value is calculated using this target aggregation value. The impact of each lexical position on the loss value is not significantly different. Thus, by pre-determining the type of the target aggregation model, and, in the case of a second aggregation model, determining whether there are any lexical positions to be adjusted, rather than making comprehensive adjustments, resources can be saved.

[0047] In some optional implementations, step S202 described above may specifically include the following two methods: Method 1: Based on the first probability value of the first lexical position in the first probability information and the second probability value in the second probability information, determine the first importance sampling ratio of the first lexical position, where the first lexical position is any one of at least one lexical position. After determining the importance sampling ratios corresponding to each of the at least one lexical position, generate behavioral change information based on the importance sampling ratios corresponding to each of the at least one lexical position.

[0048] Specifically, taking the first lexical position as an example, the ratio of the first probability value to the second probability value at the first lexical position can be determined as the first importance sampling ratio. For example, the following expression can be used: (4) in, The first importance sampling ratio, The first probability value, This is the second probability value.

[0049] For each lexical position, the probability ratio can be calculated in the manner described above. Finally, the probability ratios of all lexical positions can be used to determine the behavioral change information.

[0050] Method Two: Based on the third probability value of the second lexical position in the first probability information and the fourth probability value in the second probability information, determine the second importance sampling ratio for the second lexical position, where the second lexical position is any one of at least one lexical position. Convert the form of the second importance sampling ratio to logarithmic form. After determining the form conversion operation of the importance sampling ratio corresponding to each of the at least one lexical position, generate behavioral change information based on the logarithmic importance sampling ratio corresponding to each of the at least one lexical position.

[0051] Specifically, taking the second lexical position as an example, the ratio of the third probability value to the fourth probability value of the second lexical position can be determined as the second importance sampling ratio, specifically using the formula (4) mentioned above. Then, the computer device can further calculate the logarithmic value of the second importance sampling ratio, that is, convert the form of the second importance sampling ratio into logarithmic form. For each lexical position, the importance sampling ratio can be calculated in the above manner, and after performing the form conversion operation, the importance sampling ratio in logarithmic form of all lexical positions can be determined as behavioral change information.

[0052] In this way, by calculating the logarithmic value, the importance sampling ratio within the sequence can be stabilized. Furthermore, by using the logarithmic value to calculate the loss value, the influence of outliers on the target aggregation value can be reduced, and the stability of candidate word positions on the target aggregation value can be improved. Thus, during the training process, the weaknesses of the task model can be identified and repaired, ultimately improving the stability of the task model.

[0053] In some alternative implementations, since the importance sampling ratio in the behavior change information has already stabilized after the computer device generates the behavior change information using method two, in subsequent processes, if the target aggregation model is determined to be the second aggregation model, there is no need to identify the position of the word to be adjusted, nor is there any need to perform adjustment operations. The second aggregation model can be directly used to aggregate the importance sampling ratio included in the behavior change information.

[0054] In this way, resources can be saved while ensuring that the training process can focus on the weaknesses of the task model.

[0055] In some optional implementations, where the first and second thresholds in step one above are determined based on behavioral change information, the computer device can determine the position of the word to be adjusted using the following specific steps: Step 1: Statistically analyze the distribution information of the importance sampling ratio of at least one word position.

[0056] Step 2: Obtain the target quantile range.

[0057] Step 3: Based on the distribution information and the target quantile range, determine the first threshold and the second threshold respectively.

[0058] Step 4: Determine the word positions whose importance sampling ratio is less than the first threshold or greater than the second threshold as word positions to be adjusted.

[0059] The second threshold is greater than the first threshold, and the position of the word to be adjusted may include the position of the candidate word.

[0060] Specifically, the computer device can statistically analyze the distribution information of the importance sampling ratios of word positions in behavioral change information and obtain the target quantile range. The target quantile range can include an upper quantile and a lower quantile; for example, the target quantile range can be a pre-specified [10%, 90%]. Then, the computer device can determine a second threshold based on the upper quantile (e.g., 90%) and the distribution information (which can indicate the position of a certain importance sampling ratio within at least one importance sampling ratio), and determine a first threshold based on the lower quantile (e.g., 10%). Finally, the computer device can compare the importance sampling ratio of each word position with the first threshold and the second threshold respectively, and select the word positions to be adjusted whose importance sampling ratio is less than or greater than the second threshold.

[0061] Accordingly, the aforementioned preset adjustment method may include increasing the importance sampling ratio of word positions with an importance sampling ratio less than a first threshold by a first preset value, and decreasing the importance sampling ratio of word positions with an importance sampling ratio greater than a second threshold by a second preset value.

[0062] In this way, for different sequences, the first and second thresholds can be determined based on the distribution of importance sampling ratios within the sequence, providing greater flexibility. Furthermore, this ensures that after adjusting the positions of the words to be adjusted in different sequences, the contribution of candidate word positions to the aggregation value is increased. This allows the training process to focus on and correct weaknesses in the task model, significantly improving its accuracy.

[0063] In some optional implementations, in step S103 above, the computer device can use the harmonic mean formula described above to aggregate only the importance sampling ratios of candidate word positions that are less than a first threshold to obtain a target aggregated value. This can further enhance the attention paid to the weaknesses of the task model during training, thereby improving the accuracy of the task model.

[0064] Alternatively, the computer equipment can increase the importance sampling ratios below the first threshold to the first threshold, and then use a harmonic average formula to aggregate the importance sampling ratios of all word positions in the first sequence to generate a target aggregated value. This allows for comprehensive utilization of all information from the first training iteration, balancing attention to the weaknesses of the task model with noise resistance.

[0065] In some alternative implementations, in step 2 above, the computer device may obtain the target quantile range using the following specific steps: Step a: Obtain the target task type of the current task model, as well as the quantile ranges corresponding to various task types.

[0066] Step b: Based on the task type, determine the target quantile range corresponding to the target task type from the quantile ranges corresponding to the various task types.

[0067] Specifically, the aforementioned target quantile range can be pre-specified by technical personnel, or it can be determined based on the target task type of the current task model. Specifically, the computer device can first identify the target task type of the current task model; for example, the target task type could be a code model, a multimodal model, etc. Furthermore, the computer device can pre-set quantile ranges corresponding to various task types. After identifying the target task type, a target quantile range matching the target task type can be determined from the quantile ranges corresponding to the various task types.

[0068] In this way, the selection of the quantile range for task types based on the task model becomes more flexible.

[0069] In some alternative implementations, the computer device may also select a quantile range corresponding to the length of the first sequence as the target quantile range. Since sequence length is a significant factor influencing the aggregate value, selecting the quantile range based on the sequence length better reflects the characteristics of the sequence, resulting in more accurate aggregate value calculations and consequently improving the accuracy of the task model.

[0070] In some optional implementations, obtaining the target aggregation model in step S101 above may include the following specific steps: Step 1: Obtain the initial aggregation model and historical loss value information.

[0071] Step 2: Based on historical loss value information, adjust the values ​​of the target parameters in the initial aggregation model to obtain the target aggregation model.

[0072] Specifically, after continuous iterative training, the training state of the task model will constantly change, and the loss value will gradually stabilize. To improve training efficiency and accuracy, the computer device can record the loss value in each training epoch and use the recorded loss value to select an appropriate aggregation model to aggregate the importance sampling ratio. For example, the aggregation model can be reselected every preset number of training epochs.

[0073] When the computer device determines that the current training epoch is an integer multiple of a preset number, it obtains the loss values ​​corresponding to the preset number of historical training epochs preceding the current training epoch. Then, based on the loss values ​​corresponding to the historical training epochs, it generates historical loss value information for the current task model. Furthermore, the computer device can adjust the values ​​of the target parameters in the initial aggregation model based on the historical loss value information to obtain the target aggregation model.

[0074] In this way, by adjusting the initial aggregation model using historical loss value information, a target aggregation model suitable for the current training needs can be obtained, which is relatively simple. Furthermore, using a suitable aggregation model to aggregate the importance sampling ratio offers high flexibility, ensuring that the training process proceeds in a more accurate direction, resulting in a more accurate task model and improving the success rate of the task model in performing the task.

[0075] For example, the initial aggregation model can use the following power-mean formula, as follows: (5) Where i is the group index and j is the sequence index. Let be the aggregate value of the i-th sequence in the j-th group. Let be the number of tokens in the i-th sequence of the j-th group, and t be the index of the token. Let p be the importance sampling ratio of the t-th word in the i-th sequence, and p be the target parameter.

[0076] In some alternative implementations, in step two above, the computer device may adjust the values ​​of the target parameters of the initial aggregation model using the following specific steps: Step 1: Based on the loss values ​​corresponding to multiple training rounds, identify the target evolution characteristics of the loss values.

[0077] Step 2: Based on the target evolution characteristics, obtain the target preset values ​​corresponding to the target evolution characteristics.

[0078] Step 3: Adjust the values ​​of the target parameters in the initial aggregation model to the target preset values ​​to obtain the target aggregation model.

[0079] Specifically, in step 1, the computer device extracts the target evolution features of the loss values ​​from the loss values ​​corresponding to multiple training epochs. For example, the loss values ​​corresponding to multiple training epochs can be input into a feature extraction model to obtain the loss value evolution features output by the feature extraction model. Alternatively, the difference between the loss values ​​of every two adjacent training epochs can be calculated first. If one or more differences are greater than a preset difference threshold, the first evolution feature can be determined as the loss value evolution feature. If all differences are less than or equal to the preset difference threshold, the second evolution feature can be determined as the loss value evolution feature. The first evolution feature is used to indicate that the loss value fluctuates greatly, and the second evolution feature is used to indicate that the loss value tends to stabilize.

[0080] In step 2, the computer device can pre-create preset values ​​corresponding to various evolutionary features. Accordingly, after identifying the target evolutionary feature, the computer device can determine the target preset value that matches the target evolutionary feature based on the preset values ​​corresponding to the various evolutionary features.

[0081] In step 3, the computer device can set the value of the target parameter in the initial aggregation model to the aforementioned target preset value to obtain the target aggregation model.

[0082] For example, if the target evolutionary feature is the first evolutionary feature, the value of the target parameter can be set to -1; if the target evolutionary feature is the second evolutionary feature, the value of the target parameter can be set to 0.

[0083] Thus, when the training process is unstable, using the arithmetic mean formula to calculate the aggregated value can prevent the amplification of noise. When the training process is stable, using the harmonic mean formula allows the training process to target the weaknesses of the task model. Using an appropriate aggregation model to aggregate the importance sampling ratio under different training conditions can help train a more accurate task model and significantly improve the success rate of the task model in performing tasks.

[0084] In some alternative implementations, the computer device may also calculate the loss value using the following specific steps: Step 1: After obtaining the aggregate values ​​and initial behavioral advantage scores corresponding to the multiple sequences output by the current task model, determine whether the first sequence conforms to the sequence feature analysis rules based on the first sequence and the pre-built sequence feature analysis rules.

[0085] The first sequence can be any of multiple sequences.

[0086] Step 2: If it is determined that the first sequence does not meet the sequence feature analysis rules, the initial behavioral advantage score of the first sequence is directly determined as the final behavioral advantage score of the first sequence.

[0087] Alternatively, in step three, if the first sequence is determined to conform to the sequence feature analysis rules, obtain the increment value corresponding to the sequence feature analysis rules.

[0088] Step 4: Based on the added value and the initial behavioral advantage score of the first sequence, determine the final behavioral advantage score of the first sequence.

[0089] Step 5: After determining the final behavioral advantage scores corresponding to multiple sequences, determine the loss value based on the pre-constructed loss function, the final behavioral advantage scores corresponding to multiple sequences, and the aggregate value.

[0090] Specifically, in step one, a large amount of sample data is required to train the task model in each training round. The current task model can output a large number of sequences during training, with the first sequence being one of them. Additionally, the computer device can use a pre-constructed advantage function to score sequences within the same group, obtaining initial behavioral advantage scores for multiple sequences. To further improve the accuracy of the scoring, the computer device can introduce additional scoring rules, namely sequence feature analysis rules. Then, based on the sequence feature analysis rules, it can be determined whether each sequence conforms to the sequence feature analysis rules.

[0091] In step two, taking the first sequence as an example, if it is determined that the first sequence does not meet the sequence feature analysis rules, the initial behavioral advantage score of the first sequence can be directly determined as the final behavioral advantage score of the first sequence.

[0092] In step three, if it is determined that the first sequence conforms to the sequence feature analysis rules, a score can be added to the first sequence. Accordingly, the computer device can obtain the added value corresponding to the sequence feature analysis rules. For example, the sequence feature analysis rules can be that the sequence format is a preset format, such as "JSON".

[0093] In step four, the computer device can add the initial behavioral advantage score of the first sequence to the determined increment value to obtain the final behavioral advantage score of the first sequence.

[0094] In step five, after determining the final behavioral advantage scores for each of the multiple sequences, the loss value can be determined using a pre-constructed loss function, the final behavioral advantage scores for each sequence, and the aggregated value. For example, for each sequence, the product of the final behavioral advantage score and the aggregated value for that sequence can be used to determine the sub-loss value for that sequence. Thus, after determining the sub-loss values ​​for each of the multiple sequences, the total loss value can be determined based on these sub-loss values.

[0095] In this way, by flexibly setting sequence feature analysis rules and adjusting the initial behavioral advantage score, the aggregated value can be used to further determine the optimization direction to repair the weaknesses of the task model, making the finally trained task model more accurate. For example, task models generally perform tasks in a specific format. If the format is a preset format, rewards can be used to guide the training process towards greater accuracy.

[0096] For example, the loss function can be expressed as follows: (6) in, Here, m represents the number of groups, and i represents the index of the group. Let represent the number of sequences included in the i-th group, and j represent the index of the sequence. Let be the number of tokens included in the j-th sequence within the i-th group. This represents the importance sampling ratio of the t-th word position in the j-th sequence of the i-th group. Score the final behavioral advantage for the j-th sequence in the i-th group. The preset cropping threshold, This is the weighting coefficient for the KL item, which can be adjusted according to the task type. This represents the behavior distribution of the current task model under the target input condition q. Behavior distribution compared to pre-built reference task models The degree of difference between them.

[0097] In some optional implementations, the computer device can also identify the lexical type corresponding to at least one lexical position in the first sequence. If it is determined that the lexical type at all lexical positions in the first sequence is the first lexical type, the target aggregation value can be directly calculated using the second aggregation model without adjustment. Alternatively, if it is determined that one or more lexical positions in the first sequence have the second lexical type, the target aggregation value can be directly calculated using the first aggregation model. Or, after determining the lexical position to be adjusted, the target aggregation value can be calculated using the second aggregation model. The first lexical type includes common content words (e.g., scene nouns, verbs, adjectives in non-professional fields), grammatical filler words (common function words, prepositions, conjunctions, pronouns, etc. in language), and high-frequency words (words that mainly contribute grammatical functions rather than actual meaning). The second lexical type includes rare symbols, structural symbols, and key numerical values. In this way, since the second word type is the word type of the word position with a very small importance sampling ratio, by pre-judging the word type of the word position in the sequence, we can switch to the first aggregation model only when the second word type exists, or adjust the importance sampling ratio of the word position to be adjusted. This can save resources and ensure the accuracy of the task model.

[0098] In some alternative implementations, for different output sequences, the computer device can adjust the weight values ​​in the loss function based on the number of word positions in the output sequence. For example, the above formula (6) can also be transformed into: (7) Accordingly, computer equipment can [address] it. Make adjustments. It can be positively correlated with the number of lexical positions. Thus, in the case of long sequences, increasing the weight of the aggregation term can further enhance the impact of the aggregation value on the loss value, and further improve the accuracy of the task model.

[0099] refer to Figure 2 First, in the current training round, the computer device can acquire sample data and prompts, the current task model and historical task models, as well as task configuration rules. For example, task configuration rules may include the sequence feature analysis rules and power average formulas mentioned above.

[0100] Then, during the training process of the current task model, the computer equipment can calculate the importance sampling ratio and initial dominant behavior score for each word position in each output sequence. Furthermore, to reduce the distribution width of the importance sampling ratio, it can first be converted to logarithmic form. Additionally, according to sequence feature analysis rules, it can be determined whether to adjust the initial dominant behavior score to obtain the final dominant behavior score.

[0101] Furthermore, the computer device can use the above harmonic mean formula to calculate the aggregate value, and then use the loss function to calculate the loss value to update the current task model and obtain a new task model, which can be used to calculate the importance sampling ratio in the next training round.

[0102] After calculating the aggregate value using the above scheme, the computer device can use the length-based bucket win rate to determine whether the win rate of the long sequence bucket has been significantly improved, that is, to determine whether the success rate of the task model has been improved.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0104] Embodiments of this application also provide an apparatus for aggregating importance sampling ratios, such as... Figure 3 As shown, it includes: The acquisition module 310 is used to acquire, under the target input condition, the first probability information of the first sequence output by the current task model, the second probability information of the second sequence output by the historical task model, and the target aggregation model, wherein the current task model is a task model generated by the historical task model after one or more rounds of training. The determination module 320 is used to determine the behavior change information of the current task model compared with the historical task model based on the first probability information and the second probability information, wherein the behavior change information includes the importance sampling ratio corresponding to at least one word position; The aggregation module 330 is used to directly adopt the target aggregation model when the target aggregation model is the first aggregation model, and aggregate the importance sampling ratios corresponding to at least one word position to generate a target aggregation value for calculating the loss value. The first aggregation model is an aggregation model that improves the influence of the importance sampling ratio of candidate word positions on the target aggregation value. The candidate word position is a word position among at least one word position whose importance sampling ratio is less than a first threshold.

[0105] In some alternative implementations, the aggregation module 330 is further configured to: After determining that at least one word position has a word position to be adjusted based on the behavior change information, the importance sampling ratio of the word position to be adjusted is adjusted based on the preset adjustment method. The second aggregation model is an aggregation model that cannot improve the influence of the importance sampling ratio of the candidate word position on the target aggregation value. After adjusting the importance sampling ratios of all the lexical positions to be adjusted, the target aggregation model is used to aggregate the importance sampling ratios corresponding to at least one lexical position to generate the target aggregation value.

[0106] In some alternative implementations, the determining module 320 is specifically used for: Based on the first probability value of the first word position in the first probability information and the second probability value in the second probability information, the first importance sampling ratio of the first word position is determined, wherein the first word position is any one of at least one word position. After determining the importance sampling ratio corresponding to at least one word position, behavioral change information is generated based on the importance sampling ratio corresponding to at least one word position.

[0107] In some alternative implementations, the determining module 320 is specifically used for: Based on the third probability value of the second word position in the first probability information and the fourth probability value in the second probability information, the second importance sampling ratio of the second word position is determined, wherein the second word position is any word position among at least one word position; Convert the form of the second importance sampling ratio into logarithmic form; After determining the formal transformation operation of the importance sampling ratio corresponding to at least one word position, behavioral change information is generated based on the importance sampling ratio in logarithmic form corresponding to at least one word position.

[0108] In some alternative implementations, the aggregation module 330 is specifically used for: Statistically analyze the distribution information of the importance sampling ratio of at least one word position; Obtain the target quantile range; Based on the distribution information and the target quantile range, a first threshold and a second threshold are determined respectively, wherein the second threshold is greater than the first threshold; At least one lexical position is identified as a lexical position to be adjusted if its importance sampling ratio is less than a first threshold or greater than a second threshold. The lexical positions to be adjusted include candidate lexical positions.

[0109] In some alternative implementations, the aggregation module 330 is specifically used for: Obtain the target task type of the current task model, as well as the quantile ranges corresponding to various task types; Based on the task type, the target quantile range corresponding to the target task type is determined from the quantile ranges corresponding to various task types.

[0110] In some optional implementations, the preset adjustment methods include: Increase the importance sampling ratio of word positions whose importance sampling ratio is less than the first threshold by a first preset value; Additionally, the importance sampling ratio of word positions whose importance sampling ratio is greater than the second threshold is reduced by a second preset value.

[0111] In some optional implementations, the acquisition module 310 is specifically used for: Obtain initial aggregation model and historical loss value information; Based on historical loss values, the values ​​of the target parameters in the initial aggregation model are adjusted to obtain the target aggregation model.

[0112] In some optional implementations, the historical loss value information includes loss values ​​corresponding to multiple training epochs; the acquisition module 310 is specifically used for: Based on the loss values ​​corresponding to multiple training rounds, the target evolution characteristics of the loss values ​​are identified. Based on the target evolution characteristics, obtain the target preset values ​​corresponding to the target evolution characteristics; After adjusting the values ​​of the target parameters in the initial aggregation model to the target preset values, the target aggregation model is obtained.

[0113] In some alternative implementations, the determining module 320 is further configured to: After obtaining the aggregate values ​​and initial behavioral advantage scores corresponding to the multiple sequences output by the current task model, it is determined whether the first sequence conforms to the sequence feature analysis rules based on the first sequence and the pre-built sequence feature analysis rules, where the first sequence is any one of the multiple sequences; If it is determined that the first sequence does not meet the sequence feature analysis rules, the initial behavioral advantage score of the first sequence is directly determined as the final behavioral advantage score of the first sequence. Alternatively, if it is determined that the first sequence conforms to the sequence feature analysis rules, the increment value corresponding to the sequence feature analysis rules can be obtained; Based on the added value and the initial behavioral advantage score of the first sequence, the final behavioral advantage score of the first sequence is determined. After determining the final behavioral advantage scores corresponding to multiple sequences, the loss value is determined based on the pre-constructed loss function, the final behavioral advantage scores corresponding to multiple sequences, and the aggregate value.

[0114] In some optional implementations, the initial aggregation model is expressed as follows:

[0115] Where i is the group index and j is the sequence index. Let be the aggregate value of the i-th sequence in the j-th group. Let be the number of tokens in the i-th sequence of the j-th group, and t be the index of the token. Let p be the importance sampling ratio of the t-th word in the i-th sequence, and p be the target parameter.

[0116] In some alternative implementations, the first aggregation model adopts the following expression:

[0117] Where i is the group index, j is the sequence index, and HM is the aggregate value of the i-th sequence in the j-th group. Let be the number of tokens in the i-th sequence of the j-th group, and t be the index of the token. The importance sampling ratio of the t-th word in the i-th sequence.

[0118] For a description of the features in the embodiment corresponding to the importance sampling ratio aggregation device, please refer to the relevant description in the embodiment corresponding to the importance sampling ratio aggregation method, which will not be repeated here.

[0119] Embodiments of this application also provide an electronic device, such as... Figure 4 As shown, it includes a memory 10 and a processor 20, the memory 10 storing a computer program, and the processor 20 being configured to run the computer program to perform the steps in the above embodiments of the aggregation method for any of the importance sampling ratios.

[0120] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the above embodiments of the aggregation method for any of the importance sampling ratios.

[0121] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0122] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the above embodiments of the aggregation method for any of the importance sampling ratios.

[0123] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the above embodiments of the aggregation method for any of the importance sampling ratios.

[0124] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0125] The foregoing has provided a detailed description of the method, apparatus, electronic device, storage medium, and program product for aggregation of importance sampling ratios provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for aggregating importance sampling ratios, characterized in that, include: Under the target input condition, the first probability information of the first sequence output by the current task model, the second probability information of the second sequence output by the historical task model, and the target aggregation model are obtained, wherein the current task model is a task model generated by the historical task model after one or more rounds of training. Based on the first probability information and the second probability information, the behavior change information of the current task model compared with the historical task model is determined, wherein the behavior change information includes the importance sampling ratio corresponding to at least one word position; When the target aggregation model is the first aggregation model, the target aggregation model is directly used to aggregate the importance sampling ratios corresponding to at least one of the word positions to generate a target aggregation value, which is used to calculate the loss value. The first aggregation model is an aggregation model that improves the influence of the importance sampling ratio of candidate word positions on the target aggregation value. The candidate word position is a word position among at least one word position whose importance sampling ratio is less than a first threshold.

2. The method for aggregating importance sampling ratios according to claim 1, characterized in that, The method further includes: When the target aggregation model is the second aggregation model, and after determining that at least one of the word positions has a word position to be adjusted based on the behavior change information, the importance sampling ratio of the word position to be adjusted is adjusted based on a preset adjustment method. The second aggregation model is an aggregation model that cannot improve the influence of the importance sampling ratio of the candidate word position on the target aggregation value. After completing the adjustment of the importance sampling ratio of all the word positions to be adjusted, the target aggregation model is used to aggregate the importance sampling ratio corresponding to at least one of the word positions to generate the target aggregation value.

3. The method for aggregating importance sampling ratios according to claim 1 or 2, characterized in that, The step of determining the behavioral change information of the current task model compared to the historical task model based on the first probability information and the second probability information includes: Based on the first probability value of the first word position in the first probability information and the second probability value in the second probability information, a first importance sampling ratio of the first word position is determined, wherein the first word position is any one of at least one of the word positions. After determining the importance sampling ratio corresponding to at least one of the lexical positions, the behavior change information is generated based on the importance sampling ratio corresponding to at least one of the lexical positions.

4. The method for aggregating importance sampling ratios according to claim 1 or 2, characterized in that, The step of determining the behavioral change information of the current task model compared to the historical task model based on the first probability information and the second probability information includes: Based on the third probability value of the second word position in the first probability information and the fourth probability value in the second probability information, the second importance sampling ratio of the second word position is determined, wherein the second word position is any one of at least one of the word positions. Convert the form of the second importance sampling ratio into a logarithmic form; After determining the formal transformation operation of the importance sampling ratio corresponding to at least one of the lexical positions, the behavior change information is generated based on the importance sampling ratio in logarithmic form corresponding to at least one of the lexical positions.

5. The method for aggregating importance sampling ratios according to claim 2, characterized in that, Based on the behavioral change information, determining that at least one of the lexical positions contains the lexical position to be adjusted includes: Statistically analyze the distribution information of the importance sampling ratio of at least one of the aforementioned word positions; Obtain the target quantile range; Based on the distribution information and the target quantile range, the first threshold and the second threshold are determined respectively, wherein the second threshold is greater than the first threshold; At least one of the lexical positions whose importance sampling ratio is less than the first threshold or greater than the second threshold is determined as the lexical position to be adjusted, wherein the lexical position to be adjusted includes the candidate lexical position.

6. The method for aggregating importance sampling ratios according to claim 5, characterized in that, The acquisition of the target quantile range includes: Obtain the target task type of the current task model, as well as the quantile ranges corresponding to the various task types; Based on the task type, the target quantile range corresponding to the target task type is determined from the quantile ranges corresponding to the various task types.

7. The method for aggregating importance sampling ratios according to claim 5, characterized in that, The preset adjustment methods include: Increase the importance sampling ratio of word positions whose importance sampling ratio is less than the first threshold by a first preset value; Additionally, the importance sampling ratio of word positions whose importance sampling ratio is greater than the second threshold is reduced by a second preset value.

8. The method for aggregating importance sampling ratios according to claim 1 or 2, characterized in that, Obtaining the target aggregation model includes: Obtain initial aggregation model and historical loss value information; Based on the historical loss value information, the values ​​of the target parameters in the initial aggregation model are adjusted to obtain the target aggregation model.

9. The method for aggregating importance sampling ratios according to claim 8, characterized in that, The historical loss value information includes loss values ​​corresponding to multiple training epochs; the step of adjusting the target parameter values ​​in the initial aggregation model based on the historical loss value information to obtain the target aggregation model includes: Based on the loss values ​​corresponding to the multiple training rounds, the target evolution characteristics of the loss values ​​are identified; Based on the target evolution characteristics, obtain the target preset value corresponding to the target evolution characteristics; After adjusting the values ​​of the target parameters in the initial aggregation model to the target preset values, the target aggregation model is obtained.

10. The method for aggregating importance sampling ratios according to claim 1 or 2, characterized in that, After aggregating the importance sampling ratios corresponding to at least one of the lexical positions using the target aggregation model to generate the target aggregation value, the method further includes: After obtaining the aggregated values ​​and initial behavioral advantage scores corresponding to the multiple sequences output by the current task model, it is determined whether the first sequence conforms to the sequence feature analysis rules based on the first sequence and the pre-built sequence feature analysis rules, wherein the first sequence is any one of the multiple sequences; If it is determined that the first sequence does not conform to the sequence feature analysis rules, the initial behavioral advantage score of the first sequence is directly determined as the final behavioral advantage score of the first sequence. Alternatively, if it is determined that the first sequence conforms to the sequence feature analysis rule, an increment value corresponding to the sequence feature analysis rule can be obtained; Based on the increased value and the initial behavioral advantage score of the first sequence, the final behavioral advantage score of the first sequence is determined; After determining the final behavioral advantage scores corresponding to the multiple sequences, the loss value is determined based on the pre-constructed loss function, the final behavioral advantage scores corresponding to the multiple sequences, and the aggregated value.

11. The method for aggregating importance sampling ratios according to claim 8, characterized in that, The initial aggregation model is expressed as follows: Where i is the group index and j is the sequence index. Let be the aggregate value of the i-th sequence in the j-th group. Let be the number of tokens in the i-th sequence of the j-th group, and t be the index of the token. The importance sampling ratio of the t-th word in the i-th sequence is given by p, where p is the target parameter.

12. The method for aggregating importance sampling ratios according to claim 1 or 2, characterized in that, The first aggregation model uses the following expression: Where i is the group index, j is the sequence index, and HM is the aggregate value of the i-th sequence in the j-th group. Let be the number of tokens in the i-th sequence of the j-th group, and t be the index of the token. The importance sampling ratio of the t-th word in the i-th sequence.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the importance sampling ratio aggregation method as described in any one of claims 1 to 12 when executing the computer program.

Images