Strategy model training method and device based on pruning, equipment and medium

Abstract

The invention relates to the technical field of model training, can be applied to the fields of financial science and technology and medical health, and discloses a strategy model training method, device and equipment based on pruning and a medium. Calculating an advantage value of the completion item according to a reward function; pruning the first completion item set according to the advantage value to obtain a first effective completion item subset; according to the first effective completion item subset and the first completion item set, detecting whether a completion item supplementing condition is met or not; if the completion item supplement condition is met, a plurality of second completion item sets are obtained again, and pruning is carried out on the second completion item sets to obtain a second effective completion item subset; and training a new strategy model according to the first effective completion item subset and the second effective completion item subset. And the efficiency of strategy model training is improved.

Description

Technical Field

[0001] This invention relates to the field of model training technology and can be applied to the fields of financial technology and healthcare. In particular, it relates to a pruning-based strategy model training method, apparatus, equipment, and medium. Background Technology

[0002] With the widespread application of large language models in tasks such as mathematical reasoning and code generation, policy optimization-based model training methods are gradually becoming an important means to improve model performance in the fields of fintech and healthcare. Examples include quantitative trading strategy optimization models in fintech and medical image diagnosis models in healthcare. However, existing policy model training methods still have the following problems: 1. High training cost: Each training sample requires generating multiple completion items for intra-group comparison, leading to a multiple increase in the computational cost of forward propagation and gradient updates; 2. Uneven contribution of completion items: Theoretically, the impact of each completion item on the policy optimization objective function depends on its degree of dominance. Low-dominance samples not only contribute limitedly to the training signal but may even interfere with the gradient direction; 3. Low utilization of computational resources: In multi-device parallel training, the number of completion items processed by different devices varies greatly, forming a "bucket effect," limiting the overall training efficiency to the slowest device; 4. Difficulty in ensuring training stability: High redundancy in completion item input may lead to gradient oscillations, affecting the convergence of the optimization process. Summary of the Invention

[0003] This invention provides a pruning-based strategy model training method, apparatus, computer equipment, and medium to solve the technical problem of low training efficiency of existing strategy models.

[0004] Firstly, a pruning-based strategy model training method is provided, including:

[0005] Obtain multiple sets of first completed items, and for each completed item in the first completed item set, calculate the advantage value of the completed item according to the reward function;

[0006] The first set of completed items is pruned according to the advantage value to obtain a first valid subset of completed items;

[0007] Based on the first valid subset of completed items and the first set of completed items, check whether the conditions for supplementary completed items are met;

[0008] If the conditions for supplementary completion items are met, multiple sets of second completion items are re-acquired, and the second sets of completion items are pruned to obtain a second valid subset of completion items;

[0009] The new policy model is trained based on the first subset of valid completed items and the second subset of valid completed items.

[0010] Secondly, a pruning-based strategy model training device is provided, comprising:

[0011] A calculation unit is used to acquire multiple sets of first completed items, and for each completed item in the first completed item set, calculate the advantage value of the completed item according to a reward function.

[0012] The first pruning unit is used to prune the first set of completed items according to the advantage value to obtain a first valid subset of completed items.

[0013] The detection unit is used to detect whether the supplementary completion item conditions are met based on the first valid completion item subset and the first completion item set;

[0014] The second pruning unit is used to, if the supplementary completion item condition is met, re-acquire multiple sets of second completion items and prune the sets of second completion items to obtain a second valid subset of completion items;

[0015] The training unit is used to train the new policy model based on the first subset of valid completed items and the second subset of valid completed items.

[0016] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described pruning-based policy model training method.

[0017] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described pruning-based policy model training method.

[0018] In the above-described scheme implemented by the pruning-based strategy model training method, apparatus, computer device, and storage medium, multiple first completion item sets can be obtained. For each completion item in the first completion item set, the advantage value of the completion item is calculated according to the reward function. The first completion item set is pruned according to the advantage value to obtain a first effective completion item subset. The first effective completion item subset and the first completion item set are used to detect whether the supplementary completion item condition is met. If the supplementary completion item condition is met, multiple second completion item sets are re-obtained, and the second completion item sets are pruned to obtain a second effective completion item subset. The new strategy model is trained according to the first effective completion item subset and the second effective completion item subset. In this invention, by pruning the first and second sets of completed items, low-dominance completed items are eliminated, effectively reducing unnecessary forward and backward propagation and avoiding interference from multiple sources of low-quality completed items on the training trajectory. This makes the gradient update direction on which training depends more consistent, thereby improving the efficiency of policy model training and ensuring training stability. Training the new policy model using the first effective completed item subset and the supplementary second effective completed item subset not only improves the efficiency of policy model training but also improves the utilization of computing resources. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a pruning-based strategy model training method according to an embodiment of the present invention.

[0021] Figure 2 yes Figure 1 A schematic diagram of a specific implementation of step S110;

[0022] Figure 3 yes Figure 1 A schematic diagram of a specific implementation of step S150;

[0023] Figure 4 This is an overall structural diagram of a pruning-based strategy model training method in one embodiment of the present invention;

[0024] Figure 5 This is a schematic block diagram of a pruning-based strategy model training device according to an embodiment of the present invention;

[0025] Figure 6This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0026] Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The pruning-based policy model training method provided in this invention can be applied to either a client or a server. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. Currently, in the fintech field, the efficiency of policy model training is relatively low. To address this problem, this invention proposes a pruning-based policy model training method. This method prunes the first and second sets of completed items, eliminating low-dominance completed items, effectively reducing unnecessary forward and backward propagation, avoiding interference from multiple low-quality completed items on the training trajectory, and making the gradient update direction on which training depends more consistent, thereby improving the efficiency of policy model training and ensuring training stability. By training the new policy model using a first subset of effective completed items and a supplementary subset of second effective completed items, not only is the efficiency of policy model training improved, but the utilization of computing resources is also increased. The invention will be described in detail below through specific embodiments.

[0029] Please see Figure 1 As shown, Figure 1 A flowchart of a pruning-based strategy model training method provided in an embodiment of the present invention includes the following steps: S110-S150.

[0030] S110. Obtain multiple sets of first completed items, and for each completed item in the first completed item set, calculate the advantage value of the completed item according to the reward function.

[0031] Specifically, obtain a batch prompt set S = {s1, s2, ..., s} from the task pool. BThe set of prompts is input into the old policy model, which outputs multiple sets of first completed items. For each completed item in the first completed item set, the advantage value of the completed item is calculated according to the reward function. It should be noted that in this embodiment, s j j = 1, 2, ..., B, which are prompts. Prompts refer to the instructions or query text input to the model to guide the model to generate specific types of output.

[0032] Among them, such as Figure 2 As shown, step S110 includes the following steps: S111-S114:

[0033] S111. Calculate the first reward score based on the first reward function and the completed item;

[0034] S112. Calculate the second reward score based on the second reward function and the completed item;

[0035] S113. The first reward score and the second reward score are weighted and calculated to obtain the total reward score;

[0036] S114. Calculate the advantage value of the completed item based on the total reward score.

[0037] Specifically, the reward function is r i =R format (o i )+R accuracy (o i ), where R format (o i R is the first reward function. accuracy (o i ) is the second reward function, o iThe first reward function is a format reward, and its scoring process is based on structured compliance verification. First, the completed item text is input into a format parser, which performs multi-dimensional matching based on a pre-defined grammar rule base. For code generation tasks, the rule base includes constraints on code structural integrity (e.g., function definitions must include parameter lists and return statements), syntax tree validity (e.g., loop structures must be fully closed), and output format specifications. The format parser scans character by character using a finite state machine. When it detects missing key structures (e.g., missing logical termination markers like "therefore" or "Q.E.D." in mathematical derivation tasks) or format violations (e.g., unclosed XML tags), it outputs a score of 0; when all format constraints are fully satisfied, it outputs 1 point. This process uses a deterministic algorithm to ensure that the scoring result is not influenced by semantic content. It should also be noted that the second reward function is an accuracy reward, and its scoring depends on the verification feedback from the task solver. The task solver calls a dedicated verification module based on the task type: in mathematical reasoning tasks, it calls the symbolic computation engine to perform equivalent transformation verification on the derivation process of the completed item; in code generation tasks, it executes unit test cases and covers boundary condition detection; in diagnostic tasks in the medical and health field, it compares the accuracy of the pathological description with the medical knowledge base. The scoring uses a three-tiered system: if the completed item completely solves the core problem (e.g., correctly outputs the equation solution set and passes all test cases), it receives a full score of 2 points; if there are some defects but the core logic is correct (e.g., the mathematical derivation skips non-critical steps but the conclusion is valid, or the code passes 80% of the test cases), it outputs 1.5 points; when a fundamental error occurs (e.g., drug dosage calculation exceeds the safety threshold), it is directly judged as 0 points.

[0038] Furthermore, after obtaining the first and second reward scores, the total reward score is generated using a dynamic weighted fusion algorithm, calculated as r. i =w f *R format +w a *R accuracy Weighting coefficient (w) f ,w a Adaptively adjust based on task characteristics. Specifically, for highly structured tasks, such as generating financial statements in the fintech field: assign higher weight to formatting rewards (w f =0.6,w a =0.4), ensuring the output strictly conforms to audit specifications; for tasks with high semantic complexity (such as clinical diagnostic reasoning): increase the accuracy weight (w f =0.2,w a =0.8), enhancing the clinical reliability of medical judgment. Understandably, w can be dynamically adjusted in real time. f and w a .

[0039] Furthermore, step S114 specifically includes: calculating the average reward score and standard deviation reward score based on the total reward score corresponding to all completion items in the first completion item set; calculating the difference between the total reward score and the average reward score to obtain the reward score difference; and calculating the quotient of the reward score difference and the standard deviation reward score to obtain the advantage value of the completion item. Specifically, for the same prompt s j The derived first complete set {o1,...,o G Extract its total reward score set {r1,...,r}. G} Calculate the mean u within the first set of completed items. j As shown in formula (1), the standard deviation σ is calculated. j As shown in formula (2), the calculated advantage value As shown in formula (3), j represents the sequence number of the prompts, with a value range of j = 1, 2, ..., B, where B is the total number of prompts processed in the current batch (e.g., B = 64). i represents the sequence number of the completion under the same prompt, with a value range of i = 1, 2, ..., G, where G is the number of completions generated for each prompt (e.g., G = 8).

[0040]

[0041] S120. Prune the first set of completed items according to the advantage value to obtain a first valid subset of completed items.

[0042] Specifically, for each completion item in the first set of completion items, the absolute value of the advantage value corresponding to the completion item is calculated to obtain the absolute advantage value; the absolute advantage value is compared with a preset advantage value; if the absolute value of the advantage value is greater than the preset advantage value, the completion item is considered a valid completion item, forming the first subset of valid completion items. Understandably, if the absolute value of the advantage value is not greater than the preset advantage value, the completion item is removed. It should be noted that in this embodiment, the advantage value is used as the basis for measuring its training contribution. Theoretical analysis shows that the gradient of the policy model is mainly driven by the weight of the advantage value, and when the absolute value of the advantage value approaches zero, the weight of the completion item in the optimization objective also approaches zero. Based on this, by setting a preset advantage value, only completion items with an absolute advantage value greater than the preset advantage value are retained to participate in subsequent forward propagation and gradient calculation, filtering out redundant or interfering completion items, thereby reducing the proportion of invalid calculations. For ease of understanding, assume the following dominance values: A1 = 1.3, A2 = -0.79, A3 = 0.4, A4 = -0.94, and a preset dominance value τ = 0.5. The absolute dominance values ​​are: |A1| = 1.3 > τ (retain), |A2| = 0.79 > τ (retain), |A3| = 0.4 < τ (reject), |A4| = 0.94 > τ (retain). The effective completion items in the first effective completion item subset are A1, A2, and A4. Therefore, originally, the gradients of four policies needed to be calculated; after dominance value filtering, only three participate in training (A3 is removed). The dominance value of the removed policy (0.4) is close to zero, and its theoretically calculated gradient weight is only 0.02. Filtering reduces the computational cost by 25% while maintaining the stability of the model's convergence direction.

[0043] S130. Detect whether the supplementary completion item condition is met based on the first valid completion item subset and the first completion item set.

[0044] Specifically, the total number of valid completed items in the first valid completed item subset is counted to obtain the total number of completed items; the total number of completed items in the first completed item set is counted to obtain the batch capacity; the difference between the batch capacity and the total number of completed items is calculated to obtain the capacity difference; if the capacity difference is greater than a preset capacity difference, it is determined that the supplementary completed item condition is met. For example, suppose that the maximum number of completed items that a computer device performing pruning-based strategy model training can handle is B*G, and it receives B prompts, each prompt generates G completed items, and after pruning, each prompt retains only k completed items with the largest advantage value. At this time, the gap capacity is δ=B*(GK), that is, the capacity difference is δ. δ is compared with the preset capacity difference. If δ is greater than the preset capacity difference, it is determined that the supplementary completed item condition is met. It should be noted that δ represents unused computing resources.

[0045] S140. If the supplementary completion item condition is met, then multiple second completion item sets are re-acquired, and the second completion item sets are pruned to obtain a second valid completion item subset.

[0046] Specifically, when the conditions for supplementing completion items are met, a new set of prompts is obtained from the task pool to generate multiple sets of second completion items. Each new prompt generates G completion items from the old strategy model, followed by a pruning process: calculating the first and second reward scores, weighting the first and second reward scores to obtain the total reward score, calculating the average reward score and standard deviation reward score of the second completion item set, calculating the advantage value of the second completion item set based on the total reward score, average reward score, and standard deviation reward score, and retaining k completion items whose absolute advantage value exceeds a threshold τ to form a subset of second effective completion items. It should be noted that in each training round, new prompts are added from the task pool to fill computational gaps. All newly added completion items, i.e., the second completion item set, still need to have their advantage values ​​calculated and be pruned based on these advantage values ​​to ensure the consistency and effectiveness of the trained completion items. This dynamic scheduling mechanism breaks the limitation of "fixed sample size per round," increases the total number of problems covered per unit time, and further improves training efficiency.

[0047] By implementing the above steps S110-S140, the present invention has the following beneficial effects: 1. Reduced computational complexity: By using the pruning mechanism, unnecessary forward and backward propagation steps are effectively reduced while maintaining the quality of the training core, resulting in a significant reduction in the computational cost per completed item; 2. Improved gradient quality and optimization stability: By eliminating low-dominance completed items, the gradient update direction on which training depends becomes more consistent, avoiding interference from multiple low-quality completed items on the training trajectory, making the optimization path more stable, shortening the convergence rounds and reducing oscillations.

[0048] S150. Train the new policy model based on the first valid completion item subset and the second valid completion item subset.

[0049] Specifically, training the new strategy model based on the first and second valid completed item subsets not only improves resource utilization but also ensures that the newly added second valid completed item subset meets the consistency quality standard.

[0050] Among them, such as Figure 3 As shown, step S150 includes the following steps: S151-S153:

[0051] S151. Merge the first valid completed item subset and the second valid completed item subset to obtain the final valid completed item subset;

[0052] S152. Input the final valid completed item subset into the old strategy model, the reference model and the new strategy model to obtain the initial probability distribution, the reference probability distribution and the predicted probability distribution.

[0053] S153. The new strategy model is trained based on the final valid completed item subset, the initial probability distribution, the reference probability distribution, and the predicted probability distribution.

[0054] Specifically, the first and second valid completed item subsets are first merged to obtain the final valid completed item subset. This final valid completed item subset is then input into three models in parallel: the old strategy model receives the valid completed items from the final valid completed item subset and outputs the initial probability distribution of each token sequence through forward propagation; the reference model processes the same valid completed items synchronously and outputs a reference probability distribution; and the new strategy model outputs the predicted probability distribution. It should be noted that each valid completed item in the final valid completed item subset is a token sequence; understandably, each completed item in the first and second completed item sets is also a token sequence.

[0055] Further, step S153 includes: calculating the KL divergence of the reference probability distribution and the predicted probability distribution to obtain a divergence value; calculating the probability ratio of each valid completed item in the final valid completed item subset based on the initial probability distribution and the reference probability distribution; calculating an advantage evaluation value based on the probability ratio and the advantage value of each valid completed item in the final valid completed item subset; calculating the gradient based on the divergence value and the advantage evaluation value; and training the new policy model based on the gradient. It should be noted that the advantage evaluation value serves as an evaluation metric for the policy improvement magnitude, avoiding excessive differences between the old and new policies to maintain training stability; the divergence value constrains the degree of policy deviation. It should also be noted that the gradient is calculated based on the divergence value and the advantage evaluation value, and the parameters of the new policy model are updated based on the gradient.

[0056] By implementing the above steps S110-S150, the present invention has the following beneficial effects: 1. Fully unleashing the potential of parallel computing: Although the pruning operation reduces the number of completed items, the introduction of more completed items to participate in training through the dynamic scheduling mechanism keeps the computing device in a high-load state, solves the problem of GPU utilization decline caused by data pruning, and improves the overall throughput at the system level; 2. Strong versatility and easy integration: Only the processing flow is adjusted in the policy optimization structure, without relying on specific model architecture or reward definition, and is applicable to a variety of reinforcement learning frameworks (including rule-based, heuristic, or model scoring policy training tasks), with good cross-task adaptability; 5. Improving the controllability of the training process: Through adjustable parameters, such as preset advantage values, the training process has stronger controllability and debuggability, and developers can flexibly balance computational efficiency and information retention according to specific task requirements.

[0057] Please see Figure 4 , Figure 4 This is an overall structural diagram of a pruning-based strategy model training method according to an embodiment of the present invention. Figure 4 In this context, Policy Model represents the new policy model, Old Policy Model represents the old policy model, Prompts are hints, and Completions are the final subset of valid completed items. R D is the advantage assessment value. KL Let D be the divergence value; the objective function for policy optimization is D. R -D KL ,Depend on Figure 4 It can be seen that, under the premise that the objective function of policy optimization remains unchanged, the sampling and usage strategy for the completed terms has been structurally optimized, which can significantly reduce the variance of gradient estimation, improve the convergence speed of the model, and have good stability and scalability.

[0058] In summary, this invention constructs an efficient mechanism for evaluating the importance of completed items and for dynamic processing. While ensuring optimization quality, it systematically reduces training overhead and resource redundancy, improves the efficiency of policy model training and the utilization of computing resources, and performs particularly well in large model, long sequence and multi-device parallel training tasks.

[0059] The pruning-based policy model training method in this invention removes low-dominance completions by pruning the first and second completion sets, effectively reducing unnecessary forward and backward propagation and avoiding interference from multiple low-quality completions on the training trajectory. This makes the gradient update direction on which training depends more consistent, thereby improving the efficiency and stability of policy model training. Training the new policy model using the first effective completion subset and the supplementary second effective completion subset not only improves the efficiency of policy model training but also increases the utilization of computing resources.

[0060] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0061] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0062] In one embodiment, a pruning-based policy model training device 200 is provided, which corresponds one-to-one with the pruning-based policy model training method described in the above embodiments. For example... Figure 5 As shown, the pruning-based strategy model training device includes an acquisition and calculation unit 201, a first pruning unit 202, a detection unit 203, a second pruning unit 204, and a training unit 205. Detailed descriptions of each functional module are as follows:

[0063] The calculation unit 201 is used to acquire multiple first completion item sets, and for each completion item in the first completion item set, calculate the advantage value of the completion item according to the reward function.

[0064] The first pruning unit 202 is used to prune the first set of completed items according to the advantage value to obtain a first valid subset of completed items;

[0065] Detection unit 203 is used to detect whether the supplementary completion item condition is met based on the first valid completion item subset and the first completion item set;

[0066] The second pruning unit 204 is used to, if the supplementary completion item condition is met, re-acquire multiple sets of second completion items and prune the sets of second completion items to obtain a second valid subset of completion items;

[0067] Training unit 205 is used to train a new policy model based on the first subset of valid completed items and the second subset of valid completed items.

[0068] In one embodiment, the calculation unit 201 is specifically used for:

[0069] Calculate the first reward score based on the first reward function and the completed item;

[0070] Calculate the second reward score based on the second reward function and the completed item;

[0071] The total reward score is obtained by weighting the first reward score and the second reward score;

[0072] The advantage value of the completed item is calculated based on the total reward score.

[0073] In one embodiment, the calculation unit 201 is further specifically used for:

[0074] Calculate the average reward score and standard deviation reward score based on the total reward score corresponding to all the completed items in the first set of completed items;

[0075] The reward score difference is obtained by calculating the difference between the total reward score and the average reward score;

[0076] The advantage value of the completion item is obtained by calculating the quotient of the reward score difference and the standard deviation of the reward score.

[0077] In one embodiment, the first pruning unit 202 is specifically used for:

[0078] For each completed item in the first set of completed items, the absolute value of the advantage value corresponding to the completed item is calculated to obtain the absolute advantage value;

[0079] Compare the absolute advantage value with the preset advantage value;

[0080] If the absolute value of the advantage value is greater than the preset advantage value, then the completed item is regarded as a valid completed item, forming the first subset of valid completed items.

[0081] In one embodiment, the detection unit 203 is specifically used for:

[0082] The total number of completed items is obtained by counting the total number of valid completed items in the first valid completed item subset;

[0083] The batch capacity is obtained by counting the total number of completed items in the first set of completed items;

[0084] The capacity difference is obtained by calculating the difference between the batch capacity and the total number of completed items;

[0085] If the capacity difference is greater than the preset capacity difference, then the condition for the supplementary completion item is determined to be met.

[0086] In one embodiment, the training unit 205 is specifically used for:

[0087] The first valid completed item subset and the second valid completed item subset are merged to obtain the final valid completed item subset;

[0088] The final valid completed item subset is input into the old strategy model, the reference model, and the new strategy model to obtain the initial probability distribution, the reference probability distribution, and the predicted probability distribution;

[0089] The new policy model is trained based on the final valid completed item subset, the initial probability distribution, the reference probability distribution, and the predicted probability distribution.

[0090] In one embodiment, the training unit 205 is further specifically used for:

[0091] KL divergence calculations are performed on the reference probability distribution and the predicted probability distribution to obtain divergence values;

[0092] The probability ratio of each valid completed item in the final valid completed item subset is calculated based on the initial probability distribution and the reference probability distribution, and an advantage evaluation value is calculated based on the probability ratio and the advantage value of each valid completed item in the final valid completed item subset.

[0093] The gradient is calculated based on the divergence value and the advantage evaluation value, and the new policy model is trained based on the gradient.

[0094] The pruning-based policy model training device of this invention removes low-dominance completions by pruning the first and second completion sets, effectively reducing unnecessary forward and backward propagation and avoiding interference from multiple low-quality completions on the training trajectory. This makes the gradient update direction on which training depends more consistent, thereby improving the efficiency of policy model training and ensuring training stability. By training the new policy model using the first effective completion subset and the supplementary second effective completion subset, not only is the efficiency of policy model training improved, but the utilization rate of computing resources is also increased.

[0095] Specific limitations regarding the pruning-based policy model training device can be found in the limitations of the pruning-based policy model training method described above, and will not be repeated here. Each unit in the aforementioned pruning-based policy model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0096] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a pruning-based policy model training method on the server side.

[0097] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a pruning-based policy model training method.

[0098] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described pruning-based policy model training method.

[0099] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described pruning-based policy model training method.

[0100] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0101] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0103] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A pruning-based strategy model training method, characterized in that, include: Obtain multiple sets of first completed items, and for each completed item in the first completed item set, calculate the advantage value of the completed item according to the reward function; The first set of completed items is pruned according to the advantage value to obtain a first valid subset of completed items; Based on the first valid subset of completed items and the first set of completed items, check whether the conditions for supplementary completed items are met; If the conditions for supplementary completion items are met, multiple sets of second completion items are re-acquired, and the second sets of completion items are pruned to obtain a second valid subset of completion items; The new policy model is trained based on the first subset of valid completed items and the second subset of valid completed items.

2. The pruning-based strategy model training method as described in claim 1, characterized in that, The reward function includes a first reward function and a second reward function, and the step of calculating the advantage value of the completed item based on the reward function includes: Calculate the first reward score based on the first reward function and the completed item; Calculate the second reward score based on the second reward function and the completed item; The total reward score is obtained by weighting the first reward score and the second reward score; The advantage value of the completed item is calculated based on the total reward score.

3. The pruning-based strategy model training method as described in claim 2, characterized in that, The step of calculating the advantage value of the completed item based on the total reward score includes: Calculate the average reward score and standard deviation reward score based on the total reward score corresponding to all the completed items in the first set of completed items; The reward score difference is obtained by calculating the difference between the total reward score and the average reward score; The advantage value of the completion item is obtained by calculating the quotient of the reward score difference and the standard deviation of the reward score.

4. The pruning-based strategy model training method as described in claim 1, characterized in that, The step of pruning the first set of completed items according to the advantage value to obtain a first subset of valid completed items includes: For each completed item in the first set of completed items, the absolute value of the advantage value corresponding to the completed item is calculated to obtain the absolute advantage value; Compare the absolute advantage value with the preset advantage value; If the absolute value of the advantage value is greater than the preset advantage value, then the completed item is regarded as a valid completed item, forming the first subset of valid completed items.

5. The pruning-based strategy model training method as described in claim 1, characterized in that, The step of detecting whether the supplementary completion item condition is met based on the first valid completion item subset and the first completion item set includes: The total number of completed items is obtained by counting the total number of valid completed items in the first valid completed item subset; The batch capacity is obtained by counting the total number of completed items in the first set of completed items; The capacity difference is obtained by calculating the difference between the batch capacity and the total number of completed items; If the capacity difference is greater than the preset capacity difference, then the condition for the supplementary completion item is determined to be met.

6. The pruning-based strategy model training method as described in claim 1, characterized in that, The step of training the new policy model based on the first subset of valid completed items and the second subset of valid completed items includes: The first valid completed item subset and the second valid completed item subset are merged to obtain the final valid completed item subset; The final valid completed item subset is input into the old strategy model, the reference model, and the new strategy model to obtain the initial probability distribution, the reference probability distribution, and the predicted probability distribution; The new policy model is trained based on the final valid completed item subset, the initial probability distribution, the reference probability distribution, and the predicted probability distribution.

7. The pruning-based strategy model training method as described in claim 6, characterized in that, The step of training the new policy model based on the final valid completed item subset, the initial probability distribution, the reference probability distribution, and the predicted probability distribution includes: KL divergence calculations are performed on the reference probability distribution and the predicted probability distribution to obtain divergence values; The probability ratio of each valid completed item in the final valid completed item subset is calculated based on the initial probability distribution and the reference probability distribution, and an advantage evaluation value is calculated based on the probability ratio and the advantage value of each valid completed item in the final valid completed item subset. The gradient is calculated based on the divergence value and the advantage evaluation value, and the new policy model is trained based on the gradient.

8. A pruning-based strategy model training device, characterized in that, include: A calculation unit is used to acquire multiple sets of first completed items, and for each completed item in the first completed item set, calculate the advantage value of the completed item according to a reward function. The first pruning unit is used to prune the first set of completed items according to the advantage value to obtain a first valid subset of completed items. The detection unit is used to detect whether the supplementary completion item conditions are met based on the first valid completion item subset and the first completion item set; The second pruning unit is used to, if the supplementary completion item condition is met, re-acquire multiple sets of second completion items and prune the sets of second completion items to obtain a second valid subset of completion items; The training unit is used to train the new policy model based on the first subset of valid completed items and the second subset of valid completed items.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the pruning-based policy model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the pruning-based policy model training method as described in any one of claims 1 to 7.

Images