A method and apparatus for training reinforcement learning models based on dynamic evaluation metrics

Abstract

This invention discloses a method and apparatus for training a reinforcement learning model based on a dynamic evaluation index. The method includes: selecting target training data from predetermined original training data based on scoring results; wherein the scoring results are conclusions formed after quantifying the prediction difficulty of the original training data; predicting the target training data according to the reinforcement learning model, outputting the prediction results, and calculating an evaluation index value based on the prediction results and a preset correct result; determining a target evaluation index threshold; wherein the target evaluation index threshold is dynamically adjusted according to the prediction process of the reinforcement learning model; determining a reward function value and / or a penalty function value using the evaluation index value and the target evaluation index threshold, and optimizing the parameters of the reinforcement learning model using the reward function value and / or the penalty function value. This technical solution improves the stability and efficiency of reinforcement learning model training by setting a dynamic evaluation index for training the reinforcement learning model.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning model training technology, and in particular to a method and apparatus for training reinforcement learning models based on dynamic evaluation metrics. Background Technology

[0002] In the field of reinforcement learning, the training process of a model is essentially a process in which the model continuously optimizes its strategy to achieve the goal through continuous interaction with the environment. As the core standard for measuring the quality of a strategy, the rationality of the evaluation metric directly determines the training effect and the final performance.

[0003] Setting high evaluation metrics can make it difficult for the model to obtain positive feedback in the initial stages, causing policy iteration to stagnate. Conversely, setting low evaluation metrics also has significant drawbacks. While lower targets can allow the model to quickly obtain positive feedback and accelerate the initial training process, they can easily lead to insufficient motivation for policy optimization.

[0004] Therefore, finding a balance between high and low evaluation metrics has become a key problem that urgently needs to be solved in the training of reinforcement learning models. Summary of the Invention

[0005] This invention provides a method and apparatus for training reinforcement learning models based on dynamic evaluation metrics. By setting dynamic evaluation metrics to train reinforcement learning models, the stability and efficiency of reinforcement learning model training are improved.

[0006] According to one aspect of the present invention, a method for training a reinforcement learning model based on a dynamic evaluation metric is provided, the method comprising:

[0007] Target training data is selected from the predetermined original training data based on the scoring results; wherein, the scoring results are a conclusion formed after quantifying the prediction difficulty of the original training data;

[0008] The target training data is predicted based on the reinforcement learning model, the prediction result is output, and the evaluation index value is calculated based on the prediction result and the preset correct result.

[0009] Determine the target evaluation index threshold; wherein, the target evaluation index threshold is dynamically adjusted based on the prediction process of the reinforcement learning model;

[0010] Using the evaluation index value and the target evaluation index threshold, the reward function value and / or penalty function value are determined, and the parameters of the reinforcement learning model are optimized using the reward function value and / or penalty function value.

[0011] According to another aspect of the present invention, a reinforcement learning model training apparatus based on a dynamic evaluation metric is provided, the apparatus comprising:

[0012] The target training data filtering module is used to filter target training data from predetermined original training data based on the scoring results; wherein, the scoring results are a conclusion formed after quantifying the prediction difficulty of the original training data;

[0013] The evaluation index value calculation module is used to predict the target training data based on the reinforcement learning model, output the prediction result, and calculate the evaluation index value based on the prediction result and the preset correct result.

[0014] The target evaluation index threshold determination module is used to determine the target evaluation index threshold; wherein, the target evaluation index threshold is dynamically adjusted based on the prediction process of the reinforcement learning model;

[0015] The parameter optimization module is used to determine the reward function value and / or penalty function value using the evaluation index value and the target evaluation index threshold, and to optimize the parameters of the reinforcement learning model using the reward function value and / or penalty function value.

[0016] The technical solution of this invention involves: selecting target training data from predetermined original training data based on scoring results; predicting the target training data using a reinforcement learning model, outputting prediction results, and calculating evaluation index values ​​based on the prediction results and preset correct results; determining a target evaluation index threshold; determining reward function values ​​and / or penalty function values ​​using the evaluation index values ​​and the target evaluation index threshold; and optimizing the parameters of the reinforcement learning model using the reward function values ​​and / or penalty function values. This technical solution improves the stability and efficiency of reinforcement learning model training by setting dynamic evaluation indexes for training.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a reinforcement learning model training method based on dynamic evaluation metrics provided in Embodiment 1 of the present invention;

[0020] Figure 2This is a schematic diagram of a reinforcement learning model training method based on dynamic evaluation metrics provided in Embodiment 2 of the present invention;

[0021] Figure 3 This is a schematic diagram of another reinforcement learning model training method based on dynamic evaluation metrics provided in Embodiment 3 of the present invention;

[0022] Figure 4 This is a schematic diagram of another reinforcement learning model training method based on dynamic evaluation index provided in Embodiment 4 of the present invention;

[0023] Figure 5 A flowchart illustrating the training process of a reinforcement learning model based on a dynamic evaluation index, provided in Embodiment 5 of the present invention;

[0024] Figure 6 This is a schematic diagram of the training process of a reinforcement learning model based on a dynamic evaluation index, provided in Embodiment Six of the present invention.

[0025] Figure 7 This is a schematic diagram of another reinforcement learning model training process based on dynamic evaluation metrics provided in Embodiment 7 of the present invention;

[0026] Figure 8 This is a schematic diagram of the training process of another reinforcement learning model based on dynamic evaluation index provided in Embodiment 8 of the present invention;

[0027] Figure 9 This is a schematic diagram of the structure of a reinforcement learning model training device based on dynamic evaluation index provided in Embodiment 9 of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 This is a flowchart illustrating a reinforcement learning model training method based on a dynamic evaluation metric, according to Embodiment 1 of the present invention. This embodiment is applicable to situations involving parameter optimization of reinforcement learning models. The method can be executed by a reinforcement learning model training device based on a dynamic evaluation metric. This device can be implemented in hardware and / or software and can be configured within a device. For example, the device can be a backend server or other device with communication and computing capabilities. Figure 1 As shown, the method includes:

[0032] S110. Select target training data from the predetermined original training data based on the scoring results; wherein, the scoring results are a conclusion formed after quantifying the prediction difficulty of the original training data.

[0033] In this embodiment, the original training data refers to the initial data that has not been processed before the model is trained.

[0034] The scoring result is a conclusion drawn from a quantitative assessment of the prediction difficulty of the original training data. The scoring result can be presented in numerical form or in a graded format, etc.

[0035] In this scheme, target training data is selected from the original training data based on the scoring results.

[0036] S120. Based on the reinforcement learning model, predict the target training data, output the prediction result, and calculate the evaluation index value based on the prediction result and the preset correct result.

[0037] In this embodiment, the training objective of the reinforcement learning model is to enable the model to learn the optimal behavioral strategy through interaction with the environment in order to maximize long-term cumulative rewards.

[0038] The prediction result refers to the output value calculated by the model based on the target training data, which is the model's estimate of the correct result.

[0039] Specifically, the target training data is input into the reinforcement learning model, which then makes predictions based on the target training data and outputs the prediction results.

[0040] In this embodiment, evaluation metrics are key standards for measuring model performance. Evaluation metrics can be intersection-over-union ratio, accuracy, precision, etc.

[0041] Furthermore, the predicted results and correct results can be combined to obtain the evaluation index value. For example, the predicted results and correct results can be combined according to a preset cross-validation ratio (CUP) calculation formula to obtain the evaluation index value. Similarly, the predicted results and correct results can be combined according to a preset accuracy calculation formula to obtain the evaluation index value.

[0042] S130. Determine the target evaluation index threshold; wherein the target evaluation index threshold is dynamically adjusted based on the prediction process of the reinforcement learning model.

[0043] In this scheme, the target evaluation index threshold adopts a dynamic setting mechanism, and will be continuously optimized and dynamically adjusted based on the real-time prediction results during the prediction process of the reinforcement learning model.

[0044] S140. Using the evaluation index value and the target evaluation index threshold, determine the reward function value and / or penalty function value, and use the reward function value and / or penalty function value to optimize the parameters of the reinforcement learning model.

[0045] In this embodiment, the reward function is the positive numerical feedback given by the system when the model performs an action that helps achieve the goal; the penalty function is the negative numerical feedback given by the system when the model performs an action that is detrimental to the goal.

[0046] When the evaluation index is the intersection-union ratio, the reward function can be expressed by the following formula;

[0047] ;

[0048] ;

[0049] in, For the predicted results, For the correct result, For the threshold, The intersection-union ratio of the predicted results and the correct results. This is the reward function.

[0050] The final reward formula is: , This represents the reward coefficient. For the preset format reward function, The format is the reward coefficient.

[0051] Furthermore, when the evaluation index value exceeds the target evaluation index threshold, the reward function is triggered, and the parameters of the reinforcement learning model are optimized based on the reward function value; when the evaluation index value does not exceed the target evaluation index threshold, no reward is given, or the penalty function is triggered, and the parameters of the reinforcement learning model are optimized based on the penalty function value.

[0052] In this scheme, when the target evaluation metric threshold meets the preset threshold condition, iterative training stops and the optimized reinforcement learning model is output. When the target evaluation metric threshold does not meet the threshold condition, the optimized reinforcement learning model is used to predict the target training data, and the steps of calculating the evaluation metric value, determining the target evaluation metric threshold, and optimizing parameters are repeated until the target evaluation metric threshold meets the preset threshold condition, and the optimized reinforcement learning model is output. The threshold condition can be flexibly set according to the training requirements of the reinforcement learning model. For example, the threshold condition can be specifically set to a threshold of 0.99 for the target evaluation metric (e.g., Intersection over Union (IoU)).

[0053] Furthermore, when the reinforcement learning model parameters converge, iterative training is stopped and the optimized reinforcement learning model is output. When the reinforcement learning model parameters do not converge, the optimized reinforcement learning model is used to predict the target training data, and the steps of calculating the evaluation index value, determining the target evaluation index threshold, and optimizing the parameters are repeated until the reinforcement learning model parameters converge, and the optimized reinforcement learning model is output.

[0054] In this embodiment, when the threshold of the target evaluation metric meets the preset threshold condition, if the number or proportion of target training data whose evaluation metric value exceeds the target evaluation metric threshold exceeds the preset threshold, iterative training is stopped and the optimized reinforcement learning model is output. When the number or proportion of target training data whose evaluation metric value exceeds the target evaluation metric threshold does not exceed the preset threshold, the optimized reinforcement learning model is used to predict the target training data, and the steps of calculating the evaluation metric value, determining the target evaluation metric threshold, and optimizing parameters are repeated until the number or proportion of target training data whose evaluation metric value exceeds the target evaluation metric threshold exceeds the preset threshold, and the optimized reinforcement learning model is output.

[0055] The technical solution of this invention involves: selecting target training data from predetermined original training data based on scoring results; predicting the target training data using a reinforcement learning model, outputting prediction results, and calculating evaluation index values ​​based on the prediction results and preset correct results; determining a target evaluation index threshold; determining reward function values ​​and / or penalty function values ​​using the evaluation index values ​​and the target evaluation index threshold, and optimizing the parameters of the reinforcement learning model using the reward function values ​​and / or penalty function values. By implementing this technical solution and setting dynamic evaluation indexes to train the reinforcement learning model, the stability and efficiency of reinforcement learning model training are improved.

[0056] Example 2

[0057] Figure 2 This is a schematic diagram of a reinforcement learning model training method based on dynamic evaluation metrics provided in Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the process for determining the threshold of the target evaluation metrics. Figure 2 As shown, the method includes:

[0058] S210. Select target training data from the predetermined original training data based on the scoring results; wherein, the scoring results are a conclusion formed after quantifying the prediction difficulty of the original training data.

[0059] S220. Based on the reinforcement learning model, predict the target training data, output the prediction result, and calculate the evaluation index value based on the prediction result and the preset correct result.

[0060] S230, Detect the number of training batches of the reinforcement learning model under the threshold of the previous objective evaluation index.

[0061] Specifically, the number of training batches using the previous target evaluation metric threshold can be automatically identified by analyzing the training logs of the reinforcement learning model; alternatively, the number of training batches of the reinforcement learning model under the previous target evaluation metric threshold can be detected by using code tracking, variable monitoring, or framework APIs.

[0062] S240. If the number of training batches is less than a preset threshold, then the previous target evaluation index threshold is determined as the target evaluation index threshold.

[0063] The threshold for the number of repetitions can be flexibly adjusted according to the training requirements of the reinforcement learning model.

[0064] In this scheme, when the number of training batches has not yet reached the preset threshold, the current target evaluation index threshold will use the previous target evaluation index threshold.

[0065] S250. If the number of training batches is greater than or equal to a preset number of times threshold, the previous target evaluation index threshold is increased by a first step length or a second step length to obtain the target evaluation index threshold; wherein, the first step length is a fixed step length; the second step length is calculated based on the evaluation index value.

[0066] In this scheme, the first step size is set to a fixed step size. For example, the first step size can be set to 0.05.

[0067] In this embodiment, the second step length is a dynamic step length. It can be calculated based on the correlation between the evaluation index value and the step length, or it can be calculated based on a preset calculation formula and the evaluation index value.

[0068] Furthermore, when the number of training batches reaches a preset threshold, the current target evaluation index threshold is determined by increasing the threshold of the previous target evaluation index. Specifically, the threshold of the previous target evaluation index is increased by either the first step length or the second step length.

[0069] S260. Using the evaluation index value and the target evaluation index threshold, determine the reward function value and / or penalty function value, and use the reward function value and / or penalty function value to optimize the parameters of the reinforcement learning model.

[0070] The technical solution of this invention involves: selecting target training data from predetermined original training data based on scoring results; predicting the target training data using a reinforcement learning model, outputting prediction results, and calculating evaluation index values ​​based on the prediction results and preset correct results; determining a target evaluation index threshold based on the number of training batches; determining reward function values ​​and / or penalty function values ​​using the evaluation index values ​​and the target evaluation index threshold, and optimizing the parameters of the reinforcement learning model using the reward function values ​​and / or penalty function values. By implementing this technical solution and setting dynamic evaluation indexes to train the reinforcement learning model, the stability and efficiency of model training can be significantly improved. This approach not only meets the high accuracy requirements of model training but also achieves progressive training by gradually introducing reinforcement training steps from simple to complex.

[0071] Example 3

[0072] Figure 3 This is a schematic diagram of another reinforcement learning model training method based on dynamic evaluation metrics provided in Embodiment 3 of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the process for determining the threshold of the target evaluation metric. Figure 3 As shown, the method includes:

[0073] S310. Select target training data from the predetermined original training data based on the scoring results; wherein, the scoring results are a conclusion formed after quantifying the prediction difficulty of the original training data.

[0074] S320. Based on the reinforcement learning model, predict the target training data, output the prediction result, and calculate the evaluation index value based on the prediction result and the preset correct result.

[0075] S330. Determine whether the model performance of the previous training batch of the reinforcement learning model meets the preset first condition; if it does, then in the current training batch of the reinforcement learning model, increase the threshold of the previous target evaluation index by the first step length or the second step length to obtain the target evaluation index threshold; wherein, the first step length is a fixed step length; the second step length is calculated based on the evaluation index value.

[0076] In this embodiment, the first condition is model convergence, gradient convergence, evaluation metric convergence, or parameter convergence. Model convergence refers to the overall performance of the model (such as the loss function value) gradually stabilizing during training, eventually reaching a relatively optimal and stable state. Gradient convergence focuses on the change of the gradient (partial derivative of the loss function with respect to the parameters) during the optimization process. Gradient convergence is considered to occur when the gradient gradually approaches zero or stabilizes within a very small range. Evaluation metric convergence refers to the evaluation metric value calculated from the model's predicted output gradually stabilizing during training and no longer changing significantly. Parameter convergence refers to the gradual stabilization of the values ​​of the model's various parameters during training and no longer changing significantly.

[0077] Specifically, when the performance of the reinforcement learning model in the previous training batch meets the preset first condition, the current target evaluation metric threshold is determined in the current training batch by increasing the threshold of the previous target evaluation metric. Specifically, the threshold of the previous target evaluation metric is increased by either the first step size or the second step size to obtain the target evaluation metric threshold.

[0078] S340. Determine whether the model performance of the previous training batch of the reinforcement learning model meets the preset second condition; if it does, then in the current training batch of the reinforcement learning model, decrease the threshold of the previous target evaluation index by the third step or the fourth step to obtain the target evaluation index threshold; wherein, the third step is a fixed step; the fourth step is calculated based on the evaluation index value.

[0079] In this scheme, the third step size is set to a fixed step size. For example, the third step size can be set to 0.05.

[0080] In this embodiment, the fourth step length is a dynamic step length. It can be calculated based on the correlation between the evaluation index value and the step length, or it can be calculated based on a preset calculation formula and the evaluation index value.

[0081] In this scheme, the second condition is either model variation, gradient variation, or parameter variation. Model variation refers to the performance degradation exhibited by the model during training. Gradient variation mainly involves abnormal changes in gradients during backpropagation, including gradient vanishing, gradient variation, gradient diffusion, and gradient instability. Parameter variation typically refers to abnormal changes in model parameters during training.

[0082] Specifically, when the performance of the reinforcement learning model in the previous training batch meets the preset second condition, the current target evaluation metric threshold is determined in the current training batch by decreasing the threshold of the previous target evaluation metric. Specifically, the previous target evaluation metric threshold is decreased by a third or fourth step to obtain the target evaluation metric threshold.

[0083] S350. Using the evaluation index value and the target evaluation index threshold, determine the reward function value and / or penalty function value, and use the reward function value and / or penalty function value to optimize the parameters of the reinforcement learning model.

[0084] The technical solution of this invention involves: selecting target training data from predetermined original training data based on scoring results; predicting the target training data using a reinforcement learning model, outputting prediction results, and calculating evaluation index values ​​based on the prediction results and preset correct results; determining a target evaluation index threshold based on model performance; determining reward function values ​​and / or penalty function values ​​using the evaluation index values ​​and the target evaluation index threshold; and optimizing the parameters of the reinforcement learning model using the reward function values ​​and / or penalty function values. By implementing this technical solution and setting dynamic evaluation indexes to train the reinforcement learning model, the stability and efficiency of model training can be significantly improved. This approach not only meets the high accuracy requirements of model training but also achieves progressive training by gradually introducing reinforcement training steps from simple to complex.

[0085] Example 4

[0086] Figure 4 This is a schematic diagram of another reinforcement learning model training method based on dynamic evaluation metrics provided in Embodiment 4 of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the process for determining the threshold of the target evaluation metric. Figure 4 As shown, the method includes:

[0087] S410. Select target training data from the predetermined original training data based on the scoring results; wherein, the scoring results are a conclusion formed after quantifying the prediction difficulty of the original training data.

[0088] S420. Based on the reinforcement learning model, predict the target training data, output the prediction result, and calculate the evaluation index value based on the prediction result and the preset correct result.

[0089] S430. Determine the training data in the previous training batch of the reinforcement learning model whose evaluation index value is greater than the target evaluation index threshold.

[0090] In this approach, training data whose evaluation metric values ​​exceed the target evaluation metric threshold are selected from the previous training batch of the reinforcement learning model. Specifically, training data with evaluation metric values ​​greater than the target evaluation metric threshold needs to be selected from the target training data.

[0091] S440. If the number or proportion of the training data is greater than a preset first threshold, then in the current training batch of the reinforcement learning model, the threshold of the previous target evaluation index is increased by the first step length or the second step length to obtain the target evaluation index threshold.

[0092] The first threshold can be flexibly set and adjusted according to the training requirements of the reinforcement learning model.

[0093] In this embodiment, it is determined whether the quantity or proportion of training data exceeds a preset first threshold. If the quantity or proportion of training data exceeds the preset first threshold, the current target evaluation index threshold is determined by increasing the previous target evaluation index threshold in the current training batch of the reinforcement learning model. Specifically, the previous target evaluation index threshold is increased by either a first step size or a second step size to obtain the target evaluation index threshold.

[0094] S450. If the number or proportion of the training data is less than a preset second threshold, then in the current training batch of the reinforcement learning model, the threshold of the previous target evaluation index is decreased by a third or fourth step to obtain the target evaluation index threshold; wherein, the second threshold is less than the first threshold.

[0095] The second threshold can be flexibly set and adjusted according to the training requirements of the reinforcement learning model. The second threshold is smaller than the first threshold.

[0096] In this embodiment, it is determined whether the quantity or proportion of training data is less than a preset second threshold. If the quantity or proportion of training data is less than the preset second threshold, then in the current training batch of the reinforcement learning model, the target evaluation index threshold is determined by decreasing the previous target evaluation index threshold. Specifically, the previous target evaluation index threshold is decreased by a third or fourth step to obtain the target evaluation index threshold.

[0097] S460. If the number or proportion of the training data is less than or equal to the first threshold and greater than or equal to the second threshold, then in the current training batch of the reinforcement learning model, the previous target evaluation index threshold is determined as the target evaluation index threshold.

[0098] Furthermore, when the number or proportion of training data is less than or equal to the first threshold and greater than or equal to the second threshold, it indicates that the model training effect is moderate. In the current training batch of the reinforcement learning model, the threshold of the previous target evaluation index is determined as the target evaluation index threshold.

[0099] S470. Using the evaluation index value and the target evaluation index threshold, determine the reward function value and / or penalty function value, and use the reward function value and / or penalty function value to optimize the parameters of the reinforcement learning model.

[0100] The technical solution of this invention involves: selecting target training data from predetermined original training data based on scoring results; predicting the target training data using a reinforcement learning model, outputting prediction results, and calculating evaluation index values ​​based on the prediction results and preset correct results; determining a target evaluation index threshold based on the number or proportion of training data; determining reward function values ​​and / or penalty function values ​​using the evaluation index values ​​and the target evaluation index threshold; and optimizing the parameters of the reinforcement learning model using the reward function values ​​and / or penalty function values. By implementing this technical solution and setting dynamic evaluation indexes to train the reinforcement learning model, the stability and efficiency of model training can be significantly improved. This approach not only meets the high accuracy requirements of model training but also achieves progressive training by gradually introducing reinforcement training steps from simple to complex.

[0101] Example 5

[0102] Figure 5 This is a flowchart illustrating the training process of a reinforcement learning model based on a dynamic evaluation metric, provided in Embodiment 5 of the present invention. The relationship between this embodiment and the previous embodiments is a detailed description of the target training data determination process. Figure 5 As shown, the method includes:

[0103] S510, Obtain the raw training data.

[0104] In this solution, raw training data can be obtained from a database; alternatively, raw training data can be collected using acquisition devices.

[0105] S520. Use a data evaluator to score the prediction difficulty of the original training data from the data type dimension and / or at least one evaluation angle dimension, and obtain at least one score.

[0106] Data evaluators are tools or systems used to analyze and evaluate data quality, validity, and applicability. Examples of data evaluators include Apache Griffin (focused on data quality assessment) and Great Expectations (used to define and validate data rules).

[0107] In this embodiment, the data type dimension covers text, images, audio, video, and programs; the evaluation perspective dimensions cover complexity, diversity, noise, knowledge content, annotation, contrast, intensity, and order of magnitude.

[0108] In this approach, for example, for the original training data of text types, scores can be given based on factors such as font, number of strokes, total number of characters, richness of text content, and presence of mathematical formulas. The more regular the font, the lower the score; fewer strokes, fewer characters, lower content richness, and the absence of mathematical formulas also result in a lower score.

[0109] In this embodiment, for example, the original training data for image types can be scored based on factors such as image contrast, brightness, number of pixels, relative size of the target in the image, signal-to-noise ratio, number of specific objects to be identified, and type. Specifically, the larger the relative size of the target in the image, the higher the signal-to-noise ratio, the fewer the number of targets, and the more uniform the type of target object, the lower the prediction difficulty of the original training data, and the lower the corresponding score.

[0110] The original training data for each sound type can be scored from the perspectives of sound intensity, signal-to-noise ratio, number of targets to be identified, and type of target object. Specifically, the stronger the sound intensity, the higher the signal-to-noise ratio, the fewer the number of targets, the more uniform the type of target object, and the lower the complexity of the target object (i.e., the more regular the sound production of the target object, for example, identifying an object that can only produce a single sound, such as a drum, is less difficult than identifying an object that can produce multiple sounds, such as a piano), the lower the prediction difficulty of the original training data, and the lower the corresponding score.

[0111] Furthermore, for the raw training data of video types, a comprehensive evaluation can be conducted from two dimensions: technology and content, and a data score can be given. At the level of technical parameters, key indicators such as image clarity, frame rate stability, and color accuracy need to be evaluated. At the level of content quality, the analysis should be carried out in combination with key elements such as content completeness, narrative fluency, and information value, and finally a comprehensive score can be formed.

[0112] In this embodiment, the original training data for the program type can be comprehensively evaluated and a data score can be given from the dimensions of code and performance: code includes dimensions such as code volume, code type, and number of function calls; performance includes technical dimensions such as running stability, response speed, and functional completeness.

[0113] Complexity refers to the degree of complexity of data structures, relationships, or information content. Examples include the difference between structured and unstructured data, and the distinction between single-dimensional and multi-dimensional information. When scoring the prediction difficulty of original training data based on complexity, the following criteria can be used: Low complexity (e.g., a single-field text list, a purely numerical table, with no relationships) scores 1-3 points. Medium complexity (e.g., multi-field structured tables, images with simple labels, and basic relationships such as time series) scores 4-6 points. High complexity (e.g., nested JSON, multimodal data such as text + image + audio, knowledge graphs composed of entity-relationship networks, and dynamic real-time data streams) scores 7-10 points.

[0114] In this embodiment, diversity refers to the richness of the scenarios, categories, sources, or features covered by the data. When scoring the prediction difficulty of the original training data based on diversity, the following criteria can be used: Single category / source (e.g., containing only user data from a specific region, images of a single category, etc., with highly concentrated data dimensions) scores 1-3 points. Medium diversity (covering 3-5 categories / sources, with some data differences but limited coverage dimensions, such as a text set containing 3 languages) scores 4-6 points. High diversity (covering 6 or more categories / sources, including extreme values ​​or edge scenarios, such as a comprehensive dataset spanning industries, regions, and time periods) scores 7-10 points.

[0115] In this scheme, noise refers to the proportion of irrelevant, erroneous, or interfering information in the data (the lower the noise, the lower the score). When scoring the prediction difficulty of the original training data based on noise, the following criteria can be used: High-noise data (noise ratio > 30%, such as a large number of missing values, mislabeled data, or duplicate data) scores 7-10 points. Medium-noise data (noise ratio 10%-30%, with some missing values ​​or mislabeled data) scores 4-6 points. Low-noise data (noise ratio < 10%, complete data cleaning, and extremely low error rate) scores 1-3 points.

[0116] In this context, "knowledge content" refers to the effective information, professional knowledge, or potential value contained in the data. When scoring the prediction difficulty of raw training data based on knowledge content, the following criteria can be used: Low knowledge content (containing only basic descriptive information, without in-depth insights, such as a pure list of IDs) scores 1-3 points. Medium knowledge content (containing extractable basic patterns, such as user behavior statistics and basic product parameters) scores 4-6 points. High knowledge content (containing professional domain knowledge, implicit correlations, or deep patterns that can be mined, such as scientific research experimental data, expert-annotated case analyses, and input data for industry trend prediction models) scores 7-10 points.

[0117] In this embodiment, annotations refer to the level of detail in the labels, descriptions, or metadata accompanying the data. When scoring the prediction difficulty of the original training data based on annotations, the following criteria can be used: No annotations or minimal annotations (only the data itself, without field descriptions, source, or meaning explanations) score 1-3 points. Medium annotations (containing basic metadata such as field names, data format, and collection time, but lacking detailed logical explanations) score 4-6 points. Detailed annotations (with complete documentation, including field meanings, data sources, processing logic, outlier explanations, and labeling rules, supporting direct reuse) score 7-10 points.

[0118] Specifically, contrast refers to the degree of significance of differences between different categories or features in data. When scoring the prediction difficulty of original training data based on contrast, the following criteria can be used: Low contrast (significant feature overlap, making it difficult to distinguish categories, such as blurry images or semantically similar text) scores 7-10 points. Medium contrast (some categories show significant differences, but there are blurry boundaries, such as product images with high similarity) scores 4-6 points. High contrast (clear category / feature differences with well-defined boundaries, such as clearly labeled positive and negative samples, or two sets of data with significant numerical differences) scores 1-3 points.

[0119] In this scheme, intensity refers to the density or impact of information conveyed by the data. When scoring the prediction difficulty of the original training data based on intensity, the following criteria can be used: Low intensity (sparse information, no timeliness, such as outdated non-core business data) scores 1-3 points. Medium intensity (moderate information density, with some timeliness, such as quarterly business reports, routine user feedback) scores 4-6 points. High intensity (dense and critical information, such as real-time transaction data, core indicator early warning data, experimental records of scientific breakthroughs) scores 7-10 points.

[0120] Here, "order of magnitude" refers to the size of the data. When scoring the prediction difficulty of the original training data based on its order of magnitude, the following criteria can be used: Small scale (e.g., < 1000 records, < 100KB) scores 1-3 points. Medium scale (1000-100,000 records, 100KB-10MB) scores 4-6 points. Large scale (> 100,000 records, > 10MB, or reaching big data levels) scores 7-10 points.

[0121] Specifically, by using multiple data evaluators, the prediction difficulty of the original training data is scored from the data type dimension and at least one evaluation perspective dimension, thereby obtaining at least one score.

[0122] S530. The at least one score is weighted to obtain the score result.

[0123] In this embodiment, different weights can be set for different data evaluators. The score of at least one data evaluator for the same original training data is multiplied by the corresponding weight to generate a score result.

[0124] S540. Select target training data from the predetermined original training data based on the scoring results; wherein, the scoring results are a conclusion formed after quantifying the prediction difficulty of the original training data.

[0125] In this embodiment, a screening threshold is set based on the scoring results, and the original training data whose scores meet the preset conditions are selected as the target training data.

[0126] Optionally, at least two scoring intervals are divided based on the scoring results, and the original training data located in the same scoring interval are obtained from low to high based on the at least two scoring intervals as the target training data; or, the scoring results are sorted, and the original training data are obtained from low to high based on the sorting of the scoring results corresponding to the original training data as the target training data.

[0127] In this embodiment, at least two scoring intervals are divided according to the scoring results. Then, the original training data within the same scoring interval is extracted in ascending order of these scoring intervals and used as the target training data.

[0128] In this scheme, after sorting the scoring results, the original training data is selected from low to high according to the sorting of the scoring results corresponding to the original training data, and used as the target training data.

[0129] Alternatively, all original training data can be used as target training data.

[0130] By selecting target training data from the original training data, the model's capabilities can be improved step by step. At the same time, this process helps to scientifically evaluate model performance, provides clear guidance for optimization, and thus improves model training efficiency.

[0131] S550. Based on the reinforcement learning model, predict the target training data, output the prediction result, and calculate the evaluation index value based on the prediction result and the preset correct result.

[0132] S560. Determine the target evaluation index threshold; wherein the target evaluation index threshold is dynamically adjusted based on the prediction process of the reinforcement learning model.

[0133] S570. Using the evaluation index value and the target evaluation index threshold, determine the reward function value and / or penalty function value, and use the reward function value and / or penalty function value to optimize the parameters of the reinforcement learning model.

[0134] The technical solution of this invention calculates a scoring result, then selects target training data from a predetermined pool of original training data based on the scoring result; predicts the target training data using a reinforcement learning model, outputs the prediction result, and calculates an evaluation index value based on the prediction result and a preset correct result; determines a target evaluation index threshold based on the number of training batches; and determines a reward function value and / or a penalty function value using the evaluation index value and the target evaluation index threshold, and optimizes the parameters of the reinforcement learning model using the reward function value and / or penalty function value. By implementing this technical solution and selecting target training data from the original training data, the model's capabilities can be gradually improved, the model's learning process optimized, and the model training efficiency increased.

[0135] Example 6

[0136] Figure 6 This is a schematic diagram illustrating the training process of a reinforcement learning model based on a dynamic evaluation metric, provided in Embodiment Six of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the process of using the target training data. Figure 6As shown, the method includes:

[0137] S610. Determine the training data in the previous training batch of the reinforcement learning model whose evaluation index value is greater than the target evaluation index threshold.

[0138] In this approach, training data whose evaluation metric values ​​exceed the target evaluation metric threshold are selected from the previous training batch of the reinforcement learning model. Specifically, training data with evaluation metric values ​​greater than the target evaluation metric threshold needs to be selected from the target training data.

[0139] S620. If the number or proportion of the training data is less than or equal to a preset third threshold, then in the current training batch of the reinforcement learning model, other original training data that are located in the same scoring interval as the target training data of the previous training batch are determined as the target training data of the current training batch.

[0140] The third threshold can be flexibly set and adjusted according to the training requirements of the reinforcement learning model.

[0141] In this embodiment, it is determined whether the number or proportion of training data is less than or equal to a preset third threshold. If the number or proportion of training data is less than or equal to the preset third threshold, then in the current training batch of the reinforcement learning model, other original training data that are in the same scoring interval as the target training data of the previous training batch are determined as the target training data of the current training batch.

[0142] S630. If the number or proportion of the training data is greater than a preset third threshold, then in the current training batch of the reinforcement learning model, the original training data in the score interval that is higher than the score interval of the target training data in the previous training batch will be used as the target training data for the current training batch.

[0143] In this embodiment, it is determined whether the number or proportion of training data is greater than a preset third threshold. If the number or proportion of training data is greater than the preset third threshold, then in the current training batch of the reinforcement learning model, the portion of the original training data whose score interval is higher than the score interval of the target training data in the previous training batch is selected as the target training data of the current training batch.

[0144] S640. Based on the reinforcement learning model, predict the target training data, output the prediction result, and calculate the evaluation index value based on the prediction result and the preset correct result.

[0145] S650. Determine the target evaluation index threshold; wherein the target evaluation index threshold is dynamically adjusted based on the prediction process of the reinforcement learning model.

[0146] S660. Using the evaluation index value and the target evaluation index threshold, determine the reward function value and / or penalty function value, and use the reward function value and / or penalty function value to optimize the parameters of the reinforcement learning model.

[0147] The technical solution of this invention involves: screening target training data; predicting the target training data using a reinforcement learning model, outputting prediction results, and calculating evaluation index values ​​based on the prediction results and preset correct results; determining a target evaluation index threshold; using the evaluation index values ​​and the target evaluation index threshold to determine reward function values ​​and / or penalty function values; and using the reward function values ​​and / or penalty function values ​​to optimize the parameters of the reinforcement learning model. By implementing this technical solution and screening target training data, the model's capabilities can be gradually improved, the model's learning process optimized, and the model training efficiency increased.

[0148] Example 7

[0149] Figure 7 This is a schematic diagram illustrating another reinforcement learning model training process based on a dynamic evaluation metric, provided in Embodiment Seven of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the process of using the target training data. Figure 7 As shown, the method includes:

[0150] S710. If the model performance of the reinforcement learning model in the previous training batch meets the preset third condition, then in the current training batch of the reinforcement learning model, the original training data whose score result is higher than the target training data score result of the previous training batch shall be used as the target training data of the current training batch.

[0151] In this embodiment, the third condition is model convergence, gradient convergence, evaluation metric convergence, or parameter convergence. Model convergence refers to the overall performance of the model (such as the loss function value) gradually stabilizing during training, eventually reaching a relatively optimal and stable state. Gradient convergence focuses on the change of the gradient (partial derivative of the loss function with respect to the parameters) during the optimization process. Gradient convergence is considered to occur when the gradient gradually approaches zero or stabilizes within a very small range. Evaluation metric convergence refers to the evaluation metric value calculated from the model's predicted output gradually stabilizing during training and no longer changing significantly. Parameter convergence refers to the gradual stabilization of the values ​​of the model's various parameters during training and no longer changing significantly.

[0152] Specifically, when the performance of the reinforcement learning model in the previous training batch meets the preset third condition, in the current training batch of the reinforcement learning model, the portion of the original training data whose score is higher than the score of the target training data in the previous training batch is selected as the target training data for the current training batch.

[0153] S720. If the model performance of the reinforcement learning model in the previous training batch meets the preset fourth condition, then in the current training batch of the reinforcement learning model, the original data whose score result is not higher than the target training data score result of the previous training batch shall be used as the target training data of the current training batch.

[0154] In this scheme, the fourth condition is model variation, gradient variation, evaluation metric variation, or parameter variation. Model variation refers to the performance degradation exhibited by the model during training. Gradient variation mainly involves abnormal changes in gradients during backpropagation, including gradient vanishing, gradient variation, gradient diffusion, and gradient instability. Evaluation metric variation refers to the number or proportion of evaluation metric values ​​falling below the target evaluation metric threshold exceeding a certain threshold during training. Parameter variation typically refers to abnormal changes in model parameters during training.

[0155] Specifically, when the performance of the reinforcement learning model in the previous training batch meets the preset fourth condition, in the current training batch of the reinforcement learning model, the portion of the original training data whose score is not higher than the score of the target training data in the previous training batch is used as the target training data for the current training batch.

[0156] More specifically, when the performance of the reinforcement learning model in the previous training batch meets the preset fifth condition, in the current training batch of the reinforcement learning model, the portion of the original training data whose score is lower than the score of the target training data in the previous training batch is used as the target training data for the current training batch. Here, the preset fifth condition is lower than the preset fourth condition, meaning that the model performance under the preset fifth condition is worse than the model performance under the preset fourth condition.

[0157] S730. Based on the reinforcement learning model, predict the target training data, output the prediction result, and calculate the evaluation index value based on the prediction result and the preset correct result.

[0158] S740. Determine the target evaluation index threshold; wherein the target evaluation index threshold is dynamically adjusted based on the prediction process of the reinforcement learning model.

[0159] S750. Using the evaluation index value and the target evaluation index threshold, determine the reward function value and / or penalty function value, and use the reward function value and / or penalty function value to optimize the parameters of the reinforcement learning model.

[0160] The technical solution of this invention involves: screening target training data; predicting the target training data using a reinforcement learning model, outputting prediction results, and calculating evaluation index values ​​based on the prediction results and preset correct results; determining a target evaluation index threshold; using the evaluation index values ​​and the target evaluation index threshold to determine reward function values ​​and / or penalty function values; and using the reward function values ​​and / or penalty function values ​​to optimize the parameters of the reinforcement learning model. By implementing this technical solution and screening target training data, the model's capabilities can be gradually improved, the model's learning process optimized, and the model training efficiency increased.

[0161] Example 8

[0162] Figure 8 This is a schematic diagram illustrating the training process of another reinforcement learning model based on a dynamic evaluation metric, provided in Embodiment 8 of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the process of using the target training data. Figure 8 As shown, the method includes:

[0163] S810. Determine the training data in the previous training batch of the reinforcement learning model whose evaluation index value is greater than the target evaluation index threshold.

[0164] In this approach, training data whose evaluation metric values ​​exceed the target evaluation metric threshold are selected from the previous training batch of the reinforcement learning model. Specifically, training data with evaluation metric values ​​greater than the target evaluation metric threshold needs to be selected from the target training data.

[0165] S820. If the number or proportion of the training data is greater than the preset fourth threshold, then in the current training batch of the reinforcement learning model, the original training data with a score higher than that of the target training data in the previous training batch will be used as the target training data for the current training batch.

[0166] The fourth threshold can be flexibly set and adjusted according to the training requirements of the reinforcement learning model.

[0167] Specifically, it is determined whether the number or proportion of training data is greater than the preset fourth threshold. If the number or proportion of training data is greater than the preset fourth threshold, then in the current training batch of the reinforcement learning model, the part of the original training data whose score result is higher than the target training data score result of the previous training batch is selected as the target training data of the current training batch.

[0168] S830. If the number or proportion of the training data is less than or equal to a preset fifth threshold, then in the current training batch of the reinforcement learning model, the original training data whose score result is lower than the target training data score result of the previous training batch will be used as the target training data of the current training batch.

[0169] The fifth threshold can be flexibly set and adjusted according to the training requirements of the reinforcement learning model. The fifth threshold is lower than the fourth threshold.

[0170] Specifically, it determines whether the number or proportion of training data is less than or equal to a preset fifth threshold. If the number or proportion of training data is less than or equal to the preset fifth threshold, then in the current training batch of the reinforcement learning model, the portion of the original training data whose score is lower than the target training data score of the previous training batch is selected as the target training data for the current training batch.

[0171] S840. If the number or proportion of the training data is greater than the fifth threshold and less than or equal to the fourth threshold, then in the current training batch of the reinforcement learning model, the original training data whose score result is equal to the target training data score result of the previous training batch is determined as the target training data of the current training batch.

[0172] In this scheme, when the number or proportion of training data is greater than the fifth threshold and less than or equal to the fourth threshold, the original training data whose scoring result is equal to the target training data scoring result of the previous training batch is determined as the target training data of the current training batch of the reinforcement learning model.

[0173] S850. Based on the reinforcement learning model, predict the target training data, output the prediction result, and calculate the evaluation index value based on the prediction result and the preset correct result.

[0174] S860. Determine the target evaluation index threshold; wherein the target evaluation index threshold is dynamically adjusted based on the prediction process of the reinforcement learning model.

[0175] S870. Using the evaluation index value and the target evaluation index threshold, determine the reward function value and / or penalty function value, and use the reward function value and / or penalty function value to optimize the parameters of the reinforcement learning model.

[0176] The technical solution of this invention involves: screening target training data; predicting the target training data using a reinforcement learning model, outputting prediction results, and calculating evaluation index values ​​based on the prediction results and preset correct results; determining a target evaluation index threshold; using the evaluation index values ​​and the target evaluation index threshold to determine reward function values ​​and / or penalty function values; and using the reward function values ​​and / or penalty function values ​​to optimize the parameters of the reinforcement learning model. By implementing this technical solution and screening target training data, the model's capabilities can be gradually improved, the model's learning process optimized, and the model training efficiency increased.

[0177] Example 9

[0178] Figure 9This is a schematic diagram of a reinforcement learning model training device based on a dynamic evaluation index, provided in Embodiment 9 of the present invention. Figure 9 As shown, the device includes:

[0179] The target training data filtering module 910 is used to filter target training data from predetermined original training data based on the scoring results; wherein, the scoring results are a conclusion formed after quantifying the prediction difficulty of the original training data;

[0180] The evaluation index value calculation module 920 is used to predict the target training data based on the reinforcement learning model, output the prediction result, and calculate the evaluation index value based on the prediction result and the preset correct result.

[0181] The target evaluation index threshold determination module 930 is used to determine the target evaluation index threshold; wherein, the target evaluation index threshold is dynamically adjusted based on the prediction process of the reinforcement learning model;

[0182] The parameter optimization module 940 is used to determine the reward function value and / or penalty function value using the evaluation index value and the target evaluation index threshold, and to optimize the parameters of the reinforcement learning model using the reward function value and / or penalty function value.

[0183] Optional, the target evaluation index threshold determination module 930 is specifically used for:

[0184] The number of training batches of the reinforcement learning model at the threshold of the previous objective evaluation metric is detected.

[0185] If the number of training batches is less than the preset number of times threshold, then the previous target evaluation index threshold is determined as the target evaluation index threshold.

[0186] If the number of training batches is greater than or equal to a preset threshold, the threshold of the previous target evaluation index is increased by either the first step length or the second step length to obtain the target evaluation index threshold; wherein, the first step length is a fixed step length; and the second step length is calculated based on the evaluation index value.

[0187] Optionally, the target evaluation index threshold determination module 930 is also used for:

[0188] Determine whether the performance of the reinforcement learning model in the previous training batch meets the preset first condition; if it does, then in the current training batch of the reinforcement learning model, increase the threshold of the previous target evaluation index by a first step length or a second step length to obtain the target evaluation index threshold; wherein, the first step length is a fixed step length; the second step length is calculated based on the evaluation index value;

[0189] Determine whether the performance of the reinforcement learning model in the previous training batch meets the preset second condition; if it does, then in the current training batch of the reinforcement learning model, decrease the threshold of the previous target evaluation index by a third step or a fourth step to obtain the target evaluation index threshold; wherein, the third step is a fixed step; and the fourth step is calculated based on the evaluation index value.

[0190] Optionally, the target evaluation index threshold determination module 930 is also used for:

[0191] Identify the training data in the previous training batch of the reinforcement learning model whose evaluation index value is greater than the target evaluation index threshold;

[0192] If the number or proportion of the training data is greater than the preset first threshold, then in the current training batch of the reinforcement learning model, the threshold of the previous target evaluation index is increased by the first step length or the second step length to obtain the target evaluation index threshold.

[0193] If the number or proportion of training data is less than a preset second threshold, then in the current training batch of the reinforcement learning model, the threshold of the previous target evaluation index is decreased by a third or fourth step to obtain the target evaluation index threshold; wherein, the second threshold is less than the first threshold.

[0194] If the number or proportion of the training data is less than or equal to the first threshold and greater than or equal to the second threshold, then in the current training batch of the reinforcement learning model, the previous target evaluation index threshold is determined as the target evaluation index threshold.

[0195] Optionally, the device further includes:

[0196] The raw training data acquisition module is used to acquire raw training data;

[0197] The scoring module is used to score the prediction difficulty of the original training data from the data type dimension and / or at least one evaluation angle dimension using a data evaluator, and obtain at least one score;

[0198] The scoring result acquisition module is used to perform a weighted calculation on the at least one score to obtain the scoring result.

[0199] Optional, the target training data filtering module 910 includes:

[0200] A target training data filtering unit is configured to divide the scoring results into at least two scoring intervals, and to obtain the original training data located in the same scoring interval from low to high according to the at least two scoring intervals as the target training data; or,

[0201] The scoring results are sorted, and the original training data is obtained from low to high based on the sorting of the scoring results corresponding to the original training data as the target training data.

[0202] Optional, target training data filtering unit, specifically used for:

[0203] Identify the training data in the previous training batch of the reinforcement learning model whose evaluation index value is greater than the target evaluation index threshold;

[0204] If the number or proportion of the training data is less than or equal to a preset third threshold, then in the current training batch of the reinforcement learning model, other original training data that are located in the same scoring interval as the target training data of the previous training batch will be determined as the target training data of the current training batch.

[0205] If the number or proportion of the training data is greater than a preset third threshold, then in the current training batch of the reinforcement learning model, the original training data in the score interval that is higher than the score interval of the target training data in the previous training batch will be used as the target training data for the current training batch.

[0206] Optional, target training data filtering unit, specifically used for:

[0207] If the performance of the reinforcement learning model in the previous training batch meets the preset third condition, then in the current training batch of the reinforcement learning model, the original training data whose score result is higher than the target training data score result of the previous training batch will be used as the target training data of the current training batch.

[0208] If the model performance of the reinforcement learning model in the previous training batch meets the preset fourth condition, then in the current training batch of the reinforcement learning model, the original data whose score result is not higher than the target training data score result of the previous training batch will be used as the target training data of the current training batch.

[0209] Optional, target training data filtering unit, specifically used for:

[0210] Identify the training data in the previous training batch of the reinforcement learning model whose evaluation index value is greater than the target evaluation index threshold;

[0211] If the number or proportion of the training data is greater than the preset fourth threshold, then in the current training batch of the reinforcement learning model, the original training data with a score higher than that of the target training data in the previous training batch will be used as the target training data for the current training batch.

[0212] If the number or proportion of the training data is less than or equal to the preset fifth threshold, then in the current training batch of the reinforcement learning model, the original training data whose score result is lower than the target training data score result of the previous training batch will be used as the target training data of the current training batch.

[0213] If the number or proportion of training data is greater than the fifth threshold and less than or equal to the fourth threshold, then in the current training batch of the reinforcement learning model, the original training data whose score result is equal to the target training data score result of the previous training batch is determined as the target training data of the current training batch.

[0214] The reinforcement learning model training device based on dynamic evaluation index provided in this embodiment of the invention can execute the reinforcement learning model training method based on dynamic evaluation index provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0215] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0216] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A reinforcement learning model training method based on dynamic evaluation metrics, characterized in that, include: Target training data is selected from the predetermined original training data based on the scoring results; wherein, the scoring results are a conclusion formed after quantifying the prediction difficulty of the original training data; the original training data is image-based original data; The target training data is predicted based on the reinforcement learning model, the prediction result is output, and the evaluation index value is calculated based on the prediction result and the preset correct result. Determine the target evaluation index threshold; wherein, the target evaluation index threshold is dynamically adjusted based on the prediction process of the reinforcement learning model; Using the evaluation index value and the target evaluation index threshold, determine the reward function value and / or penalty function value, and use the reward function value and / or penalty function value to optimize the parameters of the reinforcement learning model; The method further includes: Identify the training data in the previous training batch of the reinforcement learning model whose evaluation index value is greater than the target evaluation index threshold; If the number or proportion of the training data is greater than the preset first threshold, then in the current training batch of the reinforcement learning model, the threshold of the previous target evaluation index is increased by the first step length or the second step length to obtain the target evaluation index threshold. If the number or proportion of training data is less than a preset second threshold, then in the current training batch of the reinforcement learning model, the threshold of the previous target evaluation index is decreased by a third or fourth step to obtain the target evaluation index threshold; wherein, the second threshold is less than the first threshold. If the number or proportion of the training data is less than or equal to the first threshold and greater than or equal to the second threshold, then in the current training batch of the reinforcement learning model, the previous target evaluation index threshold is determined as the target evaluation index threshold.

2. The method according to claim 1, characterized in that, Determine the threshold values ​​for the target evaluation indicators, including: The number of training batches of the reinforcement learning model at the threshold of the previous objective evaluation metric is detected. If the number of training batches is less than the preset number of times threshold, then the previous target evaluation index threshold is determined as the target evaluation index threshold. If the number of training batches is greater than or equal to a preset threshold, the threshold of the previous target evaluation index is increased by either the first step length or the second step length to obtain the target evaluation index threshold; wherein, the first step length is a fixed step length; and the second step length is calculated based on the evaluation index value.

3. The method according to claim 1, characterized in that, Determine the threshold values ​​for the target evaluation indicators, including: Determine whether the performance of the reinforcement learning model in the previous training batch meets the preset first condition; if it does, then in the current training batch of the reinforcement learning model, increase the threshold of the previous target evaluation index by a first step length or a second step length to obtain the target evaluation index threshold; wherein, the first step length is a fixed step length; the second step length is calculated based on the evaluation index value; Determine whether the performance of the reinforcement learning model in the previous training batch meets the preset second condition; if it does, then in the current training batch of the reinforcement learning model, decrease the threshold of the previous target evaluation index by a third step or a fourth step to obtain the target evaluation index threshold; wherein, the third step is a fixed step; and the fourth step is calculated based on the evaluation index value.

4. The method according to claim 1, characterized in that, Before selecting target training data from the predetermined raw training data based on the scoring results, the following steps are also included: Obtain the raw training data; The prediction difficulty of the original training data is scored using a data evaluator from the data type dimension and / or at least one evaluation angle dimension, to obtain at least one score; The at least one score is weighted to obtain the score result.

5. The method according to claim 4, characterized in that, Based on the scoring results, target training data is selected from the predetermined original training data, including: Divide the scoring results into at least two scoring intervals, and obtain the original training data located in the same scoring interval from low to high according to the at least two scoring intervals as the target training data; or, The scoring results are sorted, and the original training data is obtained from low to high based on the sorting of the scoring results corresponding to the original training data as the target training data.

6. The method according to claim 5, characterized in that, Based on the at least two rating intervals, the original training data located within the same rating interval, ranked from low to high, are obtained as the target training data, including: Identify the training data in the previous training batch of the reinforcement learning model whose evaluation index value is greater than the target evaluation index threshold; If the number or proportion of the training data is less than or equal to a preset third threshold, then in the current training batch of the reinforcement learning model, other original training data that are located in the same scoring interval as the target training data of the previous training batch will be determined as the target training data of the current training batch. If the number or proportion of the training data is greater than a preset third threshold, then in the current training batch of the reinforcement learning model, the original training data in the score interval that is higher than the score interval of the target training data in the previous training batch will be used as the target training data for the current training batch.

7. The method according to claim 5, characterized in that, Based on the ranking of the scoring results corresponding to the original training data, the original training data is obtained from low to high as the target training data, including: If the performance of the reinforcement learning model in the previous training batch meets the preset third condition, then in the current training batch of the reinforcement learning model, the original training data whose score result is higher than the target training data score result of the previous training batch will be used as the target training data of the current training batch. If the model performance of the reinforcement learning model in the previous training batch meets the preset fourth condition, then in the current training batch of the reinforcement learning model, the original data whose score result is not higher than the target training data score result of the previous training batch will be used as the target training data of the current training batch.

8. The method according to claim 5, characterized in that, Based on the ranking of the scoring results corresponding to the original training data, the original training data is obtained from low to high as the target training data, including: Identify the training data in the previous training batch of the reinforcement learning model whose evaluation index value is greater than the target evaluation index threshold; If the number or proportion of the training data is greater than the preset fourth threshold, then in the current training batch of the reinforcement learning model, the original training data with a score higher than that of the target training data in the previous training batch will be used as the target training data for the current training batch. If the number or proportion of the training data is less than or equal to the preset fifth threshold, then in the current training batch of the reinforcement learning model, the original training data whose score result is lower than the target training data score result of the previous training batch will be used as the target training data of the current training batch. If the number or proportion of training data is greater than the fifth threshold and less than or equal to the fourth threshold, then in the current training batch of the reinforcement learning model, the original training data whose score result is equal to the target training data score result of the previous training batch is determined as the target training data of the current training batch.

9. A reinforcement learning model training device based on dynamic evaluation metrics, characterized in that, include: The target training data filtering module is used to filter target training data from predetermined raw training data based on the scoring results; wherein, the scoring results are a conclusion formed after quantifying the prediction difficulty of the raw training data; and the raw training data is image-type raw data. The evaluation index value calculation module is used to predict the target training data based on the reinforcement learning model, output the prediction result, and calculate the evaluation index value based on the prediction result and the preset correct result. The target evaluation index threshold determination module is used to determine the target evaluation index threshold; wherein, the target evaluation index threshold is dynamically adjusted based on the prediction process of the reinforcement learning model; The parameter optimization module is used to determine the reward function value and / or penalty function value using the evaluation index value and the target evaluation index threshold, and to optimize the parameters of the reinforcement learning model using the reward function value and / or penalty function value. The target evaluation index threshold determination module is also used for: Identify the training data in the previous training batch of the reinforcement learning model whose evaluation index value is greater than the target evaluation index threshold; If the number or proportion of the training data is greater than the preset first threshold, then in the current training batch of the reinforcement learning model, the threshold of the previous target evaluation index is increased by the first step length or the second step length to obtain the target evaluation index threshold. If the number or proportion of training data is less than a preset second threshold, then in the current training batch of the reinforcement learning model, the threshold of the previous target evaluation index is decreased by a third or fourth step to obtain the target evaluation index threshold; wherein, the second threshold is less than the first threshold. If the number or proportion of the training data is less than or equal to the first threshold and greater than or equal to the second threshold, then in the current training batch of the reinforcement learning model, the previous target evaluation index threshold is determined as the target evaluation index threshold.

Images