An automated evaluation system and method for reinforcement learning algorithm models
Through automated evaluation systems and methods, combined with real environment and simulation environment, multi-dimensional performance indicators are used to evaluate reinforcement learning algorithm models, the problem of insufficient evaluation of single indicators in the existing technology is solved, and a comprehensive evaluation and dynamic of the model in complex environments is achieved. Reliability guarantee in scenarios.
Patent Information
- Application Number
- CN202310914664.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-07-24
AI Technical Summary
The existing reinforcement learning algorithm model evaluation methods mainly rely on single or statistical indicators, and cannot comprehensively evaluate the performance of the model in complex environments, especially when facing dynamically changing scenarios, it cannot ensure the reliability and safety of the model in specific scenarios. sex.
Provide an automated evaluation system and method for reinforcement learning algorithm models. By building a real environment and simulation environment, combining the baseline algorithm model, comprehensive performance evaluation is carried out, the scope of application and boundary conditions of the model are evaluated, and multi-dimensional performance indicators (availability, optimization Inefficiency, agility and stability) are evaluated, and the evaluation indicator weights are adaptively adjusted according to user needs.
A comprehensive and rapid evaluation of reinforcement learning algorithm models in new scenarios is achieved, and the comprehensive performance evaluation and capability improvement value of the model is provided, ensuring that the model's performance in dynamically changing scenarios meets user expectations, and improving the accuracy and reliability of the evaluation. sex.
Smart Images

Figure CN116861795B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of reinforcement model performance evaluation, and in particular to an automated evaluation system and method for reinforcement learning algorithm models. Background Art
[0002] Intelligent algorithms based on reinforcement learning have achieved extraordinary achievements in the fields of automatic control, human-computer interaction, and game-based confrontation games, and are widely used. Reinforcement learning algorithms are mainly used to deal with large or even continuous state space and action space problems. In such applications, the reinforcement learning agent uses the feedback of the simulation environment as the model algorithm of the agent input. It mainly optimizes the agent through interaction and trial and error between the agent and the environment, and uses evaluative feedback signals such as reward functions. However, since the constructed simulation environment cannot restore the real world 100%, when "unexpected" changes occur in the real world, the originally trained agent cannot adapt well to the new changed scene. Therefore, conducting evaluations of artificial intelligence algorithms is of great significance to ensure that artificial intelligence technology is easy to use, safe, reliable, and controllable.
[0003] At present, the evaluation of reinforcement learning algorithm models is often based on a single or statistical indicator, and the model with better overall performance is better. However, when facing scenarios with extremely high requirements for model performance, such as security drills and red-blue game confrontation games, since both parties are always in a game confrontation with each other, even if a 99% winning rate has been achieved, it is difficult to ensure that 1% of failures will not occur in a specific game confrontation scenario. Therefore, in practical applications under complex environments, a more comprehensive evaluation method is urgently needed to automatically and quickly evaluate the performance of reinforcement learning algorithm models. Summary of the invention
[0004] The present application provides an automated evaluation system and method for reinforcement learning algorithm models, which can evaluate the applicable scope of the algorithm model and its capabilities under boundary conditions, and conduct comprehensive performance evaluation, providing a comprehensive and rapid evaluation method for intelligent algorithms based on reinforcement learning technology.
[0005] The first aspect of the embodiment of the present application provides an automated evaluation system for a reinforcement learning algorithm model, the system comprising:
[0006] An interface module, used to communicate with the real application system to receive real environment data required to build a new application simulation environment;
[0007] The data upload module is used to upload the reinforcement learning model to be evaluated and the corresponding sample environment data and sample data;
[0008] A simulation environment construction module, used to construct a first simulation environment and a second simulation environment, wherein the first simulation environment is a simulation environment for training the reinforcement learning model to be evaluated reconstructed according to the sample environment data, and the second simulation environment is a new application simulation environment constructed according to the real environment data;
[0009] A model retest module, used to integrate the reinforcement learning model to be tested into the first simulation environment, and verify the function of the reinforcement learning model to be tested using the sample data;
[0010] A baseline algorithm model management module is used to store and manage a number of baseline algorithm models, any of which is marked with a corresponding scope of application and boundary conditions;
[0011] A performance testing module is used to test the performance of the reinforcement learning model to be evaluated and the selected baseline algorithm model in the new application simulation environment to obtain corresponding test scores;
[0012] The model evaluation module is used to calculate the capability value of the reinforcement learning model to be evaluated relative to the baseline algorithm model based on the test score, so as to realize the automatic evaluation of the reinforcement learning model to be evaluated.
[0013] Optionally, the model retest module includes:
[0014] A calling test submodule is used to call the reinforcement learning model to be evaluated and the corresponding sample environment data from the data uploading module, integrate the reinforcement learning model to be evaluated into the first simulation environment, and detect whether the reinforcement learning model to be evaluated can be called normally;
[0015] The running test submodule is used to call the sample data corresponding to the above-mentioned reinforcement learning model to be evaluated from the above-mentioned data upload module after the above-mentioned reinforcement learning model to be evaluated is successfully called, input the input data in the above-mentioned sample data into the above-mentioned reinforcement learning model to be evaluated for processing, obtain the running result data, and compare the above-mentioned running result data with the corresponding output data in the sample data to verify the function of the above-mentioned reinforcement learning model to be evaluated.
[0016] Optionally, the above performance test module includes:
[0017] The indicator determination submodule is used to determine the performance indicators used for evaluation, where the performance indicators are one or more of availability, quality, agility and stability;
[0018] A baseline algorithm model selection submodule is used to select and call a baseline algorithm model whose applicable scope and boundary conditions meet the requirements from the baseline algorithm model management module according to the requirements of the new application simulation environment;
[0019] The model testing submodule is used to test the performance of the reinforcement learning model to be evaluated and the called baseline algorithm model in the new application simulation environment to obtain the performance scores of the corresponding performance indicators;
[0020] The score calculation submodule is used to perform comprehensive calculations based on the above performance scores to obtain the test scores of the above reinforcement learning model to be evaluated and the called baseline algorithm model.
[0021] Optionally, the above performance test module is also used to select a working mode of the evaluation, including:
[0022] The first working mode is used to perform corresponding evaluation according to one or more performance indicators selected by the user, obtain a test score, and compare it with the test score of the above-mentioned baseline algorithm model;
[0023] The second working mode is used to automatically complete the comprehensive evaluation of the above-mentioned four performance indicators of availability, pros and cons, agility and stability, obtain the test score, and compare it with the test score of the above-mentioned baseline algorithm model.
[0024] Optionally, the above model testing submodule includes:
[0025] The availability score calculation unit is used to call the real environment data in the above interface module, input it into the above reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding output result, and count the success rate of the above reinforcement learning model / baseline algorithm model to be evaluated according to the preset task objectives and the output results, and calculate the corresponding availability score according to the success rate:
[0026]
[0027] Among them, P1 represents the availability score, p represents the success rate, and a represents the preset standard value;
[0028] The merit score calculation unit is used to call the real environment data in the above interface module, input it into the above reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding output reward value, and obtain the merit score by calculating the average reward value:
[0029]
[0030] Among them, P2 represents the superiority score, N represents the number of real environment data, and M n Represents the reward value corresponding to the nth set of real environment data;
[0031] The agility score calculation unit is used to call the real environment data in the above interface module, input it into the above reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding average running time, and calculate the agility score according to the average running time and the preset task requirement time:
[0032]
[0033] Among them, P3 represents the agility score, t1 represents the average running time, and t2 represents the preset task requirement time;
[0034] The stability score calculation unit is used to call the real environment data in the above interface module, input it into the above reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding output result, and count the number of successes of the above reinforcement learning model / baseline algorithm model to be evaluated according to the preset task objectives and the output results, and calculate the corresponding stability score according to the number of successes:
[0035]
[0036] Among them, P4 represents the stability score, N represents the number of real environment data, and n represents the number of successes.
[0037] Optionally, the score calculation submodule includes:
[0038] The test scores of the reinforcement learning model to be evaluated and the called baseline algorithm model are calculated in the following way:
[0039]
[0040] Among them, H represents the comprehensive test score, P i represents the performance score of the i-th performance indicator, W i It represents the weight coefficient of the i-th performance indicator preset by the user, and m represents the number of performance indicators.
[0041] Optionally, the above model evaluation module includes:
[0042] The capability value calculation submodule is used to calculate the capability value of the reinforcement learning model to be evaluated relative to the baseline algorithm model based on the test score in the following manner:
[0043]
[0044] Among them, A represents the capability value of the reinforcement learning model to be evaluated, and H 测 represents the test score of the reinforcement learning model to be evaluated, H 基 Represents the test score of the baseline algorithm model;
[0045] The model updating submodule is used to save the above-mentioned reinforcement learning model to be evaluated whose capability value is greater than or equal to the preset threshold into the above-mentioned baseline algorithm model management module, and update the above-mentioned baseline algorithm model.
[0046] A second aspect of the embodiment of the present application provides an automated evaluation method for a reinforcement learning algorithm model, the method comprising:
[0047] Obtaining real environment data required for building a new application simulation environment, a reinforcement learning model to be evaluated, and corresponding sample environment data and sample data, wherein the reinforcement learning model to be evaluated is obtained by training the sample data in the sample environment;
[0048] Constructing a first simulation environment and a second simulation environment, wherein the first simulation environment is a simulation environment for training the reinforcement learning model to be evaluated reconstructed according to the sample environment data, and the second simulation environment is a new application simulation environment constructed according to the real environment data;
[0049] Integrating the reinforcement learning model to be tested into the first simulation environment, and retesting the function of the reinforcement learning model to be tested using the sample data;
[0050] According to the requirements of the new application simulation environment, a baseline algorithm model with applicable scope and boundary conditions meeting the requirements is selected from a preset baseline algorithm model management library;
[0051] According to the above real environment data, the performance of the above reinforcement learning model to be evaluated and the selected baseline algorithm model in the above new application simulation environment are tested respectively to obtain corresponding test scores;
[0052] Based on the above test scores, the capability values of the above reinforcement learning model to be evaluated relative to the above baseline algorithm model are calculated respectively to realize automatic evaluation of the above reinforcement learning model to be evaluated.
[0053] Optionally, the performance of the reinforcement learning model to be evaluated and the selected baseline algorithm model in the new application simulation environment are tested according to the real environment data to obtain corresponding test scores, including:
[0054] Determine the performance indicators used for the evaluation, where the performance indicators are one or more of availability, quality, agility and stability;
[0055] Respectively test the performance of the reinforcement learning model to be evaluated and the baseline algorithm model in the new application simulation environment to obtain the performance scores of the corresponding performance indicators;
[0056] Based on the above performance scores, a comprehensive calculation is performed to obtain the test scores of the above reinforcement learning model to be evaluated and the above baseline algorithm model.
[0057] Optionally, the method further includes selecting a working mode of the evaluation:
[0058] When the first working mode is selected, a corresponding evaluation is performed according to one or more performance indicators selected by the user to obtain a test score, and the score is compared with the test score of the baseline algorithm model;
[0059] When the second working mode is selected, a comprehensive evaluation of the above four performance indicators of availability, pros and cons, agility and stability is automatically completed to obtain a test score, which is then compared with the test score of the above baseline algorithm model.
[0060] Compared with the prior art, this application has the following advantages:
[0061] The embodiments of the present application evaluate the applicable scope of the algorithm model and its capabilities under boundary conditions, and provide measured values for various evaluations of the algorithm model, as well as the capability improvement value of the reinforcement learning model to be evaluated relative to the baseline algorithm model, so as to determine whether the performance of the reinforcement learning model to be evaluated in the new scenario can meet user expectations, thereby providing a convenient solution for the rapid evaluation of intelligent algorithms based on reinforcement learning technology.
[0062] The embodiment of the present application designs two working modes for users to choose when evaluating the reinforcement learning model to be evaluated. The first working mode is used when the user has limited time to evaluate the model or has his own evaluation direction. He can choose one or more evaluation angles from the four angles of model availability, pros and cons, agility and stability, and compare the evaluation results with the evaluation results of the baseline algorithm model; the second working mode is used when the user has sufficient time to evaluate the model or wants to evaluate the model more comprehensively. It can automatically complete the comprehensive evaluation of the model's availability, pros and cons, agility and stability from multiple angles with one click, and compare the comprehensive evaluation results with the evaluation results of the baseline algorithm model.
[0063] The embodiment of this application introduces model evaluation technology and weighting mechanism for dynamically changing scenarios, combines the comprehensive indicators of the model with the application scenarios, and systematically and comprehensively evaluates the model results. At the same time, during the evaluation process, according to the actual needs of users, you can choose the evaluation indicators and adaptively adjust the weight of each evaluation indicator to obtain a model that is closer to user needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] Figure 1 It is a structural diagram of an automated evaluation system for a reinforcement learning algorithm model proposed in an embodiment of the present application;
[0065] Figure 2It is a structural diagram of an automated evaluation system for a reinforcement learning algorithm model proposed in another embodiment of the present application;
[0066] Figure 3 This is a flowchart of an automated evaluation method for a reinforcement learning algorithm model proposed in one embodiment of the present application.
[0067] Figure numerals: 1. Interface module; 2. Data upload module; 3. Simulation environment construction module; 4. Model retest module; 41. Call test submodule; 42. Run test submodule; 5. Baseline algorithm model management module; 6. Performance test module; 61. Index determination submodule; 62. Baseline algorithm model selection submodule; 63. Model test submodule; 64. Score calculation submodule; 7. Model evaluation module. DETAILED DESCRIPTION
[0068] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0069] Intelligent algorithms based on reinforcement learning have achieved extraordinary achievements in the fields of automatic control, human-computer interaction, and game-based confrontation games, and are widely used. Reinforcement learning algorithms are mainly used to deal with large or even continuous state space and action space problems. In such applications, the reinforcement learning agent uses the feedback of the simulation environment as the model algorithm of the agent input. It mainly optimizes the agent through interaction and trial and error between the agent and the environment, and uses evaluative feedback signals such as reward functions. However, since the constructed simulation environment cannot restore the real world 100%, when "unexpected" changes occur in the real world, the originally trained agent cannot adapt well to the new changed scene. Therefore, conducting evaluations of artificial intelligence algorithms is of great significance to ensure that artificial intelligence technology is easy to use, safe, reliable, and controllable.
[0070] At present, the evaluation of reinforcement learning algorithm models is often based on a single or statistical indicator, and the model with better overall performance is better. However, when facing scenarios with extremely high requirements for model performance, such as security drills and red-blue game confrontation games, since both parties are always in a game confrontation with each other, even if a 99% winning rate has been achieved, it is difficult to ensure that 1% of failures will not occur in a specific game confrontation scenario.
[0071] In view of this, the present application proposes an automated evaluation system and method for reinforcement learning algorithm models. By evaluating the applicable scope of the algorithm model and its capabilities under boundary conditions, the measured values of various evaluations of the algorithm model and the capability improvement value of the reinforcement learning model to be evaluated relative to the baseline algorithm model are given to determine whether the performance of the reinforcement learning model to be evaluated in the new scenario can meet user expectations, providing a more comprehensive and convenient solution for the rapid evaluation of intelligent algorithms based on reinforcement learning technology.
[0072] Please refer to Figure 1 , Figure 1 Schematic diagram of the structure of an automated evaluation system for a reinforcement learning algorithm model proposed in one embodiment of the present application. Figure 1 As shown, the system includes an interface module 1, a data upload module 2, a simulation environment construction module 3, a model retest module 4, a baseline algorithm model management module 5, a performance test module 6 and a model evaluation module 7. Specifically, the functions of each module are as follows:
[0073] The interface module 1 is used to communicate with the real application system to receive the real environment data required to build a new application simulation environment.
[0074] In this embodiment, the automated evaluation system is connected to the front-end server system through the interface module 1, thereby interconnecting information with the front-end server system to dynamically receive the real environment data required to build the new application simulation environment to be applied in the future. The front-end server system sends the current environment data to the automated evaluation system (such as the current position of the player's chess piece, the position of the opponent's player and the number of chess pieces, etc.), and the automated evaluation system calls the deployed reinforcement model to predict which position to move to in the current step. After the conversion from the label to the specific position, the prediction result is sent to the front-end server system for specific execution, thereby performing a simulation confrontation simulation of the reinforcement model.
[0075] Exemplarily, the new application simulation environment to be applied in the future can be a simulation environment under the red-blue game confrontation scenario. The red-blue game confrontation scenario includes a variety of application simulation environments, such as chessboard confrontation, security drill confrontation, commercial price confrontation, etc. In the chessboard confrontation, it is necessary to consider the current positions of both parties, the remaining positions that can be walked, the constraints and rules of the action, etc. For example, in a special position, you can only go forward or go right. At this time, the real environment data that the automated evaluation system can receive from the front-end server system through the interface module 1 includes the basic environment data (character image, chessboard settings, background settings, etc.) required to build the chessboard confrontation environment and the status data of the current user (position, remaining positions that can be walked, etc.). In the security drill confrontation, it is necessary to consider the types of equipment, equipment quantity, use time, special restrictions, etc. of both parties. For example, within a special geographical area, the number of equipment used shall not exceed 2. At this time, the real environment data that the automated assessment system can receive from the front-end server system through interface module 1 includes the basic environment data required to build a security drill confrontation environment (character image, equipment settings, background settings, etc.) and the current user's status data (both parties' assumed inputs, equipment capabilities, equipment types and quantities, etc.).
[0076] In commercial price confrontation, it is necessary to consider the strength of both parties, the current adjustment plan, and the impact of the execution plan on the other party. For example, one party is telecom operator A and the other is telecom operator B. The two parties compete in a certain field of telecom, and the initial price is P0. Telecom operator A is an old enterprise with strong strength and has occupied the vast majority of the market share. Telecom operator B has just been established and is relatively weak, but it was established by the government to break the monopoly and encourage market competition. It has received government support, so there are some subsidies and discounts. Its price P0 can be 5% lower than that of telecom operator A, but even so, it will not have much impact on telecom operator A because its foundation is relatively solid. However, as telecom operator B uses its price advantage to occupy more and more market share, to a certain extent, it will affect telecom operator A. At this time, A has to adjust its plan and simulate the impact of the execution plan on the other party in order to maximize its benefits. At this time, the real environment data that the automated evaluation system can receive from the front-end server system through interface module 1 includes the basic environment data required to build a commercial price confrontation environment (corporate image, strength settings, background settings, etc.) and the current user's status data (the maximum loss that each party can bear, adjustment plans, the impact of the execution plan on the other party, etc.).
[0077] The data uploading module 2 is used to upload the reinforcement learning model to be evaluated and the corresponding sample environment data and sample data.
[0078] In this embodiment, the reinforcement learning model to be evaluated and the corresponding sample environment data and sample data to be used in the subsequent evaluation can be uploaded through the data upload module 2, and the uploaded data and model are stored and managed. Among them, the reinforcement learning model to be evaluated is obtained by training the sample data under the sample environment. Sample environment data refers to the basic environment data required to build simulation environments such as chessboard confrontation, security drill confrontation, and commercial price confrontation. Sample data refers to a group of sample data consisting of an input data and corresponding output data of the reinforcement learning model to be evaluated. The input data of the sample data includes task data such as constraints and optimization goals in the simulation environment, such as constraints such as limiting the direction of action and limiting the number of equipment used, and optimization goals such as requiring the red side to win in the shortest time, or making the red side win with the least loss to the red side. The output data of the sample data is the action plan of the red and blue sides, that is, the action strategy consisting of a series of action sequences that need to be executed.
[0079] The sample environment data corresponding to the reinforcement learning model to be evaluated stored in the data upload module 2 can be called by the simulation environment construction module 3 to reconstruct the simulation environment when the reinforcement learning model to be evaluated is trained. The reinforcement learning model to be evaluated and the corresponding sample data can be called by the simulation environment construction module 3, the model retest module 4 and the performance test module 6 to retest the function of the reinforcement learning model to be evaluated, and if it is verified that the function of the reinforcement learning model to be evaluated is normal, then its performance in the new application simulation environment is further tested to evaluate its ability and performance in the new application simulation environment to determine whether it can meet user expectations.
[0080] The simulation environment construction module 3 is used to construct a first simulation environment and a second simulation environment. The first simulation environment is a simulation environment for training the reinforcement learning model to be evaluated reconstructed based on the sample environment data, and the second simulation environment is a new application simulation environment constructed based on the real environment data.
[0081] In this embodiment, the simulation environment construction module 3 is mainly used to construct a verification simulation environment for the reinforcement learning model to be evaluated, that is, by calling the sample environment data in the data upload module 2, the simulation environment during the training of the reinforcement learning model to be evaluated is rebuilt, and it is used as the first simulation environment. At the same time, it is also used to construct a simulation environment for the future application of the reinforcement learning model to be evaluated, that is, by calling the real environment data in the interface module 1 to build a new application simulation environment, and use it as the second simulation environment.
[0082] It should be noted that the simulation environment for training the reinforcement learning model to be evaluated, that is, the first simulation environment, is re-tested to verify whether its function is normal. Since the reinforcement learning model to be evaluated uploaded directly through the data upload module 2 cannot guarantee that it can be used normally in this evaluation system, it is necessary to verify its function before comprehensively evaluating and comparing it. The evaluation can begin after the verification is passed. Constructing the simulation environment in which the reinforcement learning model to be evaluated is intended to be applied in the future, that is, the second simulation environment, is to test the ability of the reinforcement learning model to be evaluated in the new application simulation environment, to see whether the evaluation reinforcement learning model can better adapt to the changing new environment, and to judge whether the performance of the reinforcement learning model to be evaluated in the new environment can meet user expectations.
[0083] The model retest module 4 is used to integrate the reinforcement learning model to be tested into the first simulation environment and verify the function of the reinforcement learning model to be tested using the sample data.
[0084] In this embodiment, the model retest module 4 is used to integrate the reinforcement learning model to be evaluated into the first simulation environment, and use sample data to retest the reinforcement learning model to be evaluated to verify whether its function is normal. Figure 2 The functions of model retest module 4 mainly include calling test and running test:
[0085] The calling test submodule 41 is used to call the reinforcement learning model to be evaluated and the corresponding sample environment data from the data uploading module 2, integrate the reinforcement learning model to be evaluated into the first simulation environment, and detect whether the reinforcement learning model to be evaluated can be called normally.
[0086] In the specific implementation of this embodiment, by calling the reinforcement learning model to be evaluated and the corresponding sample environment data stored in the data upload module 2, the reinforcement learning model to be evaluated is integrated into the first simulation environment using the deployment tool to test whether the reinforcement learning model to be evaluated can be called normally in the evaluation system of this application. If the call fails, the evaluation result of the reinforcement learning model to be evaluated is 0 points, that is, the test score of the reinforcement learning model to be evaluated obtained by the evaluation system of this application is 0 points, and the next step of evaluation is no longer performed. If the call is successful, the function operation test is further performed.
[0087] The operation test submodule 42 is used to call the sample data corresponding to the reinforcement learning model to be evaluated from the data upload module 2 after the reinforcement learning model to be evaluated is successfully called, input the input data in the sample data into the reinforcement learning model to be evaluated for processing, obtain the operation result data, and compare the operation result data with the corresponding output data in the sample data to verify the function of the reinforcement learning model to be evaluated.
[0088] In the specific implementation of this embodiment, if the reinforcement learning model to be evaluated is called successfully, the sample data corresponding to the reinforcement learning model to be evaluated is called from the data upload module 2, and the input data in the sample data is input into the reinforcement learning model to be evaluated for processing to obtain the operation result data. Then, the operation result data is compared with the corresponding output data in the sample data to determine whether the operation result data is consistent with the corresponding output data in the sample data. If the operation result data is inconsistent with the corresponding output data in the sample data, it means that there is a problem with the operation function of the reinforcement learning model to be evaluated in the evaluation system of this application, and the evaluation result of the reinforcement learning model to be evaluated is 0 points, that is, the test score of the reinforcement learning model to be evaluated obtained by the evaluation system of this application is 0 points, and no further evaluation is performed. If the operation result data is consistent with the corresponding output data in the sample data, it means that the function of the reinforcement learning model to be evaluated in the evaluation system of this application is running normally, and then a comprehensive performance evaluation is further performed.
[0089] Exemplarily, if the simulation environment for training the reinforcement learning model to be evaluated, that is, the first simulation environment, is a snake-shaped chessboard confrontation environment. There are many small grids on the chessboard, each grid represents a position, and there are several chess pieces and dice. The ultimate goal of the game is to make the red side reach "40" first. Each player has a chess piece, and the starting point is at the grid "1". In the process, the players roll the dice in turn and move forward the corresponding number of steps according to the number of dice. Assuming that the player's chess piece is at "1" and throws "4", the chess piece can reach the position of "5". Among them, there are two ways to roll the dice, one can evenly throw out the 6 numbers 1-6, and the other can evenly throw out the 3 numbers 1-3. Finally, if the number thrown when arriving plus the current position exceeds 40, then the chess piece will first reach 40, and the remaining steps will move in the opposite direction.
[0090] After the above information is used as the input data of the sample data into the reinforcement learning model to be evaluated, the operation result data obtained through deduction is: the state-action chain of the red side {s0, a0, s1, a1, ... s t-1 ,a t-1 ,s t}, where s t represents the game state at time t, that is, the position of the red chess piece, a t Indicates the method chosen by the red team at time t. If, during the training process of the reinforcement learning model to be evaluated, the output data corresponding to the input data of the sample data itself is: {s0, a0, s1, a1', ... s t-1 ,a t-1 ',s t}, that is, when the red chess piece is in position s1, the a1' throwing technique is selected, and when the red chess piece is in position s t-1 When you select a t-1 'Throwing technique. Since the result deduced by the reinforcement learning model to be evaluated in the evaluation system of this application is that when the red chess piece is in the position of s1, the a1 throwing technique is selected, and when the red chess piece is in the position of s t-1 When you select a t-1 The throwing technique is inconsistent with the output data corresponding to the input data of the sample data itself, so the operating function of the reinforcement learning model to be evaluated in the evaluation system of this application has problems, the evaluation result is 0 points, and no further evaluation is carried out.
[0091] The baseline algorithm model management module 5 is used to store and manage a number of baseline algorithm models, and any of the above baseline algorithm models is marked with a corresponding applicable scope and boundary conditions.
[0092] In this embodiment, the baseline algorithm model management module 5 integrates multiple baseline algorithm models, including rule-based, traversal optimization, branch and bound, Monte Carlo simulation, genetic algorithm, particle swarm algorithm, PPO algorithm, SAC algorithm and other algorithm models, and each algorithm model will mark its own scope of application and boundary conditions.
[0093] Exemplarily, the algorithm model based on traversal optimization will simulate each situation and find the optimal solution, so the output solution obtained is better, but it also takes longer time and is not suitable for environments with shorter time requirements. The PPO algorithm is applicable to various types of tasks, not just limited to the processing of a single task. It has higher stability and is more tolerant to environmental changes through multi-step optimization. Compared with other algorithms, the convergence of the PPO algorithm is also more stable. The SAC algorithm is a random strategy method with high sampling efficiency, strong exploration ability and high stability. In actual use, the user can select a baseline algorithm model with applicable scope and boundary conditions that meet the requirements from the baseline algorithm model management module 5 according to the requirements and goals of the current new application simulation environment. For example, the current new application simulation environment is a security drill confrontation environment, and its goal is to make the red side win in the shortest time, indicating that it is more biased towards deduction speed. When selecting, considering that the speed of the particle swarm algorithm model is relatively faster, the particle swarm algorithm model is selected to compare and test with the reinforcement learning model to be evaluated to determine the effect of the reinforcement learning model to be evaluated.
[0094] The performance testing module 6 is used to test the performance of the reinforcement learning model to be evaluated and the selected baseline algorithm model in the new application simulation environment to obtain corresponding test scores.
[0095] In this embodiment, first, according to the requirements of the new application simulation environment, a suitable baseline algorithm model is selected. Then, according to the user's focus, the corresponding performance indicators used in the evaluation are selected, including availability, pros and cons, agility and stability. For example, if the user focuses more on the stability of the algorithm model, stability is selected as the performance indicator of the evaluation, and the performance of the reinforcement learning model to be evaluated and the selected baseline algorithm model in the new application simulation environment are compared. If the user focuses more on the comprehensive performance of the algorithm model, a comprehensive calculation can be further performed based on the performance of the indicators from various angles, and the comprehensive performance of the reinforcement learning model to be evaluated and the selected baseline algorithm model in the new application simulation environment is compared.
[0096] After selecting the baseline algorithm model for comparison and determining the performance indicators for evaluation, the reinforcement learning model to be evaluated is deployed to the new application simulation environment, and the dynamic environment data of the front-end service system received in real time through the interface module 1 is loaded, and the reinforcement learning model to be evaluated is used for simulation deduction to obtain the corresponding output data. Then, according to the preset goals, the task completion of the reinforcement learning model to be evaluated is judged. Furthermore, multiple groups of dynamic environment data can be used for testing, and data such as the success rate of task completion, average running time, average reward value, etc. can be counted to calculate the performance scores of the reinforcement learning model to be evaluated from multiple angles such as availability, pros and cons, agility and stability, as well as the comprehensive test scores. The same is true for the baseline algorithm model.
[0097] The model evaluation module 7 is used to calculate the capability value of the reinforcement learning model to be evaluated relative to the baseline algorithm model based on the test score, so as to realize the automatic evaluation of the reinforcement learning model to be evaluated.
[0098] In this embodiment, after measuring the test scores of the reinforcement learning model to be evaluated and the selected baseline algorithm model in the new application simulation environment using dynamic environment data, the reinforcement learning model to be evaluated and the selected baseline algorithm model can be compared. Specifically, it includes:
[0099] The capability value calculation submodule is used to calculate the capability value of the reinforcement learning model to be evaluated relative to the baseline algorithm model based on the test score in the following manner:
[0100]
[0101] Among them, A represents the capability value of the reinforcement learning model to be evaluated, and H 测 represents the test score of the reinforcement learning model to be evaluated, H 基 Represents the test score of the baseline algorithm model.
[0102] Through the above method, we can understand the performance superiority of the reinforcement learning model to be evaluated in the new application simulation environment relative to the baseline algorithm model, and the larger A is, the better the performance of the reinforcement learning model to be evaluated in the new application simulation environment. At this point, the automated evaluation of the reinforcement learning model to be evaluated is achieved. Compared with the evaluation of existing reinforcement learning algorithm models, which is based on a single or statistical indicator, the model with better comprehensive performance is better, but there is no comprehensive evaluation of the scope of use of the algorithm model and its capabilities under boundary conditions. This application introduces model evaluation technology and weighting mechanisms for dynamically changing scenarios, combines the comprehensive indicators of the model with the application scenarios, and conducts a more systematic and comprehensive evaluation of the algorithm model.
[0103] In addition, it also includes: a model updating submodule, which is used to save the above-mentioned reinforcement learning model to be evaluated whose capability value is greater than or equal to the preset threshold into the above-mentioned baseline algorithm model management module 5, and update the above-mentioned baseline algorithm model.
[0104] For example, the user can set a capability value standard according to the actual situation, such as 20%. When A ≥ 20%, it means that the performance of the reinforcement learning model to be evaluated is good in the new application simulation environment, so the reinforcement learning model to be evaluated is saved in the baseline algorithm model management module 5, and the baseline algorithm model is updated. When it is necessary to test the performance of another reinforcement learning model to be evaluated in a similar environment in the future, the newly added reinforcement learning model to be evaluated can be selected from the baseline algorithm model management module 5 as the baseline algorithm model for comparison.
[0105] Please refer to Figure 2 , Figure 2 : is a schematic diagram of the structure of an automated evaluation system for a reinforcement learning algorithm model proposed in another embodiment of the present application. Optionally, the performance test module 6 includes:
[0106] The indicator determination submodule 61 is used to determine the performance indicators used for evaluation, where the above performance indicators are one or more of availability, superiority, agility and stability.
[0107] Specifically, when determining the performance indicators used for the evaluation, it can be determined by selecting a working mode, including: the first working mode, which is used to perform corresponding evaluations according to one or more performance indicators selected by the user, obtain test scores, and compare them with the test scores of the above-mentioned baseline algorithm model. When the user has limited time to evaluate the algorithm model or has his own evaluation direction, he can select one or more evaluation angles from the perspectives of the model's availability, pros and cons, agility, and stability, and compare the results after evaluation with the results of the baseline algorithm. The second working mode is used to automatically complete the comprehensive evaluation of the above-mentioned four performance indicators of availability, pros and cons, agility, and stability, obtain test scores, and compare them with the test scores of the above-mentioned baseline algorithm model. When the user has sufficient time to evaluate the algorithm model or wants to evaluate the algorithm model more comprehensively, he can automatically complete the comprehensive evaluation of the model's availability, pros and cons, agility, and stability from multiple angles with one click, and compare the results after evaluation with the results of the baseline algorithm.
[0108] The baseline algorithm model selection submodule 62 is used to select and call the baseline algorithm model whose applicable scope and boundary conditions meet the requirements from the baseline algorithm model management module 5 according to the requirements of the new application simulation environment. For details, refer to the baseline algorithm model management module 5.
[0109] The model testing submodule 63 is used to test the performance of the reinforcement learning model to be evaluated and the called baseline algorithm model in the new application simulation environment to obtain the performance scores of the corresponding performance indicators.
[0110] Specifically, the performance score mainly includes four aspects:
[0111] The availability score calculation unit is used to call the real environment data in the interface module 1, input it into the reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding output result, and count the success rate of the reinforcement learning model / baseline algorithm model to be evaluated according to the preset task objectives and the output results, and calculate the corresponding availability score according to the success rate:
[0112]
[0113] Among them, P1 represents the availability score, p represents the success rate, and a represents the preset standard value.
[0114] In the specific implementation of this embodiment, multiple sets of dynamic environment data can be used for testing, and the success rate of task completion can be calculated according to the preset task objectives and output results, and the success rate can be compared with the preset standard value to determine the availability of the reinforcement learning model to be evaluated / baseline algorithm model. For example, if the success rate of the reinforcement learning model to be evaluated is 80% and the preset standard value is 70%, the availability score of the reinforcement learning model to be evaluated is 1, which reflects the model applicability of the reinforcement learning model to be evaluated in this new application simulation environment.
[0115] The merit score calculation unit is used to call the real environment data in the interface module 1, input it into the reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding output reward value, and obtain the merit score by calculating the average reward value:
[0116]
[0117] Among them, P2 represents the superiority score, N represents the number of real environment data, and M n Represents the reward value corresponding to the nth set of real environment data.
[0118] In the specific implementation of this embodiment, the superiority represents the model reward score of the reinforcement learning model / baseline algorithm model to be evaluated in this new application simulation environment. By using multiple sets of dynamic environment data for testing, the reward value output by the model each time is statistically calculated, and the average reward value of multiple sets of data is calculated to obtain the superiority score of the model. Among them, the reward value is the cumulative reward value deduced by the model after the dynamic environment data is input. For example, at a certain position, going forward will get a reward value of 5 points, and going right will get a reward value of 7 points; then going right at the next position will get 3 points, and going left will get 3 points. Finally, the cumulative reward value is obtained according to the action sequence output by the model.
[0119] The agility score calculation unit is used to call the real environment data in the interface module 1, input it into the reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding average running time, and calculate the agility score based on the average running time and the preset task requirement time:
[0120]
[0121] Among them, P3 represents the agility score, t1 represents the average running time, and t2 represents the preset task requirement time.
[0122] In the specific implementation of this embodiment, agility represents the model running time of the reinforcement learning model / baseline algorithm model to be evaluated in this new application simulation environment. By using multiple sets of dynamic environment data for testing, the model running time during each test is counted, and the average running time of multiple sets of data is calculated, and it is compared with the expected time preset by the user, that is, the task requirement time, to obtain an agility score.
[0123] The stability score calculation unit is used to call the real environment data in the interface module 1, input it into the reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding output result, and count the number of successes of the reinforcement learning model / baseline algorithm model to be evaluated according to the preset task objectives and the output results, and calculate the corresponding stability score according to the number of successes:
[0124]
[0125] Among them, P4 represents the stability score, N represents the number of real environment data, and n represents the number of successes.
[0126] In the specific implementation of this embodiment, stability refers to the model operation completion rate of the reinforcement learning model / baseline algorithm model to be evaluated in this new application simulation environment. The higher the completion rate, the better the stability of the model.
[0127] The score calculation submodule 64 is used to perform comprehensive calculations based on the above performance scores to obtain the test scores of the above reinforcement learning model to be evaluated and the called baseline algorithm model.
[0128] Specifically, it mainly includes: using the following method to calculate the test scores of the reinforcement learning model to be evaluated and the called baseline algorithm model:
[0129]
[0130] Among them, H represents the comprehensive test score, P i represents the performance score of the i-th performance indicator, W i It represents the weight coefficient of the i-th performance indicator preset by the user, and m represents the number of performance indicators.
[0131] It should be noted that during the evaluation process, users can select the performance indicators to be evaluated and adaptively adjust the weight of each performance indicator, that is, the weight coefficient W. i , thus obtaining a model algorithm that is closer to user needs.
[0132] Based on the same inventive concept, an embodiment of the present application provides an automated evaluation method for a reinforcement learning algorithm model. Figure 3 , Figure 3: is a flowchart of an automated evaluation method for a reinforcement learning algorithm model proposed in an embodiment of the present application, the method comprising:
[0133] Step S101: Acquire real environment data required for building a new application simulation environment, a reinforcement learning model to be evaluated, and corresponding sample environment data and sample data, wherein the reinforcement learning model to be evaluated is obtained by training the sample data in the sample environment;
[0134] Step S102: constructing a first simulation environment and a second simulation environment, wherein the first simulation environment is a simulation environment for training the reinforcement learning model to be evaluated reconstructed according to the sample environment data, and the second simulation environment is a new application simulation environment constructed according to the real environment data;
[0135] Step S103: integrating the reinforcement learning model to be tested into the first simulation environment, and retesting the function of the reinforcement learning model to be tested using the sample data;
[0136] Step S104: According to the requirements of the new application simulation environment, a baseline algorithm model whose applicable scope and boundary conditions meet the requirements is selected from a preset baseline algorithm model management library;
[0137] Step S105: testing the performance of the reinforcement learning model to be evaluated and the selected baseline algorithm model in the new application simulation environment according to the real environment data to obtain corresponding test scores;
[0138] Step S106: Based on the test scores, the capability values of the reinforcement learning model to be evaluated relative to each baseline algorithm model are calculated respectively, so as to realize automatic evaluation of the reinforcement learning model to be evaluated.
[0139] Optionally, the step of integrating the reinforcement learning model to be evaluated into the first simulation environment and retesting the function of the reinforcement learning model to be evaluated using the sample data includes:
[0140] Calling the reinforcement learning model to be evaluated and the corresponding sample environment data, integrating the reinforcement learning model to be evaluated into the first simulation environment to detect whether the reinforcement learning model to be evaluated can be called normally;
[0141] After the above-mentioned reinforcement learning model to be evaluated is called successfully, the sample data corresponding to the above-mentioned reinforcement learning model to be evaluated is called, the input data in the above-mentioned sample data is input into the above-mentioned reinforcement learning model to be evaluated for processing, and the operation result data is obtained, and the above-mentioned operation result data is compared with the corresponding output data in the sample data to verify the function of the above-mentioned reinforcement learning model to be evaluated.
[0142] Optionally, the performance of the reinforcement learning model to be evaluated and the selected baseline algorithm model in the new application simulation environment are tested according to the real environment data to obtain corresponding test scores, including:
[0143] Determine the performance indicators used for the evaluation, where the performance indicators are one or more of availability, quality, agility and stability;
[0144] Respectively test the performance of the reinforcement learning model to be evaluated and the baseline algorithm model in the new application simulation environment to obtain the performance scores of the corresponding performance indicators;
[0145] Based on the above performance scores, a comprehensive calculation is performed to obtain the test scores of the above reinforcement learning model to be evaluated and the above baseline algorithm model.
[0146] Optionally, the method further includes selecting a working mode of the evaluation:
[0147] When the first working mode is selected, a corresponding evaluation is performed according to one or more performance indicators selected by the user to obtain a test score, and the score is compared with the test score of the baseline algorithm model;
[0148] When the second working mode is selected, a comprehensive evaluation of the above four performance indicators of availability, pros and cons, agility and stability is automatically completed to obtain a test score, which is then compared with the test score of the above baseline algorithm model.
[0149] Optionally, the performance of the reinforcement learning model to be evaluated and the baseline algorithm model in the new application simulation environment are tested respectively to obtain the performance scores of the corresponding performance indicators, including:
[0150] Call the real environment data, input it into the above reinforcement learning model / baseline algorithm model to be evaluated, and obtain the corresponding output results. According to the preset task objectives and output results, the success rate of the above reinforcement learning model / baseline algorithm model to be evaluated is counted, and the corresponding availability score is calculated according to the success rate:
[0151]
[0152] Among them, P1 represents the availability score, p represents the success rate, and a represents the preset standard value;
[0153] Call the real environment data, input it into the above reinforcement learning model / baseline algorithm model to be evaluated, and obtain the corresponding output reward value. Then, calculate the average reward value to obtain the merit score:
[0154]
[0155] Among them, P2 represents the superiority score, N represents the number of real environment data, and M n Represents the reward value corresponding to the nth set of real environment data;
[0156] Call the real environment data, input the above reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding average running time, and calculate the agility score based on the average running time and the preset task requirement time:
[0157]
[0158] Among them, P3 represents the agility score, t1 represents the average running time, and t2 represents the preset task requirement time;
[0159] Call the real environment data, input it into the above reinforcement learning model / baseline algorithm model to be evaluated, and obtain the corresponding output results. According to the preset task objectives and output results, the number of successes of the above reinforcement learning model / baseline algorithm model to be evaluated is counted, and the corresponding stability score is calculated according to the number of successes:
[0160]
[0161] Among them, P4 represents the stability score, N represents the number of real environment data, and n represents the number of successes.
[0162] Optionally, the above-mentioned comprehensive calculation based on the performance scores to obtain the test scores of the above-mentioned reinforcement learning model to be evaluated and the above-mentioned baseline algorithm model respectively includes:
[0163] The test scores of the reinforcement learning model to be evaluated and the called baseline algorithm model are calculated as follows:
[0164]
[0165] Among them, H represents the comprehensive test score, P i represents the performance score of the i-th performance indicator, W i It represents the weight coefficient of the i-th performance indicator preset by the user, and m represents the number of performance indicators.
[0166] Optionally, based on the test scores, the capability values of the reinforcement learning model to be evaluated relative to each baseline algorithm model are calculated respectively to realize automatic evaluation of the reinforcement learning model to be evaluated, including:
[0167] Based on the above test scores, the capability value of the above reinforcement learning model to be evaluated relative to the above baseline algorithm model is calculated in the following way:
[0168]
[0169] Among them, A represents the capability value of the reinforcement learning model to be evaluated, and H 测 represents the test score of the reinforcement learning model to be evaluated, H 基 Represents the test score of the baseline algorithm model;
[0170] The reinforcement learning model to be evaluated whose capability value is greater than or equal to the preset threshold is saved in the baseline algorithm model management library, and the baseline algorithm model is updated.
[0171] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0172] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0173] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0174] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0175] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.
[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0177] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0178] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0179] The above is a detailed introduction to an automated evaluation system and method for a reinforcement learning algorithm model provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. An automated evaluation system for reinforcement learning algorithm models. It is characterized in that The system comprises: An interface module, used to communicate with the real application system to receive real environment data required to build a new application simulation environment; The data upload module is used to upload the reinforcement learning model to be evaluated and the corresponding sample environment data and sample data; A simulation environment construction module, used to construct a first simulation environment and a second simulation environment, wherein the first simulation environment is a simulation environment for training the reinforcement learning model to be evaluated reconstructed according to the sample environment data, and the second simulation environment is a new application simulation environment constructed according to the real environment data; A model retest module, used to integrate the reinforcement learning model to be tested into the first simulation environment, and verify the function of the reinforcement learning model to be tested using the sample data; A baseline algorithm model management module, used to store and manage a number of baseline algorithm models, any of which is marked with a corresponding scope of application and boundary conditions; A performance testing module, used to test the performance of the reinforcement learning model to be evaluated and the selected baseline algorithm model in the new application simulation environment, respectively, to obtain corresponding test scores; A model evaluation module, used to calculate the capability value of the reinforcement learning model to be evaluated relative to the baseline algorithm model based on the test score, so as to realize automatic evaluation of the reinforcement learning model to be evaluated; Wherein, the performance test module includes: An indicator determination submodule is used to determine the performance indicators used for evaluation, wherein the performance indicators are one or more of availability, superiority, agility and stability; A baseline algorithm model selection submodule is used to select and call a baseline algorithm model whose applicable scope and boundary conditions meet the requirements from the baseline algorithm model management module according to the requirements of the new application simulation environment; A model testing submodule is used to test the performance of the reinforcement learning model to be evaluated and the called baseline algorithm model in the new application simulation environment, and obtain the performance scores of the corresponding performance indicators; The score calculation submodule is used to perform comprehensive calculations based on the performance scores to obtain the test scores of the reinforcement learning model to be evaluated and the called baseline algorithm model.
2. The system according to claim 1, It is characterized in that The model retest module includes: A calling test submodule is used to call the reinforcement learning model to be evaluated and the corresponding sample environment data from the data uploading module, integrate the reinforcement learning model to be evaluated into the first simulation environment, and detect whether the reinforcement learning model to be evaluated can be called normally; The running test submodule is used to call the sample data corresponding to the reinforcement learning model to be evaluated from the data upload module after the reinforcement learning model to be evaluated is successfully called, input the input data in the sample data into the reinforcement learning model to be evaluated for processing, obtain the running result data, and compare the running result data with the corresponding output data in the sample data to verify the function of the reinforcement learning model to be evaluated.
3. The system according to claim 1, It is characterized in that The performance test module is also used to select the working mode of the evaluation, including: The first working mode is used to perform corresponding evaluation according to one or more performance indicators selected by the user, obtain a test score, and compare it with the test score of the baseline algorithm model; The second working mode is used to automatically complete the comprehensive evaluation of the four performance indicators of availability, quality, agility and stability, obtain the test score, and compare it with the test score of the baseline algorithm model.
4. The system according to claim 1, It is characterized in that The model testing submodule includes: The availability score calculation unit is used to call the real environment data in the interface module, input it into the reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding output result, and count the success rate of the reinforcement learning model / baseline algorithm model to be evaluated according to the preset task objectives and the output results, and calculate the corresponding availability score according to the success rate: Among them, P 1 represents the availability score, p represents the success rate, and a represents the preset standard value; The merit score calculation unit is used to call the real environment data in the interface module, input it into the reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding output reward value, and obtain the merit score by calculating the average reward value: Among them, P 2 represents the quality score, N represents the number of real environment data, M n Represents the reward value corresponding to the nth set of real environment data; The agility score calculation unit is used to call the real environment data in the interface module, input it into the reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding average running time, and calculate the agility score according to the average running time and the preset task requirement time: Among them, P 3 represents the agility score, t 1 represents the average running time, t 2 Indicates the preset task requirement time; The stability score calculation unit is used to call the real environment data in the interface module, input it into the reinforcement learning model / baseline algorithm model to be evaluated for deduction, obtain the corresponding output result, and count the number of successes of the reinforcement learning model / baseline algorithm model to be evaluated according to the preset task objectives and the output results, and calculate the corresponding stability score according to the number of successes: Among them, P 4 represents the stability score, N represents the number of real environment data, and n represents the number of successes.
5. The system according to claim 1, It is characterized in that The score calculation submodule includes: The test scores of the reinforcement learning model to be evaluated and the called baseline algorithm model are calculated in the following way: Among them, H represents the comprehensive test score, P i represents the performance score of the ith performance indicator, W i It represents the weight coefficient of the i-th performance indicator preset by the user, and m represents the number of performance indicators.
6. The system according to claim 1, It is characterized in that The model evaluation module includes: The capability value calculation submodule is used to calculate the capability value of the reinforcement learning model to be evaluated relative to the baseline algorithm model based on the test score in the following manner: Among them, A represents the capability value of the reinforcement learning model to be evaluated, and H 测 represents the test score of the reinforcement learning model to be evaluated, H 基 Represents the test score of the baseline algorithm model; The model updating submodule is used to save the reinforcement learning model to be evaluated whose capability value is greater than or equal to a preset threshold into the baseline algorithm model management module, and update the baseline algorithm model.
7. An automated evaluation method for reinforcement learning algorithm models. It is characterized in that The method comprises: Acquire real environment data required for building a new application simulation environment, a reinforcement learning model to be evaluated, and corresponding sample environment data and sample data, wherein the reinforcement learning model to be evaluated is obtained by training the sample data in the sample environment; Constructing a first simulation environment and a second simulation environment, wherein the first simulation environment is a simulation environment for training the reinforcement learning model to be evaluated reconstructed according to the sample environment data, and the second simulation environment is a new application simulation environment constructed according to the real environment data; Integrating the reinforcement learning model to be evaluated into the first simulation environment, and retesting the function of the reinforcement learning model to be evaluated using the sample data; According to the requirements of the new application simulation environment, a baseline algorithm model whose applicable scope and boundary conditions meet the requirements is selected from a preset baseline algorithm model management library; According to the real environment data, the performance of the reinforcement learning model to be evaluated and the selected baseline algorithm model are respectively tested in the new application simulation environment to obtain corresponding test scores; Based on the test scores, respectively calculating the capability values of the reinforcement learning model to be evaluated relative to the baseline algorithm model, so as to realize automatic evaluation of the reinforcement learning model to be evaluated; Among them, the performance of the reinforcement learning model to be evaluated and the selected baseline algorithm model in the new application simulation environment are tested respectively according to the real environment data to obtain corresponding test scores, including: Determine the performance indicators used for evaluation, where the performance indicators are one or more of availability, quality, agility, and stability; Respectively testing the performance of the reinforcement learning model to be evaluated and the baseline algorithm model in the new application simulation environment to obtain performance scores of corresponding performance indicators; A comprehensive calculation is performed based on the performance scores to obtain test scores of the reinforcement learning model to be evaluated and the baseline algorithm model.
8. The method according to claim 7, It is characterized in that It also includes the selection of the working mode of the evaluation: When the first working mode is selected, a corresponding evaluation is performed according to one or more performance indicators selected by the user to obtain a test score, and the score is compared with the test score of the baseline algorithm model; When the second working mode is selected, a comprehensive evaluation of the four performance indicators of availability, quality, agility and stability is automatically completed to obtain a test score, which is compared with the test score of the baseline algorithm model.
