Adaptive evaluation function dynamic weight adjustment method based on reinforcement learning

Abstract

The invention provides an adaptive evaluation function dynamic weight adjustment method based on reinforcement learning. The adaptive evaluation function dynamic weight adjustment method comprises the steps of obtaining an initial weight, optimizer parameters and online reinforcement parameters of a preset online reinforcement learning algorithm; unit type weight initialization is carried out based on the initial weight, an initial score is recorded, an online reinforcement learning algorithm physical state is retrieved based on the initial score, scores of different unit types are initialized, and a resource score is calculated; according to the difference between the current resource score and the last score, updating a learning rate and an attenuation rate in online reinforcement learning through a preset optimizer; and updating the unit type weight through online reinforcement learning based on the updated learning rate and attenuation rate, and recording the latest score and weight until the evaluation is finished. According to the method, the problems of low situation assessment efficiency and poor accuracy of existing online reinforcement learning are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of game development, and in particular to a method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning. Background Art

[0002] In recent years, research on situation assessment functions in real-time strategy (RTS) tasks has played an important role in optimizing game AI performance and balancing game design. Many studies focus on how to build and optimize battlefield assessment methods for real-time strategy games (RTS) to drive AI to conduct autonomous battlefield situation assessment. There are mainly three types of methods: linear methods, tree search methods, and neural network methods.

[0003] Linear function method. A set of values ​​is added together, and each value is multiplied by a weight. These values ​​usually represent simple features, such as the number of units of each type owned by the player, and different weights reflect their estimated value. Although the use of linear methods in situation assessment is simple and intuitive, there are also some obvious shortcomings and limitations, mainly including the following points. First, the complexity is insufficient, and the complex interactions and nonlinear relationships between features cannot be captured. At the same time, this method has insufficient consideration of multi-dimensional factors such as military strength, economic development, spatial control, and the player's game level. Finally, the neglect of dynamic changes, linear methods usually cannot cope well with dynamic changes in the game. In order to solve the above shortcomings, researchers have proposed some more complex and comprehensive evaluation methods. For example, the global evaluation function adjusts the weights through logistic regression, and considers military, economic, spatial and other factors at the same time, and more innovatively considers the player's game level factor outside the game. In addition, there is an advanced evaluation method based on the Lanchester loss law, which takes into account the type of combat units and their health status, making the evaluation more accurate and dynamic. These improved methods make up for the shortcomings and limitations of linear methods to a certain extent. However, the weights of these methods are all pre-set, and they all have the problem of not being able to dynamically adjust the weights according to the battlefield situation.

[0004] Tree search method. Evaluate the win rate by simulating the rest of the game. Unlike calling a static function, tree search utilizes a fast script to simulate the rest of the game and assigns a positive score to the winning player. The theoretical basis for this approach is that even if the script quality is not high, since both players use equal strategies, the player who wins more simulations is likely to be the one with a greater win rate. If running the simulation until the end of the game is not feasible, a hybrid approach can be used, where a finite game execution is performed for a predetermined number of frames and then the evaluation function is called. The evaluation function is usually more accurate near the end of the game, so moving the application of the evaluation function to the end of the game execution usually results in a more accurate evaluation of the value of the game state. However, this approach faces some challenges. First, it requires a lot of computing resources, especially in complex games, which makes the evaluation process time-consuming. Second, the quality of the simulation depends on the quality of the script used, and low-quality scripts may lead to inaccurate evaluation results. In addition, the algorithmic complexity of tree search is high, and it is relatively difficult to implement and debug. At the same time, it is difficult to implement dynamic evaluation function adjustment, which may cause lagging evaluation results in RTS tasks with high real-time requirements. Taking all factors into consideration, although tree search performs well in some cases, its application is limited to a certain extent when dealing with RTS tasks with high real-time requirements and fast scene changes.

[0005] Neural network-based situation assessment methods. Researchers use CNN to extract spatial features, for example, using CNN to achieve situation assessment with improved accuracy but reduced computational speed. Or FCN applicable to maps of any size. Or map-independent feature learning is achieved by separating global and spatial information. In order to increase speed, a multi-size CNN architecture is proposed, although it still has the problem of low computational efficiency. Neural network methods have advantages in RTS task situation assessment, but they also face challenges such as high computing resource requirements, strong data dependence, and insufficient generalization ability.

[0006] In actual work, the evaluation function is usually based on a static linear weighted method, which reacts slowly to the dynamic changes of the battlefield situation. First, manually setting weights requires a lot of expert knowledge, is highly subjective, and it is difficult to ensure the optimality of the weights. Second, static weights cannot cope with the real-time changing battlefield environment, resulting in delayed evaluation results. The above two reasons lay the foundation for the necessity of studying more intelligent adaptive evaluation functions. Although some methods based on deep learning and tree search have achieved certain results, due to their high computational complexity and low real-time performance, they cannot meet the real-time requirements of actual RTS tasks. It is urgent to carry out more efficient and intelligent evaluation function research to break through the limitations of existing methods. Summary of the invention

[0007] The present invention provides a method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning, which is used to solve the problems of low efficiency and poor accuracy of situation evaluation in existing online reinforcement learning.

[0008] The present invention provides a method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning, comprising: Get the initial weights, optimizer parameters, and online reinforcement parameters of the preset online reinforcement learning algorithm; Initializing the unit type weights based on the initial weights and recording the initial scores, retrieving the physical state of the online reinforcement learning algorithm based on the initial scores, initializing the scores of different unit types, and calculating the resource scores; According to the difference between the current resource score and the last score, the learning rate and decay rate in online reinforcement learning are updated through the preset optimizer; Update the unit type weights through online reinforcement learning based on the updated learning rate and decay rate, and record the latest scores and weights until the end of the evaluation.

[0009] According to a method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning provided by the present invention, the unit type weight is initialized based on the initial weight and the initial score is recorded, the physical state of the online reinforcement learning algorithm is retrieved based on the initial score, the scores of different unit types are initialized, and the resource score is calculated, which specifically includes: The initial score is obtained after weight initialization; Based on the initial score, resource status information of the online reinforcement learning algorithm is retrieved, scores of resources of different unit types are calculated, and resource scores are obtained.

[0010] According to a method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning provided by the present invention, the method of updating the learning rate and the decay rate in online reinforcement learning by a preset optimizer specifically includes: The parameter gradient is calculated at each time step through the preset optimizer, and the first-order and second-order moment estimates in the parameter gradient are updated respectively; Bias correction is performed based on the updated results of the first-order and second-order moment estimates, and parameters are updated through learning rate adjustment and weight decay.

[0011] According to a method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning provided by the present invention, the parameter gradient is calculated at each time step by a preset optimizer, and the first-order and second-order moment estimates are updated respectively, specifically including: The parameter gradient is calculated at each time step through the preset optimizer, and the first-order momentum estimate is updated; Updating the second-order momentum estimation based on the first-order momentum estimation result to generate a second-order momentum estimation result; The deviation correction estimate is calculated based on the second-order momentum estimation result to complete the first-order and second-order moment estimation.

[0012] According to a method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning provided by the present invention, the unit type weight is updated through online reinforcement learning based on the updated learning rate and decay rate, and the latest score and weight are recorded until the evaluation is completed, which specifically includes: Calculate the score weight of the evaluation function in the updated unit type through online reinforcement learning based on the updated learning rate and decay rate; The score weight of the evaluation function in the updated unit type is used as the input for the next update until the evaluation is completed, and the latest score and weight are recorded.

[0013] The present invention also discloses a dynamic weight adjustment system for adaptive evaluation functions based on reinforcement learning, the system comprising: A data acquisition module is used to obtain the initial weights, optimizer parameters and online reinforcement parameters of a preset online reinforcement learning algorithm; A score calculation module, used to initialize the unit type weights based on the initial weights and record the initial scores, retrieve the physical state of the online reinforcement learning algorithm based on the initial scores, initialize the scores of different unit types, and calculate the resource scores; The update module is used to update the learning rate and decay rate in online reinforcement learning through a preset optimizer according to the difference between the current resource score and the previous score; The online reinforcement learning module is used to update the unit type weights through online reinforcement learning based on the updated learning rate and decay rate, and record the latest scores and weights until the evaluation ends.

[0014] According to a reinforcement learning-based adaptive evaluation function dynamic weight adjustment system provided by the present invention, the update module calculates the parameter gradient at each time step through a preset optimizer, and updates the first-order and second-order moment estimates in the parameter gradient respectively; Bias correction is performed based on the updated results of the first-order and second-order moment estimates, and parameters are updated through learning rate adjustment and weight decay.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning as described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for adjusting the dynamic weight of an adaptive evaluation function based on reinforcement learning as described above is implemented.

[0018] The present invention provides a method for dynamic weight adjustment of adaptive evaluation function based on reinforcement learning. The method realizes dynamic weight adjustment through online reinforcement learning, so that the evaluation function can adapt to the changes in battlefield situation in real time. At the same time, the method studies the adaptive parameter update method based on optimizer, completes the automatic adjustment of hyperparameters, and improves the robustness and versatility of the algorithm. It provides new ideas and methods for military situation assessment in complex dynamic environments in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of the method for dynamic weight adjustment of adaptive evaluation function based on reinforcement learning provided by the present invention.

[0021] Figure 2 It is an architecture diagram of the method for dynamic weight adjustment of adaptive evaluation function based on reinforcement learning provided by the present invention.

[0022] Figure 3 This is a round-robin experimental map provided by the present invention.

[0023] Figure 4 It is a schematic diagram of the round-robin results in the M1 map provided by the present invention.

[0024] Figure 5 It is a schematic diagram of the round-robin results in the M2 map provided by the present invention.

[0025] Figure 6 It is a schematic diagram of the round-robin results in the M3 map provided by the present invention.

[0026] Figure 7 It is a schematic diagram of module connection of the adaptive evaluation function dynamic weight adjustment system based on reinforcement learning provided by the present invention.

[0027] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention.

[0028] Reference numerals: 110: data acquisition module; 120: score calculation module; 130: update module; 140: online reinforcement learning module; 810: processor; 820: communication interface; 830: memory; 840: communication bus. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Combine the following Figure 1 and Figure 2 The invention describes a method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning, comprising: step 100, obtaining initial weights, optimizer parameters and online reinforcement parameters of a preset online reinforcement learning algorithm.

[0031] Specifically, when performing dynamic weight evaluation of the online reinforcement learning algorithm, it is necessary to input the initial weight, AdamW optimizer parameters, and online reinforcement parameters. Through the dynamic weight adjustment algorithm based on reinforcement learning and AdamW optimizer, the evaluation weight can be adjusted in real time to accurately reflect the battlefield situation, provide guarantee for AI decision-making and real-time response in RTS tasks, and provide support for task AI optimization and strategy research.

[0032] Step 200: Initialize the unit type weights based on the initial weights and record the initial scores, retrieve the physical state of the online reinforcement learning algorithm based on the initial scores, initialize the scores of different unit types, and calculate the resource scores.

[0033] Specifically including: obtaining the initial score after weight initialization of the initial score; Based on the initial score, resource status information of the online reinforcement learning algorithm is retrieved, scores of resources of different unit types are calculated, and resource scores are obtained.

[0034] In the present invention, in the initial state, referring to Table 1, in the dynamic weight adjustment method of the evaluation function based on online reinforcement learning, Represents the weight of the action unit. The specific weight parameters are as follows: Represents the weight of the base unit; Represents the weight of the barracks unit (producing combat units); Represents the weight of worker units (collecting resources, building buildings, and also attacking); A weight representing light combat units (fast and cheap); Represents the weight of ranged combat units (long-range strike); Represents the weight of heavy combat units (high health and high attack power). Represents the action unit's score for this round. Represents the resource weight that the player already has; Represents the resource weight held by the worker unit, It represents the score of the resource in this round.

[0035] Specifically, the weight adjustment in steps 14 to 17 of the algorithm flow is determined by the learning rate ( ) and the decay rate ( ) to control. After each game iteration, the weight of a unit type The adjustment is calculated using the following formula: (1)

[0036] in, and is the score before and after the update, and is the score before and after the update. This formula ensures that the weights are updated proportionally to the relative improvement in the score (controlled by the learning rate), while also applying decay (controlled by the decay rate) to reduce overfitting.

[0037] The evaluation function calculates the maximum and minimum player's score by the following formula And comprehensively consider and input AI planning methods as guidance for situation assessment: (2)

[0038] in, is the final score of the evaluation function, and are the scores calculated for the maximizing and minimizing players, respectively, The function is defined as: This normalization ensures balanced evaluations across players, mitigates extremes in score differences, enhances game strategy analysis, and makes scores calculated by different methods comparable to each other.

[0039] Table 1

[0040] Step 300: According to the difference between the current resource score and the previous score, the learning rate and decay rate in the online reinforcement learning are updated through a preset optimizer.

[0041] In the present invention, the optimizer adopts the AdamW optimizer, which is an improved version based on the Adam optimization algorithm. It mainly solves some problems of the Adam optimizer in implementing weight decay. The Adam optimizer itself combines the advantages of the momentum and RMSprop algorithms, and adjusts the learning rate by maintaining the first-order moment estimate (momentum moment) and the second-order moment estimate of the gradient, thereby improving the efficiency and effect of training deep learning models. However, in the standard Adam algorithm, weight decay is achieved by directly adding an L2 regularization term to the gradient. This approach may cause the effect of weight decay to be affected by the learning rate and momentum, making it impossible to accurately control the strength of regularization. In contrast, AdamW decouples weight decay from gradient updates and applies it directly to weight parameters, thereby ensuring the independence and consistency of weight decay. Specifically, in AdamW, the weight decay λ is multiplied by the learning rate η , and then subtracted from the parameters instead of adding to the gradient. This direct manipulation of the parameters is believed to provide better control over the complexity of the model, prevent overfitting, and in many cases improve the performance of the model.

[0042] The AdamW optimizer improves the effect and efficiency of model training by improving the implementation of weight decay. It not only retains the advantages of the Adam optimizer, but also overcomes its shortcomings in weight decay to a certain extent, making AdamW an optimization algorithm that performs well in a variety of tasks.

[0043] Specifically, the AdamW optimizer calculates the parameter gradient at each time step, updates the first-order and second-order moment estimates respectively, performs bias correction, and finally applies learning rate adjustment and weight decay to update the parameters, thereby effectively preventing overfitting and improving the generalization ability of the model. The specific process is as follows.

[0044] First, update the first-order momentum estimate : (3)

[0045] in, is the first-order momentum term coefficient, is the current step length score changes.

[0046] The second step is to update the second-order momentum estimate : (4)

[0047] in, is the coefficient of the second-order momentum term.

[0048] The third step is to calculate the bias-corrected estimate: (5)

[0049] (6)

[0050] Finally, calculate the step size The adaptive learning rate ( ) and adaptive decay rate ( ): (7)

[0051] (8)

[0052] in, and Represent the first-order moment momentum term and the second-order moment momentum term respectively; accordingly, the first-order moment deviation correction estimate and the second-order moment deviation correction estimate of the learning rate are and ; The first-order moment bias correction estimate and the second-order moment bias correction estimate of the decay rate are and ; To prevent small amounts where the denominator is zero.

[0053] Step 400: Update the unit type weights through online reinforcement learning based on the updated learning rate and decay rate, and record the latest scores and weights until the evaluation is completed.

[0054] After updating the learning rate and decay rate with the AdamW optimizer, the algorithm structure of the dynamic weight adjustment method of the evaluation function based on online reinforcement learning is as follows: Figure 3 As shown in Figure 1, it includes a scoring module (module 1), a hyperparameter update module (module 2, AdamW optimizer) and a weight update module (module 3, online reinforcement learning). The scoring elements in the RTS task can be divided into combat units ( ) and resources ( ) are of two types. Specifically, combat units include bases ( )、Barracks( ),Worker( ), Light Combat Units ( ), ranged combat units ( ) and heavy combat units ( ); Resources include resources that the player already has ( ) and the in-transit resources held by worker units ( ).

[0055] by For example, Represents the base in steps The evaluation function score at time , Represents the base in steps The score weight of the evaluation function score at time , Represents the base in steps Game status information at the time. Initially, the scoring module Calculate the score. Combined with the previous step Score Calculate the score change between two steps , and send this result to the second link (hyperparameter update module).

[0056] In the second stage, the Adamw optimizer is combined with the momentum coefficient , and the momentum estimation of the previous step is calculated based on the input ASMAINBASEt. ,based on Get the deviation correction estimate process and calculate the adaptive learning rate (L t ) and adaptive decay rate (D t ), and then input the updated hyperparameters into the third stage. In the third stage, the online reinforcement learning method (hyperparameters are determined according to the previous stage) is used to and score changes Calculate the updated weights , as the next step number Input.

[0057] Referring to Table 2, specifically to the algorithm flow, first, input the initial weights, AdamW optimizer parameters, and online reinforcement parameters. First, initialize the unit type weights and record the initial scores. When the game state needs to be evaluated, retrieve the physical game state, initialize the scores of different unit types, and calculate the resource scores. Based on the difference between the current score and the last recorded score, use the AdamW optimizer to update the learning rate and decay rate in online reinforcement learning, and use the updated learning rate and decay rate to update the unit type weights with online reinforcement learning, and use the latest scores and weights to record until the end of the evaluation.

[0058] Table 2

[0059] After using the AdamW optimizer to update the learning rate and decay rate in online reinforcement learning, the online reinforcement learning method's dependence on the initial learning rate and initial decay rate set is greatly reduced, and there is no need to manually repeat the parameter adjustment when used for different planning methods.

[0060] In a specific embodiment, the microRTS (Micro Real-Time Strategy) platform used is a mature platform for researching and developing RTS algorithms, and has been widely used in the research community to study RTS problems. microRTS provides a complex and dynamic experimental environment that is suitable for testing and verifying the effectiveness of AI strategies. This experiment aims to dynamically adjust the weight of the evaluation function through online reinforcement learning to optimize the performance of AI under different strategies.

[0061] The experimental environment is configured as follows: processor i7-8850H, memory 80GB; computing language version is Java 22.0.1; CPU time for each round of computing is 100ms. Evaluation schemes include dynamic adaptive Lanchester model, traditional Lanchester model, classic simple method and classic simple sqrt method; planning schemes include IDABCD and PortfolioAI.

[0062] A round-robin experiment was conducted based on the M1 (16×16), M2 (24×24) and M3 (32×32) maps, with the maximum number of game rounds set to 10,000. Figure 3 The middle M1 (left, 16×16), M2 (middle, 24×24), and M3 (right, 32×32) maps. The green squares are neutral resources, the blue-framed gray squares are the barracks built by the blue side, the blue-framed orange circles are the blue side's light combat units, the blue (red) framed white squares are the blue (red) side bases, and the blue (red) dots are the blue (red) side's worker units.

[0063] In the round-robin match, each method played 40 rounds with the other methods (20 rounds as player 1 and 20 rounds as player 2). The evaluation schemes included the dynamic adaptive Lanchester model (DL), the dynamic adaptive simple evaluation function (DS), the dynamic adaptive simple sqrt evaluation function (DSQ), the traditional Lanchester model (L), the classic simple evaluation function (S), and the classic simple sqrt evaluation function (SQ); the planning schemes included IDABCD, IDRTMinimax, and PortfolioAI. Therefore, when running 6 evaluation functions (15 combinations of 40 rounds of battles in pairs) on 3 planning schemes for testing on 3 maps, a total of 5400 rounds of battles were required (3×15×40×3).

[0064] Select the commonly used AdamW parameter momentum term coefficient and the second-order momentum coefficient , for the initial learning rate Tests were conducted. According to the results, factors including the standard algorithm, map size, and the characteristics of the evaluation function itself will affect the improvement effect of the dynamic weight adjustment method on the evaluation function. For IDABCD, the improvement of the dynamic weight adjustment method on the dynamic adaptive simple sqrt evaluation function and the dynamic adaptive Lanchester model is more obvious in the three map sizes. However, the improvement effect of the dynamic adaptive simple evaluation function is not obvious in M1, which may be due to the greater randomness of small-size maps. The impact of map size on the results is more obvious in IDRTMinimax. In M1, only DL is significantly improved compared to L. As the map size increases, the improvement effect of the two improved evaluation functions DS and DSQ relative to the original evaluation functions S and SQ becomes more obvious. However, in Portfolio, the improvement effect is not significantly affected by the map size. The improvement effects of DS and DSQ are always more significant, while the improvement effect of DL is always smaller. Reference Figure 4 In M1, the round-robin results obtained by using the Lanchester evaluation function (DL), Simple evaluation function (DS), and SimpleSqrt evaluation function (DSQ) with dynamic weight adjustment based on online reinforcement learning of AdamW; and the non-dynamic Lanchester evaluation function (L), Simple evaluation function (S), and SimpleSqrt evaluation function (SQ) applied to the AI ​​methods of IDABCD, IDRTMinimax, and Portfolio. Figure 5 In M2, the round-robin results obtained by using the Lanchester evaluation function (DL), Simple evaluation function (DS), and SimpleSqrt evaluation function (DSQ) with dynamic weight adjustment based on AdamW online reinforcement learning; and the non-dynamic Lanchester evaluation function (L), Simple evaluation function (S), and SimpleSqrt evaluation function (SQ) applied to the AI ​​methods of IDABCD, IDRTMinimax, and Portfolio. Figure 6 In M3, the round-robin results were obtained by using the Lanchester evaluation function (DL), Simple evaluation function (DS) and SimpleSqrt evaluation function (DSQ) with dynamic weight adjustment based on AdamW online reinforcement learning; and the non-dynamic Lanchester evaluation function (L), Simple evaluation function (S) and SimpleSqrt evaluation function (SQ) applied to the AI ​​methods of IDABCD, IDRTMinimax and Portfolio.

[0065] Based on the reinforcement learning-based adaptive evaluation function dynamic weight adjustment method provided by the present invention, a breakthrough has been made in the static weight evaluation technology. In view of the problem of insufficient real-time performance of the evaluation function, dynamic weight adjustment is achieved through online reinforcement learning, so that the evaluation function can adapt to changes in battlefield situation in real time. At the same time, the parameter adaptive update method based on AdamW is studied, and the hyperparameter automatic adjustment is completed, which improves the robustness and versatility of the algorithm. It provides new ideas and methods for military situation assessment in complex dynamic environments in the future.

[0066] refer to Figure 7 The present invention also discloses a dynamic weight adjustment system for adaptive evaluation function based on reinforcement learning, the system comprising: The data acquisition module 110 is used to obtain the initial weights, optimizer parameters and online reinforcement parameters of the preset online reinforcement learning algorithm; A score calculation module 120, configured to initialize the unit type weights based on the initial weights and record the initial scores, retrieve the physical state of the online reinforcement learning algorithm based on the initial scores, initialize the scores of different unit types, and calculate the resource scores; An updating module 130, used to update the learning rate and decay rate in the online reinforcement learning through a preset optimizer according to the difference between the current resource score and the previous score; The online reinforcement learning module 140 is used to update the unit type weights through online reinforcement learning based on the updated learning rate and decay rate, and record the latest scores and weights until the evaluation is completed.

[0067] The initialization of the unit type weight based on the initial weight and recording the initial score, retrieving the physical state of the online reinforcement learning algorithm based on the initial score, initializing the scores of different unit types, and calculating the resource score specifically include: The initial score is obtained after weight initialization; Based on the initial score, resource status information of the online reinforcement learning algorithm is retrieved, scores of resources of different unit types are calculated, and resource scores are obtained.

[0068] The updating of the learning rate and decay rate in the online reinforcement learning by the preset optimizer specifically includes: The parameter gradient is calculated at each time step through the preset optimizer, and the first-order and second-order moment estimates in the parameter gradient are updated respectively; Bias correction is performed based on the updated results of the first-order and second-order moment estimates, and parameters are updated through learning rate adjustment and weight decay.

[0069] The preset optimizer calculates the parameter gradient at each time step and updates the first-order and second-order moment estimates respectively, specifically including: The parameter gradient is calculated at each time step through the preset optimizer, and the first-order momentum estimate is updated; Updating the second-order momentum estimation based on the first-order momentum estimation result to generate a second-order momentum estimation result; The deviation correction estimate is calculated based on the second-order momentum estimation result to complete the first-order and second-order moment estimation.

[0070] The unit type weights are updated through online reinforcement learning based on the updated learning rate and decay rate, and the latest scores and weights are recorded until the evaluation is completed, specifically including: Calculate the score weight of the evaluation function in the updated unit type through online reinforcement learning based on the updated learning rate and decay rate; The score weight of the evaluation function in the updated unit type is used as the input for the next update until the evaluation is completed, and the latest score and weight are recorded.

[0071] Based on the reinforcement learning-based adaptive evaluation function dynamic weight adjustment system provided by the present invention, dynamic weight adjustment is achieved through online reinforcement learning, so that the evaluation function can adapt to battlefield situation changes in real time. At the same time, the optimizer-based parameter adaptive update method is studied, the hyperparameter automatic adjustment is completed, and the robustness and versatility of the algorithm are improved. It provides new ideas and methods for military situation assessment in complex dynamic environments in the future.

[0072] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning, the method comprising: obtaining the initial weight, optimizer parameters and online reinforcement parameters of a preset online reinforcement learning algorithm; initializing the unit type weight based on the initial weight and recording the initial score, retrieving the physical state of the online reinforcement learning algorithm based on the initial score, initializing the scores of different unit types, and calculating the resource score; updating the learning rate and decay rate in the online reinforcement learning through the preset optimizer according to the difference between the current resource score and the last score; updating the unit type weight through the online reinforcement learning based on the updated learning rate and decay rate, and recording the latest score and weight until the evaluation is completed.

[0073] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0074] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning provided by the above methods, the method including: obtaining the initial weights, optimizer parameters and online reinforcement parameters of a preset online reinforcement learning algorithm; initializing the unit type weights based on the initial weights and recording the initial scores, retrieving the physical state of the online reinforcement learning algorithm based on the initial scores, initializing the scores of different unit types, and calculating the resource scores; updating the learning rate and decay rate in the online reinforcement learning through a preset optimizer based on the difference between the current resource score and the last score; updating the unit type weights through online reinforcement learning based on the updated learning rate and decay rate, and recording the latest scores and weights until the evaluation is completed.

[0075] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute a method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning provided by the above-mentioned methods, the method comprising: obtaining the initial weights, optimizer parameters and online reinforcement parameters of a preset online reinforcement learning algorithm; initializing the unit type weights based on the initial weights and recording the initial scores, retrieving the physical state of the online reinforcement learning algorithm based on the initial scores, initializing the scores of different unit types, and calculating the resource scores; updating the learning rate and decay rate in the online reinforcement learning through a preset optimizer based on the difference between the current resource score and the last score; updating the unit type weights through online reinforcement learning based on the updated learning rate and decay rate, and recording the latest scores and weights until the evaluation is completed.

[0076] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamic weight adjustment of adaptive evaluation function based on reinforcement learning, characterized in that: include: Get the initial weights, optimizer parameters, and online reinforcement parameters of the preset online reinforcement learning algorithm; Initializing the unit type weights based on the initial weights and recording the initial scores, retrieving the physical state of the online reinforcement learning algorithm based on the initial scores, initializing the scores of different unit types, and calculating the resource scores; According to the difference between the current resource score and the last score, the learning rate and decay rate in online reinforcement learning are updated through the preset optimizer; Update the unit type weights through online reinforcement learning based on the updated learning rate and decay rate, and record the latest scores and weights until the end of the evaluation.

2. The method for dynamic weight adjustment of adaptive evaluation function based on reinforcement learning according to claim 1, characterized in that: The unit type weights are initialized based on the initial weights and the initial scores are recorded, the physical state of the online reinforcement learning algorithm is retrieved based on the initial scores, the scores of different unit types are initialized, and the resource scores are calculated, specifically including: The initial score is obtained after weight initialization; Based on the initial score, resource status information of the online reinforcement learning algorithm is retrieved, scores of resources of different unit types are calculated, and resource scores are obtained.

3. The method for dynamic weight adjustment of adaptive evaluation function based on reinforcement learning according to claim 1, characterized in that: The updating of the learning rate and decay rate in the online reinforcement learning by the preset optimizer specifically includes: The parameter gradient is calculated at each time step through the preset optimizer, and the first-order and second-order moment estimates in the parameter gradient are updated respectively; Bias correction is performed based on the updated results of the first-order and second-order moment estimates, and parameters are updated through learning rate adjustment and weight decay.

4. The method for dynamic weight adjustment of adaptive evaluation function based on reinforcement learning according to claim 3, characterized in that: The preset optimizer calculates the parameter gradient at each time step and updates the first-order and second-order moment estimates respectively, specifically including: The parameter gradient is calculated at each time step through the preset optimizer, and the first-order momentum estimate is updated; Updating the second-order momentum estimation based on the first-order momentum estimation result to generate a second-order momentum estimation result; The deviation correction estimate is calculated based on the second-order momentum estimation result to complete the first-order and second-order moment estimation.

5. The method for dynamic weight adjustment of adaptive evaluation function based on reinforcement learning according to claim 1, characterized in that: The unit type weights are updated through online reinforcement learning based on the updated learning rate and decay rate, and the latest scores and weights are recorded until the evaluation is completed, specifically including: Calculate the score weight of the evaluation function in the updated unit type through online reinforcement learning based on the updated learning rate and decay rate; The score weight of the evaluation function in the updated unit type is used as the input for the next update until the evaluation is completed, and the latest score and weight are recorded.

6. A dynamic weight adjustment system for adaptive evaluation function based on reinforcement learning, characterized in that: The system comprises: A data acquisition module is used to obtain the initial weights, optimizer parameters and online reinforcement parameters of a preset online reinforcement learning algorithm; A score calculation module, used to initialize the unit type weights based on the initial weights and record the initial scores, retrieve the physical state of the online reinforcement learning algorithm based on the initial scores, initialize the scores of different unit types, and calculate the resource scores; The update module is used to update the learning rate and decay rate in online reinforcement learning through a preset optimizer according to the difference between the current resource score and the previous score; The online reinforcement learning module is used to update the unit type weights through online reinforcement learning based on the updated learning rate and decay rate, and record the latest scores and weights until the evaluation ends.

7. The adaptive evaluation function dynamic weight adjustment system based on reinforcement learning according to claim 6, characterized in that: The update module calculates the parameter gradient at each time step through a preset optimizer, and updates the first-order and second-order moment estimates in the parameter gradient respectively; Bias correction is performed based on the updated results of the first-order and second-order moment estimates, and parameters are updated through learning rate adjustment and weight decay.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning as described in any one of claims 1 to 5 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning as described in any one of claims 1 to 5 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for dynamic weight adjustment of an adaptive evaluation function based on reinforcement learning as described in any one of claims 1 to 5 is implemented.

Images