Agent environment generation method and device based on regret estimation
By generating an environment map based on the regret matrix of tiles, the problems of credit allocation failure and low training efficiency in the existing technology are solved, enabling more targeted agent training and improving training efficiency and continuity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TSINGHUA SHENZHEN INTERNATIONAL GRADUATE SCHOOL
- Filing Date
- 2026-04-30
- Publication Date
- 2026-05-29
AI Technical Summary
Existing environment generation methods struggle to accurately identify the weaknesses of agents in complex and ever-changing environments, leading to ineffective credit allocation, low training efficiency, and a lack of specificity.
By generating an environment map based on the regret matrix of tile pairs, recording the regret values of the robot agent in different terrain tile transfer pairs, updating the regret matrix to improve training targeting, and generating terrain tiles for tile transfer pairs with high regret values.
This improves training efficiency, ensures that the robot agent can train its weak points multiple times, avoids ineffective repetitive training, and enhances the relevance and continuity of training.
Smart Images
Figure CN122113992A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for generating intelligent agent environments based on regret estimation. Background Technology
[0002] In the fields of autonomous navigation and reinforcement learning for robots, efficiently training agents to adapt to complex and ever-changing environments is a key challenge. Unsupervised Environment Design (UED) technology addresses this problem by automatically generating training environments, avoiding the high cost and incompleteness of manually designed environments. UED is a typical training method based on the Automated Curriculum Learning paradigm. Its core idea is to automatically generate a series of training environments through a "teacher" algorithm to improve the generalization ability of the "student" (i.e., the robot agent). In this field, regret is a core evaluation metric used to quantify the performance gap between the agent's current policy and the optimal policy. However, calculating regret in complex environments is difficult because it requires obtaining the optimal policy for that environment. Therefore, existing environment generation methods typically rely on heuristic functions to estimate regret.
[0003] In existing priority-level replay (PLR) environment generation methods, the regret value estimation primarily relies on global metrics, such as the maximum cumulative return or the average advantage over the entire episode. However, this regret value estimation method based on global metrics suffers from the following problems when handling long-sequence combinatorial tasks: 1. Credit Assignment Failure: The global metric treats the entire map as a "black box." If the robot agent fails because it cannot pass through a specific structure in the middle of the map (e.g., going down a slope to flat ground), the global metric can only provide feedback such as "the map is difficult" or "training failed," without being able to further locate the specific terrain structure, connection relationship, or local transfer pattern that caused the failure. Therefore, it is difficult to provide effective and accurate feedback information for subsequent map generation.
[0004] 2. Low training efficiency and lack of specificity: Current technology cannot identify the specific weaknesses of the agent in the environmental map, nor does it possess the ability to fine-grainedly evaluate local terrain units or local transition relationships. Therefore, the environment generator struggles to generate targeted maps based on the agent's actual shortcomings. Furthermore, even if the system detects that the robot agent performs poorly on a certain type of terrain, it cannot directly guide the environment generator to increase the proportion of the corresponding local map, resulting in insufficient targeting of the training curriculum and reduced training efficiency. Summary of the Invention
[0005] This invention provides a method and apparatus for generating intelligent agent environment based on regret estimation, which solves the problems of credit allocation failure, low training efficiency and poor training targeting in existing environment generation methods.
[0006] This invention provides a method for generating an intelligent agent's environment based on regret estimation, comprising the following steps: Step S1: Generate an environment map based on the tile pair regret matrix, wherein the tile pair regret matrix records the regret value of the tile transfer pair in the environment map when the robot agent moves from the first terrain tile to the second terrain tile of a different terrain type, and the larger the regret value, the greater the probability of the tile transfer pair being generated in the environment map. Step S2: Obtain the transition results of each tile transition pair statistically analyzed by the robot agent during the current training round based on the environment map; Step S3: Estimate the regret value of each tile transfer pair based on the transfer results. The higher the transfer failure rate, the larger the regret value, so as to obtain a temporary regret matrix. Step S4: Update the tile pair regret matrix based on the temporary regret matrix, and jump to step S1 until the robot agent training ends.
[0007] According to the present invention, an agent-based environment generation method based on regret estimation generates an environment map based on a tile-based regret matrix, comprising: Step S11: Obtain the generated current terrain tile; Step S12: Query the regret value of each tile transfer pair from the current terrain tile to other terrain tiles in the tile pair regret matrix; Step S13: Determine the sampling weight of each other topographic tile as the next topographic tile to be generated based on the regret value of each tile transfer pair. The larger the regret value, the larger the corresponding sampling weight. Step S14: Perform weighted probability sampling on each of the other terrain tiles based on the sampling weight to obtain the next generated terrain tile, and jump to step S11 until all tiles of the environmental map are generated.
[0008] According to the present invention, an agent-based environment generation method based on regret estimation determines the sampling weight of each other terrain tile as the next terrain tile to be generated based on the regret value of each tile transition pair, including calculating the sampling weight according to the following formula. : ; in, Indicates temperature parameter, Indicates the current topographic map tile. This indicates the next terrain tile to be generated. The tile pair in the regret matrix represents the... Transferred to The regret value, It represents a proportional relationship operation.
[0009] According to the present invention, an agent-based environment generation method based on regret estimation estimates the regret value of each tile transition pair based on the transition results, wherein the higher the transition failure rate, the larger the regret value, including: Based on the transfer results of each tile transfer pair during the current training round, the number of transfer failures and the total number of transfers for each tile transfer pair are obtained, and the transfer failure rate is calculated. The transfer failure rate is used as the regret value of the tile transfer pair.
[0010] According to the present invention, an agent-based environment generation method based on regret estimation estimates the regret value of each tile transition pair based on the transition results, wherein the higher the transition failure rate, the larger the regret value, including: Based on the transfer results of each tile transfer pair during the current training round, the number of transfer failures and the total number of transfers for each tile transfer pair are obtained, and the transfer failure rate and transfer success rate are calculated. The product of the transfer failure rate and the transfer success rate is used as the regret value of the tile transfer pair.
[0011] According to the present invention, an agent-based environment generation method based on regret estimation updates the tile pair regret matrix based on the temporary regret matrix, comprising: The temporary regret matrix is determined to be a new tile pair regret matrix, or a new tile pair regret matrix is obtained by weighted merging of the temporary regret matrix and the current tile pair regret matrix.
[0012] The present invention also provides an intelligent agent environment generation device based on regret estimation, comprising the following modules: An environment generation module is used to generate an environment map based on a tile pair regret matrix. The tile pair regret matrix records the regret value of a tile transfer pair in the environment map where the robot agent moves from a first terrain tile to a second terrain tile of a different terrain type. The larger the regret value, the greater the probability that the tile transfer pair will be generated in the environment map. The local transfer tracking module is used to obtain the transfer results of each tile transfer pair statistically analyzed by the robot agent during the current training round based on the environment map; The regret value estimation module is used to estimate the regret value of each tile transfer pair based on the transfer results. The higher the transfer failure rate, the larger the regret value, so as to obtain a temporary regret matrix. The matrix update module is used to update the tile pair regret matrix based on the temporary regret matrix, and then jump to the execution environment generation module until the robot agent training is completed.
[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the agent environment generation method based on regret estimation as described above.
[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the agent environment generation method based on regret estimation as described above.
[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the agent environment generation method based on regret estimation as described above.
[0016] The present invention provides an agent-based environment generation method and apparatus based on regret estimation. The method comprises the following steps: Step S1: Generating an environment map based on a tile-pair regret matrix. The tile-pair regret matrix records the regret value of tile transfer pairs in the environment map when the robot agent moves from a first terrain tile to a second terrain tile of a different terrain type. A higher regret value indicates a higher probability of generating a tile transfer pair in the environment map. Step S2: Obtaining the transfer results of each tile transfer pair during the current training round based on the environment map. Step S3: Estimating the regret value of each tile transfer pair based on the transfer results. A higher transfer failure rate corresponds to a higher regret value, resulting in a temporary regret matrix. Step S4: Updating the tile-pair regret matrix based on the temporary regret matrix and returning to Step S1 until the robot agent training ends. Through these steps, the method tends to generate terrain tiles with high regret values for tile transfer pairs when generating the environment map, making training more targeted and improving training efficiency. Furthermore, it can distinguish tile transfer pairs that the robot agent is not good at during training, rather than simply judging the map as too difficult, thus solving the problem of credit allocation. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0018] Figure 1 This is one of the flowcharts of the intelligent agent environment generation method based on regret estimation provided by the present invention.
[0019] Figure 2 This is a diagram of the agent training framework in the agent environment generation method based on regret estimation provided by the present invention.
[0020] Figure 3 This is the second flowchart of the intelligent agent environment generation method based on regret estimation provided by the present invention.
[0021] Figure 4 This is a schematic diagram of the intelligent agent environment generation device based on regret estimation provided by the present invention.
[0022] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0024] In existing related technologies, it is regrettable that they are not used to quantify the performance gap between the agent's current policy and the optimal policy, given a policy. and environmental parameters The formal definition of regret is as follows: ; in, In environmental parameters The optimal strategy under these circumstances It is a strategy The expected discounted return, where, This represents the total number of steps in one round of reinforcement learning (one round is defined as the total number of steps taken by the robot agent). Indicates the step index. This represents the reward discount factor in reinforcement learning. Indicates the first The regret value is calculated based on the reward value of each step. Since optimal strategies are often difficult to obtain, it's usually challenging to calculate the regret value using the formula above. Therefore, existing methods typically rely on heuristic functions to estimate the regret. In priority-based replay environment generation methods, the regret value is primarily estimated using global metrics, such as the maximum cumulative return value for the entire episode. .
[0025] Alternatively, the average of advantages: .
[0026] in, This represents the maximum single-step reward in a round. Indicates the first intelligent agent The state of the step, Represents the state value function. This represents the index-weighted coefficient for the dominance estimate. Indicates the first The time difference error of the step.
[0027] However, this regret value estimation method based on global metrics suffers from problems such as credit allocation failure, low sampling efficiency, and poor training specificity when dealing with long-sequence compositional tasks (such as complex terrain navigation tasks).
[0028] To address the aforementioned problems in existing related technologies, embodiments of the present invention provide a method for generating intelligent agent environments based on regret estimation. Figure 1 This is a flowchart illustrating the agent environment generation method based on regret estimation provided by the present invention, as shown below. Figure 1 As shown, the method includes steps S110 to S140.
[0029] Step S110: Generate an environment map based on the tile pair regret matrix, wherein the tile pair regret matrix records the regret value of the tile transfer pair in the environment map when the robot agent moves from the first terrain tile to the second terrain tile of a different terrain type, and the larger the regret value, the greater the probability of the tile transfer pair being generated in the environment map.
[0030] It is understandable that an environmental map includes multiple topographic tiles, that is, it is composed of several discrete topographic tiles pieced together. These topographic tiles have different terrain types, such as flat land, highlands, depressions, and slopes (uphill and downhill). Definition and There are two different types of terrain tiles, which correspond to the regret matrix. elements in Indicates from Move to The regret value of this tile transfer pair is a fine-grained regret value related to the local transfer of two specific terrain tiles during a training round, rather than a global regret value. Through this fine-grained regret value of the tile transfer pair, it is possible to distinguish whether the robot agent cannot "go uphill" or "go downhill", that is, to distinguish the tile transfer pairs that the agent is not good at, rather than generally judging that the environment map is too difficult. This solves the problem of credit allocation and can provide effective and accurate feedback for subsequent environment map generation.
[0031] Specifically, such as Figure 2 As shown, the robot agent needs to generate an environment map before each training round. The environment map can be generated based on the regret matrix of the tile pair by an environment generator (which can be implemented based on wave function collapse WFC or other stitching algorithms).
[0032] It should be noted that during the first iteration (one iteration is one cycle of steps S110 to S140, including one environment map generation and one round of training for the robot agent), the tile pair regret matrix can be initialized to be a matrix with all zeros or a uniform distribution (i.e. all element values are equal). Since the tile pair regret matrix is a uniformly distributed matrix, the number of terrain tiles of each type in the generated environment map is also basically equal and the distribution is relatively uniform.
[0033] Step S120: Obtain the transition results of each tile transition pair statistically analyzed by the robot agent during the current training round based on the environment map.
[0034] After generating the environment map, the robot agent is trained on the environment map. For example, given a training round of 1000 steps, the robot agent moves 1000 steps on the environment map. It can move randomly, or it can move according to a pre-planned route, or it can move according to a self-planned route based on a specified starting point and target point.
[0035] like Figure 2 As shown, during the movement of the robot agent, a local transfer tracker can be used to record the local transfer data of the robot agent, thereby tracking the movement trajectory of the robot agent and recording each tile transfer pair, that is, recording the robot agent's movement from type 1 to 2. The terrain tile was moved to a type of The terrain tiles are analyzed, and the transfer results for each tile transfer pair are statistically analyzed. This can be understood as: the statistical transfer results can include: the total number of transfers for each tile transfer pair and the number of failed transfers (or successful transfers), indicating that the robotic agent successfully transferred tiles from the terrain map. Move to topographic tile If the result is positive, the transfer is successful; otherwise, the transfer fails (e.g., the robot falls or encounters an obstacle). For example, the transfer result can be cached in a trajectory cache along with the robot agent's movement trajectory data.
[0036] Step S130: Estimate the regret value of each tile transfer pair based on the transfer results, and the higher the transfer failure rate, the larger the regret value, so as to obtain a temporary regret matrix.
[0037] Specifically, the transfer results for each tile transfer pair can be read from the trajectory cache. A higher transfer failure rate indicates that the robot agent is less skilled or less adept at transferring that tile transfer pair, and therefore has a larger regret value. After each training round, the regret value is estimated based on the transfer results of each tile transfer pair statistically analyzed during that round, resulting in a temporary regret matrix. That is, the temporary regret matrix reflects which tile transfer pairs the robot agent is good at transferring and which it is not good at transferring in this training round. For example, such as... Figure 3 As shown, the regret value can be estimated by setting a fine-grained regret estimator.
[0038] Step S140: Update the tile pair regret matrix based on the temporary regret matrix, and proceed to step S110 until the robot agent training ends. That is, after each training round, the tile pair regret matrix is updated. After proceeding to step S110, the environment generator regenerates the environment map based on the updated tile pair regret matrix. The robot agent then performs the next training on the regenerated environment map until training ends (e.g., reaching a predetermined number of training rounds or a preset total transfer success rate). Since the updated tile pair regret matrix reflects which tile transfer pairs the robot agent was not proficient in during the previous training round, i.e., the regret values of the unproficient tile transfer pairs are larger in the updated tile pair regret matrix, the probability of generating terrain tiles for these unproficient tile transfer pairs in the environment map is also higher when proceeding to S110. This allows the robot agent to train multiple times on previously unproficient tile transfer pairs, making the training more targeted and avoiding ineffective repetitive training on many proficient tile transfer pairs, thereby improving training efficiency. Experiments show that the number of sampling steps required to achieve the same success rate (total transfer success rate in one training round) is significantly reduced.
[0039] In the agent environment generation method based on regret estimation in this embodiment, the above steps S110 to S140 solve the problems of credit allocation failure, low training efficiency and poor training targeting in the existing environment generation methods.
[0040] In some embodiments, step S110, generating an environment map based on the tile pair regret matrix, includes: Step S111: Obtain the generated current terrain tile.
[0041] Step S112: Query the regret value of each tile transfer pair from the current terrain tile to other terrain tiles in the tile pair regret matrix.
[0042] Step S113: Determine the sampling weight of each other topographic tile as the next topographic tile to be generated based on the regret value of each tile transfer pair. The larger the regret value, the larger the corresponding sampling weight.
[0043] Step S114: Perform weighted probability sampling on each of the other terrain tiles based on the sampling weight to obtain the next generated terrain tile, and jump to step S111 until all terrain tiles of the environment map are generated.
[0044] Specifically, such as Figure 3As shown, obtaining the current terrain tile also yields its terrain type, and also a set of terrain tiles of other terrain types besides the current one, i.e., a candidate terrain set. It should be noted that when generating the environment map for the first time, and generating the first terrain tile, a terrain tile of a random type can be generated. For example: the current terrain tile is T. current The candidate terrain set formed by terrain tiles of other terrain types includes: T A T B and T C Topographic tiles of three terrain types. Query the current topographic tile T in the regret matrix. current To other topographic tiles T A T B and T C The transfer of each tile to T current →T A T current →T B and T current →T C The regret value. Based on the transition of each tile to T current →T A T current →T B and T current →T C The regret value determines the sampling weight of each other topographic tile as the next topographic tile to be generated, and the larger the regret value, the larger the corresponding sampling weight. Based on the sampling weight, weighted probability sampling is performed on each other topographic tile to obtain the next generated topographic tile. Since the larger the sampling weight, the greater the probability of being sampled during weighted probability sampling, the transfer of each tile to T... current →T A T current →T B and T current →T C The higher the regret value, the higher the T value of the destination shape tile in the tile transfer pair. A T B and T C Sampled and used as the next generated topographic tile T next The higher the probability, for example: T current →T A If the regret value is large, then T A As T next The probability of it being generated is relatively high. The next terrain tile T to be generated will be... next The process continues until a complete environment map is formed, then the process ends; otherwise, it jumps to retrieve the currently generated terrain tiles.
[0045] In this embodiment, the sampling weight of each other terrain tile as the next terrain tile to be generated is determined by the regret value of each tile transfer pair. The larger the regret value, the larger the corresponding sampling weight. This results in a larger sampling weight for the destination shape tile in the tile transfer pair with a larger regret value, and a greater probability that the destination shape tile will be generated as the next terrain tile. As a result, the training of the robot agent is more targeted.
[0046] Specifically, the sampling weights are calculated according to the following formula. : .
[0047] in, This represents a temperature parameter, with values ranging from 0.1 to 10.0. Indicates the current topographic map tile. This indicates the next terrain tile to be generated. The tile pair in the regret matrix represents the... Transferred to The regret value, It represents a proportional relationship operation.
[0048] In this embodiment, by... The proportional relationship determines the tile transfer pair → middle The sampling weights ensure that destination shape tiles in tile transfer pairs with larger regret values are generated preferentially or with a high probability.
[0049] In some embodiments, a regret value for each tile transfer pair is estimated based on the transfer results, and the higher the transfer failure rate, the greater the regret value. Specifically, this includes: Based on the transfer results of each tile transfer pair during the current training round, the number of failed transfers and the total number of transfers for each tile transfer pair are obtained, and the transfer failure rate is calculated. This failure rate is used as the regret value of the tile transfer pair. For example: in N The second attempt to transfer from flat ground to a slope failed. M The transfer failure rate was [number] times. M / N That is, the regret value of the "flat land -> slope" tile transfer pair is M / N .
[0050] In this embodiment, the transfer failure rate is directly used as the regret value of the tile transfer pair. Moreover, the larger the transfer failure rate of the tile transfer pair, the larger the corresponding regret value. This makes it easier or more inclined to generate terrain tiles corresponding to tile transfer pairs with larger regret values when generating environment maps, and the training is more targeted.
[0051] In some embodiments, a regret value for each tile transfer pair is estimated based on the transfer results, and the higher the transfer failure rate, the greater the regret value. Specifically, this includes: Based on the transfer results of each tile transfer pair during the current training round, the number of transfer failures and the total number of transfers for each tile transfer pair are obtained. The transfer failure rate and transfer success rate are then calculated, and the product of the failure rate and the success rate is used as the regret value of the tile transfer pair. For example: in N The second attempt to transfer from flat ground to a slope failed. M The transfer failure rate was [number] times. M / N The transfer success rate is ( N - M ) / N That is, the regret value of the "flat ground -> slope" tile transfer pair is (( N - M ) / N )×( M / N ).
[0052] In this embodiment, instead of directly using the transfer failure rate as the regret value of the tile transfer pair, the product of the transfer failure rate and the transfer success rate is used as the regret value of the tile transfer pair. This balances the regret values of each tile transfer pair, that is, it reduces the gap between the regret values of tile transfer pairs with high transfer failure rates and those with low transfer failure rates. During the environmental map generation process, while ensuring strong training targeting, it does not generate too many terrain tiles corresponding to tile transfer pairs with extremely high (or too large) regret values, making the training difficulty of the robot agent for the generated environmental map moderate.
[0053] In some embodiments, updating the tile pair regret matrix based on the temporary regret matrix includes: The temporary regret matrix is determined to be a new tile pair regret matrix, or a new tile pair regret matrix is obtained by weighted merging of the temporary regret matrix and the current tile pair regret matrix.
[0054] Specifically, the temporary regret matrix can be directly used as the new tile pair regret matrix.
[0055] Preferably, a new tile pair regret matrix is obtained by weighted merging of the temporary regret matrix and the current tile pair regret matrix. That is, the new tile pair regret matrix is obtained by weighted summation of the elements at corresponding positions of the temporary regret matrix and the current tile pair regret matrix, with each weight summing to 1, for example, each weight is 0.5.
[0056] In this embodiment, a new tile-pair regret matrix is obtained by weighted merging the temporary regret matrix with the current tile-pair regret matrix. This avoids the problem that large changes in the new tile-pair regret matrix lead to large changes in the difficulty of the newly generated environment map, causing the overall success rate to fluctuate back and forth in each round (instead of tending to increase). In other words, it avoids the problem of unstable training process, so that the preset success rate (the total transfer success rate of a training round) can be achieved after fewer training rounds. This makes the generated map more consistent with the current ability level of the robot agent, ensures that the difficulty of the environment map changes more smoothly and reasonably, improves the continuity, stability and convergence effect of the training process, and further improves training efficiency.
[0057] The following describes the intelligent agent environment generation device based on regret estimation provided by the present invention. The intelligent agent environment generation device based on regret estimation described below can be referred to in correspondence with the intelligent agent environment generation method based on regret estimation described above.
[0058] The intelligent agent environment generation device based on regret estimation according to embodiments of the present invention, such as Figure 4 As shown, it includes: The environment generation module 410 is used to generate an environment map based on a tile pair regret matrix. The tile pair regret matrix records the regret value of a tile transfer pair in the environment map where the robot agent moves from a first terrain tile to a second terrain tile of a different terrain type. The larger the regret value, the greater the probability that the tile transfer pair will be generated in the environment map.
[0059] The local transfer tracking module 420 is used to obtain the transfer results of each tile transfer pair statistically analyzed by the robot agent during the current training round based on the environment map.
[0060] The regret value estimation module 430 is used to estimate the regret value of each tile transfer pair based on the transfer results, and the higher the transfer failure rate, the larger the regret value, so as to obtain a temporary regret matrix.
[0061] The matrix update module 440 is used to update the tile pair regret matrix based on the temporary regret matrix, and then jump to the execution environment generation module until the robot agent training ends.
[0062] In some embodiments, the environment generation module 410 includes: The Current Topographic Patch Acquisition Module is used to acquire the currently generated topographic patches.
[0063] The regret value query module is used to query the regret value of each tile transfer pair from the current terrain tile to other terrain tiles in the tile pair regret matrix.
[0064] The sampling weight determination module is used to determine the sampling weight of each other topographic tile as the next topographic tile to be generated based on the regret value of each tile transfer pair. The larger the regret value, the larger the corresponding sampling weight. The weighted sampling module is used to perform weighted probability sampling on each other terrain tile based on the sampling weight, obtain the next generated terrain tile, and jump to execute the current terrain tile acquisition module until all tiles of the environmental map are generated.
[0065] In some embodiments, the sampling weight determination module is specifically used to calculate the sampling weight according to the following formula. : .
[0066] in, Indicates temperature parameter, Indicates the current topographic map tile. This indicates the next terrain tile to be generated. The tile pair in the regret matrix represents the... Transferred to The regret value, It represents a proportional relationship operation.
[0067] In some embodiments, the regret value estimation module 430 is specifically used to obtain the number of times each tile transfer pair failed to transfer and the total number of transfers based on the transfer results of each tile transfer pair statistically analyzed during the current training round, and to calculate the transfer failure rate, using the transfer failure rate as the regret value of the tile transfer pair.
[0068] In some embodiments, the regret value estimation module 430 is specifically used to obtain the number of times each tile transfer pair failed to transfer and the total number of transfers based on the transfer results of each tile transfer pair statistically analyzed during the current training round, and to calculate the transfer failure rate and the transfer success rate, and to use the product of the transfer failure rate and the transfer success rate as the regret value of the tile transfer pair.
[0069] In some embodiments, the matrix update module 440 is specifically used to determine that the temporary regret matrix is a new tile pair regret matrix, or to obtain a new tile pair regret matrix by weighted merging of the temporary regret matrix and the current tile pair regret matrix.
[0070] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a regret estimation-based agent environment generation method, which includes: Step S1: Generate an environment map based on the tile pair regret matrix, wherein the tile pair regret matrix records the regret value of the tile transfer pair in the environment map when the robot agent moves from the first terrain tile to the second terrain tile of a different terrain type, and the larger the regret value, the greater the probability of the tile transfer pair being generated in the environment map.
[0071] Step S2: Obtain the transition results of each tile transition pair statistically analyzed by the robot agent during the current training round based on the environment map.
[0072] Step S3: Estimate the regret value of each tile transfer pair based on the transfer results. The higher the transfer failure rate, the larger the regret value, so as to obtain a temporary regret matrix.
[0073] Step S4: Update the tile pair regret matrix based on the temporary regret matrix, and jump to step S1 until the robot agent training ends.
[0074] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0075] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the agent environment generation method based on regret estimation provided by the above methods, the method comprising: Step S1: Generate an environment map based on the tile pair regret matrix, wherein the tile pair regret matrix records the regret value of the tile transfer pair in the environment map when the robot agent moves from the first terrain tile to the second terrain tile of a different terrain type, and the larger the regret value, the greater the probability of the tile transfer pair being generated in the environment map.
[0076] Step S2: Obtain the transition results of each tile transition pair statistically analyzed by the robot agent during the current training round based on the environment map.
[0077] Step S3: Estimate the regret value of each tile transfer pair based on the transfer results. The higher the transfer failure rate, the larger the regret value, so as to obtain a temporary regret matrix.
[0078] Step S4: Update the tile pair regret matrix based on the temporary regret matrix, and jump to step S1 until the robot agent training ends.
[0079] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the agent environment generation method based on regret estimation provided by the methods described above, the method comprising: Step S1: Generate an environment map based on the tile pair regret matrix, wherein the tile pair regret matrix records the regret value of the tile transfer pair in the environment map when the robot agent moves from the first terrain tile to the second terrain tile of a different terrain type, and the larger the regret value, the greater the probability of the tile transfer pair being generated in the environment map.
[0080] Step S2: Obtain the transition results of each tile transition pair statistically analyzed by the robot agent during the current training round based on the environment map.
[0081] Step S3: Estimate the regret value of each tile transfer pair based on the transfer results. The higher the transfer failure rate, the larger the regret value, so as to obtain a temporary regret matrix.
[0082] Step S4: Update the tile pair regret matrix based on the temporary regret matrix, and jump to step S1 until the robot agent training ends.
[0083] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0085] All actions involving the acquisition of signal information or data in this invention are carried out in compliance with the relevant data protection laws and policies of the country where the device is located, and with the authorization granted by the owner of the device.
[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for generating an intelligent agent's environment based on regret estimation, characterized in that, include: Step S1: Generate an environment map based on the tile pair regret matrix, wherein the tile pair regret matrix records the regret value of the tile transfer pair in the environment map when the robot agent moves from the first terrain tile to the second terrain tile of a different terrain type, and the larger the regret value, the greater the probability of the tile transfer pair being generated in the environment map. Step S2: Obtain the transition results of each tile transition pair statistically analyzed by the robot agent during the current training round based on the environment map; Step S3: Estimate the regret value of each tile transfer pair based on the transfer results. The higher the transfer failure rate, the larger the regret value, so as to obtain a temporary regret matrix. Step S4: Update the tile pair regret matrix based on the temporary regret matrix, and jump to step S1 until the robot agent training ends.
2. The agent environment generation method based on regret estimation according to claim 1, characterized in that, Environment maps are generated based on the regret matrix of tile pairs, including: Step S11: Obtain the generated current terrain tile; Step S12: Query the regret value of each tile transfer pair from the current terrain tile to other terrain tiles in the tile pair regret matrix; Step S13: Determine the sampling weight of each other topographic tile as the next topographic tile to be generated based on the regret value of each tile transfer pair. The larger the regret value, the larger the corresponding sampling weight. Step S14: Perform weighted probability sampling on each of the other terrain tiles based on the sampling weight to obtain the next generated terrain tile, and jump to step S11 until all tiles of the environmental map are generated.
3. The agent environment generation method based on regret estimation according to claim 2, characterized in that, The sampling weights of each other topographic tile as the next to be generated topographic tile are determined based on the regret value of each tile transfer pair, including calculating the sampling weights according to the following formula. : ； in, Indicates temperature parameter, Indicates the current topographic map tile. This indicates the next terrain tile to be generated. The tile pair in the regret matrix represents the... Transferred to The regret value, It represents a proportional relationship operation.
4. The agent environment generation method based on regret estimation according to claim 1, characterized in that, Based on the transfer results, the regret value of each tile transfer pair is estimated, and the higher the transfer failure rate, the greater the regret value, including: Based on the transfer results of each tile transfer pair during the current training round, the number of transfer failures and the total number of transfers for each tile transfer pair are obtained, and the transfer failure rate is calculated. The transfer failure rate is used as the regret value of the tile transfer pair.
5. The agent environment generation method based on regret estimation according to claim 1, characterized in that, Based on the transfer results, the regret value of each tile transfer pair is estimated, and the higher the transfer failure rate, the greater the regret value, including: Based on the transfer results of each tile transfer pair during the current training round, the number of transfer failures and the total number of transfers for each tile transfer pair are obtained, and the transfer failure rate and transfer success rate are calculated. The product of the transfer failure rate and the transfer success rate is used as the regret value of the tile transfer pair.
6. The agent-based environment generation method based on regret estimation according to any one of claims 1 to 5, characterized in that, Updating the tile pair regret matrix based on the temporary regret matrix includes: The temporary regret matrix is determined to be a new tile pair regret matrix, or a new tile pair regret matrix is obtained by weighted merging of the temporary regret matrix and the current tile pair regret matrix.
7. A device for generating an intelligent agent's environment based on regret estimation, characterized in that, include: An environment generation module is used to generate an environment map based on a tile pair regret matrix. The tile pair regret matrix records the regret value of a tile transfer pair in the environment map where the robot agent moves from a first terrain tile to a second terrain tile of a different terrain type. The larger the regret value, the greater the probability that the tile transfer pair will be generated in the environment map. The local transfer tracking module is used to obtain the transfer results of each tile transfer pair statistically analyzed by the robot agent during the current training round based on the environment map; The regret value estimation module is used to estimate the regret value of each tile transfer pair based on the transfer results. The higher the transfer failure rate, the larger the regret value, so as to obtain a temporary regret matrix. The matrix update module is used to update the tile pair regret matrix based on the temporary regret matrix, and then jump to the execution environment generation module until the robot agent training is completed.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the agent environment generation method based on regret estimation as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the agent environment generation method based on regret estimation as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the agent environment generation method based on regret estimation as described in any one of claims 1 to 6.