An evaluation method and system for reverse engineering of reinforcement learning rewards

Abstract

The present invention belongs to the field of reinforcement learning security technology, and specifically relates to an evaluation method for reverse engineering of reinforcement learning rewards. The method comprises: calculating the standard mutual information between real reward clusters obtained in real time and reward clusters obtained by inverse reinforcement learning; generating a standard mutual information sequence based on a pre-set range of cluster numbers, a cluster sorting range, and a contamination trajectory ratio range; inputting the standard mutual information into a pre-established fourth-order tensor model to obtain an evaluation result; and evaluating the reward clusters obtained by inverse reinforcement learning based on the evaluation result.

Description

Technical Field

[0001] The present invention belongs to the field of reinforcement learning security technology, and more specifically, relates to an evaluation method and system for reverse engineering of reinforcement learning rewards. Background Art

[0002] Reinforcement learning, supervised learning, and unsupervised learning form a comprehensive machine learning framework within the field of artificial intelligence. Compared to the expert-supervised model in supervised learning, reinforcement learning relies on the autonomous interaction of an agent with its environment, receiving gradual reward feedback to learn. Due to this characteristic, reinforcement learning is considered a form of autonomous learning and has significantly driven the development of real-world AI applications, particularly in autonomous systems and gaming tasks, such as the highly successful Boston Dynamics robots and AlphaGo. In reinforcement learning, rewards are crucial hyperparameters; different reward settings can lead to significantly different performance results. Due to their commercial value, reinforcement learning rewards have become a target for reverse engineering of reinforcement learning algorithms.

[0003] Currently, two metrics are primarily used to evaluate reverse engineering of reinforcement learning rewards: mean reward loss (MRL) and expected value divergence (EVD). However, because MRL focuses on rewards across the entire state space, while EVD only considers a subset of sampled rewards, MRL and EVD can produce contradictory evaluation results, or even reduce evaluation accuracy. Summary of the Invention

[0004] To address the above-mentioned deficiencies in the prior art, the present invention proposes an evaluation method for reverse engineering of reinforcement learning rewards, which is applied to evaluate the performance of reverse engineering of reinforcement learning rewards. This method utilizes the normalized mutual information of reward clusters (C-NMI), where C-NMI aims to fill the intermediate granularity between the global state space and the specific sampled rewards. In combination with the Aggregate Nesting Algorithm (AGNES) to dynamically measure the similarity of reward clusters, a fourth-order tensor model is constructed to uniformly capture and store evaluation indicators of different dimensions: MRL, EVD, and C-NMI, to achieve a comprehensive evaluation of inverse reinforcement learning performance.

[0005] The present invention provides an evaluation method for reverse engineering of reinforcement learning rewards, the method comprising:

[0006] Calculate the standard mutual information between the real reward cluster obtained in real time and the reward cluster obtained by inverse reinforcement learning;

[0007] Generate a standard mutual information sequence based on the pre-set range of cluster number, cluster sorting range and contamination trajectory ratio range;

[0008] The standard mutual information is input into the pre-established fourth-order tensor model to obtain the evaluation results;

[0009] Based on the evaluation results, the reward clustering obtained by inverse reinforcement learning is evaluated.

[0010] As one of the improvements to the above technical solution, the standard mutual information between the real reward clusters obtained in real time and the reward clusters obtained by inverse reinforcement learning is calculated; the specific process is:

[0011] Clustering based on real rewards obtained in real time and reward clustering obtained by inverse reinforcement learning Calculate the mutual information between the two

[0012]

[0013] in,

[0014] in, for the σth category in ; for The εth category in ; S′ is the real state space;

[0015] According to the calculated mutual information between the two Calculate the standard mutual information between the two

[0016]

[0017] in,

[0018]

[0019] As one of the improvements to the above technical solution, the specific process of obtaining the real reward cluster is as follows:

[0020] Initialize the reward cluster:

[0021] C i ={C i；1 ,...,C i；|r|}={{r1},...,{r |r|}}

[0022] According to the given real reward set r e ={r1,r2,...,r |r|} and the pre-set number of clusters y={y1,y2,...,y |y|}, using the clustering nesting algorithm, the real reward set re ={r1,r2,...,r |r|} perform clustering;

[0023] Real reward collection r e ={r1,r2,...,r |r| Each element in} is considered as a category. In the i-th iteration, the number of clusters currently selected is y i , calculate the distance between any two categories in the real reward set

[0024]

[0025] Among them, d(r x ,r y ) are two adjacent real rewards and The Euclidean distance between C i The xth category in ; C i The y-th category in ;

[0026] Traverse C i All categories in the , and calculate the distance between any two categories, and sort the calculated multiple distances in order from small to large;

[0027] Select Category Distance The two smallest categories are merged to obtain the cluster subset New_C after this round of merging;

[0028]

[0029] New_C=C α ∪C β

[0030] Among them, C α , C β is the distance D obtained in this round al The two smallest categories; New_C is the cluster subset after this round of merging;

[0031] Add New_C to C i , obtain the merged reward set, then traverse all categories in the merged reward set, calculate the distance between any two categories, and sort the calculated multiple distances in ascending order;

[0032] Using the above merging process, merge the two categories with the smallest category distance to obtain the cluster subset after the round of merging, and add it to the merged reward set. Repeat the above process until the number of clusters in the merged real reward set reaches the currently selected number of clusters y i After that, the clustering is completed.

[0033] Get the true reward clustering

[0034] As one of the improvements to the above technical solution, the specific process of reward clustering obtained by inverse reinforcement learning is as follows:

[0035] Clustering of real rewards obtained in real time Sort in descending order according to the size of each element, and select the number of top clusters o in advance j , get the reward cluster set after selection Get the corresponding real state space

[0036] According to the pollution ratio m z ={m1,m2,...,m |m|}, use the inverse reinforcement learning algorithm to obtain the corresponding reward cluster set r u,z , clustering is performed in the state space S′;

[0037] Specifically, the reward cluster is initialized:

[0038]

[0039] The reward set obtained according to the given inverse reinforcement learning and the pre-set number of clusters y i , using the clustering nesting algorithm, for a given reward set obtained by inverse reinforcement learning Perform clustering;

[0040] Reward set obtained by inverse reinforcement learning Each element in is considered as a category. In the i-th iteration, the number of clusters currently selected is y i , calculate the distance between any two categories in the inverse reward set

[0041]

[0042] in, There are two rewards and The Euclidean distance between for The xth category in ; for The y-th category in ;

[0043] Traversal All categories in the dataset are included, and the Euclidean distance between any two categories is calculated, and the calculated distances are sorted in ascending order;

[0044] Select Category Distance The two smallest categories are merged.

[0045]

[0046] New_C u =C α u ∪C β u

[0047] Among them, C α u , C β u is the distance D obtained in this round al The two smallest categories; New_C u is the inverse clustering subset after this round of merging;

[0048] New_C u Add , obtain the merged inverse reward set, then traverse all categories in the merged inverse reward set, calculate the distance between any two categories, and sort the calculated multiple distances in ascending order;

[0049] Using the above merging process, merge the two categories with the smallest category distance to obtain the cluster subset after the merger, and add it to the merged inverse reward set. Repeat the above process until the number of clusters in the merged inverse reward set reaches the currently selected number of clusters y i After that, the clustering is completed.

[0050] Get the reward clustering obtained by inverse reinforcement learning

[0051] As one of the improvements to the above technical solution, the standard mutual information sequence is generated according to the pre-set range of cluster number, cluster sorting range and contamination trajectory ratio range; the specific process is as follows:

[0052] The number of clusters is pre-set to (y1,y |y| ), the cluster sorting range is (o1,o |o| ), the pollution trajectory ratio range is (m1,m |m| ),

[0053] When the number of clusters is y1, the cluster order is o1, the proportion of contaminated trajectories is m1, and the corresponding standard mutual information is obtained;

[0054] Traverse the range of cluster sorting and the range of pollution trajectory proportion to obtain the corresponding multiple standard mutual information;

[0055] Each time the number of clusters is changed, the above process is performed to obtain the corresponding multiple standard mutual information;

[0056] The above multiple standard mutual information are combined into a standard mutual information sequence.

[0057] As one of the improvements of the above technical solution, the fourth-order tensor model includes a fourth-order tensor layer, an extraction layer and an output layer connected in sequence;

[0058] The fourth-order tensor layer is used to set the required cluster number range, cluster sorting range, and pollution trajectory ratio range according to actual requirements, and to set the three evaluation indicators of MRL, EVD, and C-NMI required to be output by the output layer;

[0059] The fourth-order tensor layer includes a fourth-order tensor vector Where I = |y|, J = |o|, Z = |m|, K = |g|, the first-order tensor is the number of clusters I, the second-order tensor is the cluster ranking J, the third-order tensor is the proportion of contaminated trajectories Z, and the fourth-order tensor is the index g. The fourth-order tensor g uses three dimensions to represent the three evaluation indicators, expressed as (g1, g2, g3) = (MRL, EVD, C-NMI).

[0060] The extraction layer is used to extract the corresponding key indicator values ​​from the input standard mutual information sequence according to different evaluation indicators;

[0061] The extraction layer establishes the extraction matrix B according to the three evaluation indicators of MRL, EVD and C-NMI. (g) :

[0062]

[0063] in, represents a multidimensional sequence of MRLs, and represent the multidimensional sequences of EVD and C-NMI, respectively;

[0064]

[0065] Based on the interquartile range (IQR), key indicator values ​​of MRL, EVD, and C-NMI were extracted respectively;

[0066] Specifically, using As a key indicator value for MRL and EVD, while using As the key indicator value of C-NMI, and Respectively represent B i,j,z,g The upper and lower quartiles of , where

[0067] The output layer is used to output the key indicator values ​​corresponding to the extracted different evaluation indicators.

[0068] The present invention also provides an evaluation system for reverse engineering of reinforcement learning rewards, the system comprising:

[0069] The standard mutual information sequence acquisition module is used to calculate the standard mutual information between the real reward clusters obtained in real time and the reward clusters obtained by inverse reinforcement learning;

[0070] Generate a standard mutual information sequence based on the pre-set range of cluster number, cluster sorting range and contamination trajectory ratio range;

[0071] a data processing module, configured to input the standard mutual information into a pre-established fourth-order tensor model to obtain an evaluation result; and

[0072] The evaluation module is used to evaluate the reward clustering obtained by inverse reinforcement learning based on the evaluation results.

[0073] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method described above is implemented when the processor executes the computer program.

[0074] The present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is caused to perform the method.

[0075] The beneficial effects of the present invention compared with the prior art are:

[0076] 1. This method is the first to evaluate inverse reinforcement learning based on the canonical mutual information between the true reward clusters and the established clusters obtained by inverse reinforcement learning. This fills the intermediate granularity between the global state space and the specific sampled reward set, and can provide a more accurate and comprehensive inverse reinforcement learning evaluation.

[0077] 2. The method of the present invention adopts the Aggregate Nested Sense (AGNES) algorithm to dynamically control the C-NMI calculation and constructs a fourth-order tensor model to achieve a comprehensive evaluation of the inverse reinforcement learning performance, greatly enhancing the evaluation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 is a flow chart of an evaluation method for reverse engineering of reinforcement learning rewards according to the present invention;

[0079] Figure 2 It is a structural diagram of a fourth-order tensor model of an evaluation method for reverse engineering of reinforcement learning rewards according to the present invention. DETAILED DESCRIPTION

[0080] The present invention will now be further described with reference to the accompanying drawings.

[0081] like Figure 1 As shown, the present invention provides an evaluation method for reverse engineering of reinforcement learning rewards, the method comprising:

[0082] Step 1) Calculate the standard mutual information between the real reward cluster obtained in real time and the reward cluster obtained by inverse reinforcement learning;

[0083] Specifically, according to the real reward clustering obtained in real time and reward clustering obtained by inverse reinforcement learning Calculate the mutual information between the two

[0084]

[0085] in,

[0086] in, for the σth category in ; for The εth category in ; S′ is the real state space; different pollution ratios will result in different reward clusters obtained by your reinforcement learning

[0087] According to the calculated mutual information between the two Calculate the standard mutual information between the two

[0088]

[0089] in,

[0090]

[0091] The specific process of obtaining the real reward cluster is as follows:

[0092] Initialize the reward cluster:

[0093] C i={C i；1 ,...,C i；|r|}={{r1},...,{r |r|}}

[0094] According to the given real reward set r e ={r1,r2,...,r |r|} and the pre-set number of clusters y={y1,y2,...,y |y|}, using the clustering nesting algorithm, the real reward set r e ={r1,r2,...,r |r|} perform clustering;

[0095] Real reward collection r e ={r1,r2,...,r |r| Each element in} is considered as a category. In the i-th iteration, the number of clusters currently selected is y i , calculate the distance between any two categories in the real reward set

[0096]

[0097] Among them, d(r x ,r y ) are two adjacent real rewards and The Euclidean distance between C i The xth category in ; C i The y-th category in ;

[0098] Traverse C i All categories in the , and calculate the distance between any two categories, and sort the calculated multiple distances in order from small to large;

[0099] Select Category Distance The two smallest categories are merged to obtain the cluster subset New_C after this round of merging;

[0100]

[0101] New_C=C α ∪C β

[0102] Among them, C α , C β is the distance D obtained in this round al The two smallest categories; New_C is the cluster subset after this round of merging;

[0103] Add New_C to C i , obtain the merged reward set, then traverse all categories in the merged reward set, calculate the distance between any two categories, and sort the calculated multiple distances in ascending order;

[0104] Using the above merging process, merge the two categories with the smallest category distance to obtain the cluster subset after the round of merging, and add it to the merged reward set. Repeat the above process until the number of clusters in the merged real reward set reaches the currently selected number of clusters y i After that, the clustering is completed.

[0105] Get the true reward clustering Indicates the number of clusters is y i The reward subset includes the categories in the merged true reward set.

[0106] The specific process of reward clustering obtained by inverse reinforcement learning is as follows:

[0107] Clustering of real rewards obtained in real time Sort in descending order according to the size of each element, and select the number of top clusters o in advance j , get the reward cluster set after selection Get the corresponding real state space

[0108] According to the pollution ratio m z ={m1,m2,...,m |m|}, use the inverse reinforcement learning algorithm to obtain the corresponding reward cluster set r u,z , clustering is performed in the state space S′;

[0109] Specifically, the reward cluster is initialized:

[0110]

[0111] The reward set obtained according to the given inverse reinforcement learning and the pre-set number of clusters y i , using the clustering nesting algorithm, for a given reward set obtained by inverse reinforcement learning Perform clustering;

[0112] Reward set obtained by inverse reinforcement learning Each element in is considered as a category. In the i-th iteration, the number of clusters currently selected is y i, calculate the distance between any two categories in the inverse reward set

[0113]

[0114] in, There are two rewards and The Euclidean distance between for The xth category in ; for The y-th category in ;

[0115] Traversal All categories in the dataset are included, and the Euclidean distance between any two categories is calculated, and the calculated distances are sorted in ascending order;

[0116] Select Category Distance The two smallest categories are merged.

[0117]

[0118] New_C u =C α u ∪C β u

[0119] Among them, C α u , C β u is the distance D obtained in this round al The two smallest categories; New_C u is the inverse clustering subset after this round of merging;

[0120] New_C u Add , obtain the merged inverse reward set, then traverse all categories in the merged inverse reward set, calculate the distance between any two categories, and sort the calculated multiple distances in ascending order;

[0121] Using the above merging process, merge the two categories with the smallest category distance to obtain the cluster subset after the merger, and add it to the merged inverse reward set. Repeat the above process until the number of clusters in the merged inverse reward set reaches the currently selected number of clusters y i After that, the clustering is completed.

[0122] Get the reward clustering obtained by inverse reinforcement learning

[0123] Step 2) Generate a standard mutual information sequence based on the pre-set range of cluster number, cluster sorting range and contamination trajectory ratio range;

[0124] Specifically, the number of clusters is pre-set to (y1,y |y| ), the cluster sorting range is (o1,o |o| ), the pollution trajectory ratio range is (m1,m |m| ),

[0125] When the number of clusters is y1, the cluster order is o1, the proportion of contaminated trajectories is m1, and the corresponding standard mutual information is obtained;

[0126] Traverse the range of cluster sorting and the range of pollution trajectory proportion to obtain the corresponding multiple standard mutual information;

[0127] Each time the number of clusters is changed, the above process is performed to obtain the corresponding multiple standard mutual information;

[0128] The above multiple standard mutual information are combined into a standard mutual information sequence.

[0129] Step 3) inputting the standard mutual information sequence into a pre-established fourth-order tensor model to obtain an evaluation result;

[0130] Specifically, if Figure 2 As shown, the fourth-order tensor model includes a fourth-order tensor layer, an extraction layer, and an output layer connected in sequence;

[0131] The fourth-order tensor layer is used to set the required cluster number range, cluster sorting range, and pollution trajectory ratio range according to actual requirements, and to set the three evaluation indicators of MRL, EVD, and C-NMI required to be output by the output layer;

[0132] The fourth-order tensor layer includes a fourth-order tensor vector Where I = |y|, J = |o|, Z = |m|, K = |g|, the first-order tensor is the number of clusters I, the second-order tensor is the cluster ranking J, the third-order tensor is the proportion of contaminated trajectories Z, and the fourth-order tensor is the index g. The fourth-order tensor g uses three dimensions to represent the three evaluation indicators, expressed as (g1, g2, g3) = (MRL, EVD, C-NMI).

[0133] The extraction layer is used to extract the corresponding key indicator values ​​from the input standard mutual information sequence according to different evaluation indicators;

[0134] The extraction layer establishes the extraction matrix B according to the three evaluation indicators of MRL, EVD and C-NMI. (g) :

[0135]

[0136] in, represents a multidimensional sequence of MRLs, and represent the multidimensional sequences of EVD and C-NMI, respectively;

[0137]

[0138] Key indicator values ​​of MRL, EVD, and C-NMI were extracted based on the interquartile range (IQR);

[0139] Specifically, using As a key indicator value for MRL and EVD, while using As the key indicator value of C-NMI;

[0140] in, and Respectively represent B i,j,z,g The upper and lower quartiles of ;

[0141] in,

[0142] The output layer is used to output the key indicator values ​​corresponding to the extracted different evaluation indicators.

[0143] Step 4) Evaluate the reward clustering obtained by inverse reinforcement learning based on the evaluation results.

[0144] The present invention also provides an evaluation system for reverse engineering of reinforcement learning rewards, the system comprising:

[0145] The standard mutual information sequence acquisition module is used to calculate the standard mutual information between the real reward clusters obtained in real time and the reward clusters obtained by inverse reinforcement learning;

[0146] Generate a standard mutual information sequence based on the pre-set range of cluster number, cluster sorting range and contamination trajectory ratio range;

[0147] a data processing module, configured to input the standard mutual information into a pre-established fourth-order tensor model to obtain an evaluation result; and

[0148] The evaluation module is used to evaluate the reward clustering obtained by inverse reinforcement learning based on the evaluation results.

[0149] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method described above is implemented when the processor executes the computer program.

[0150] The present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is caused to perform the method.

[0151] Example 1.

[0152] The experimental environment required for the method of the present invention: the number of clusters y set in advance i The range is {2,3,...,10}, and the cluster sorting is o j The range is {2,3,...,10}, and the proportion of contaminated trajectories m i The range is {1,2,...,7},

[0153] Calculate the standard mutual information between the real reward cluster obtained in real time and the reward cluster obtained by inverse reinforcement learning;

[0154] Specifically, for the inverse reinforcement learning algorithm GPIRL, when the number of selected clusters is y i =8, cluster sorting o j =8, when the pollution trajectory ratio is 2,

[0155] Clustering using true rewards There are 8 clusters in , repeat the above clustering process 8 times, after completing the 8th clustering, sort the merged reward clusters, select the first 8 elements, construct the state space S ′, and the elements with a contamination trajectory ratio of 2 as the merged real reward clusters,

[0156] and reward clustering obtained by inverse reinforcement learning There are also 8 clusters in the state space S′. The above clustering process is repeated 8 times. After the 8th clustering is completed, the element with a contaminated trajectory ratio of 2 is selected as the merged inverse learning reward cluster.

[0157] Calculate the standard mutual information between the merged true reward cluster and the merged inverse learning reward cluster:

[0158] Generate a standard mutual information sequence based on the pre-set range of cluster number, cluster sorting range and contamination trajectory ratio range;

[0159] Specifically, the number of clusters is pre-set to be (2, 10), the cluster sorting range is pre-set to be (2, 10), and the pollution trajectory ratio range is pre-set to be (2, 7).

[0160] When the number of clusters is 2, the cluster ranking is 2, the contaminated trajectory ratio is 2, and the corresponding standard mutual information is obtained;

[0161] Traverse each element in the cluster sorting and each element in the pollution trajectory ratio, that is, the values ​​of the number of clusters, cluster sorting, and pollution trajectory ratio are 233, 234, 235, 236, 237, 242, 243, 244, 245, 246, 247...10107 respectively;

[0162] Each time a value is taken, a corresponding plurality of standard mutual information will be obtained; the plurality of standard mutual information are combined into a standard mutual information sequence.

[0163] The standard mutual information sequence is input into the pre-established fourth-order tensor model, and the key indicator value for C-NMI is output. The key indicator value is a statistical estimate of multiple standard mutual information. This is repeated 7 times. In the set DS containing 120 different samples, the average evaluation result for each inverse reinforcement learning algorithm is calculated. The calculation formula is as follows:

[0164]

[0165] Among them, C-NMI i Indicates that for sample DS i The calculated key C-NMI index values ​​are shown in Table 1 below;

[0166] The evaluation criteria for the evaluation indicators of MRL and EVD are that the closer to 0, the better the evaluation result;

[0167] The evaluation criteria of the C-NMI evaluation index is that the closer it is to 1, the better the evaluation result is;

[0168] Table 1

[0169]

[0170] According to the evaluation results obtained above, the average evaluation index obtained by this method using standard mutual information C-NMI is significantly better than the average evaluation index obtained by evaluating MRL and EVD using existing evaluation methods, which proves that the evaluation method proposed in this invention can evaluate the performance of various inverse reinforcement learning algorithms with high accuracy and complete the evaluation of reward clustering obtained by inverse reinforcement learning.

[0171] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and are intended to be encompassed by the claims of the present invention.

Claims

1. An evaluation method for reverse engineering of reinforcement learning rewards, comprising: Calculate the standard mutual information between the real reward cluster obtained in real time and the reward cluster obtained by inverse reinforcement learning; Generate a standard mutual information sequence based on the pre-set range of cluster number, cluster sorting range and contamination trajectory ratio range; The standard mutual information is input into the pre-established fourth-order tensor model to obtain the evaluation results; Based on the evaluation results, evaluate the reward clustering obtained by inverse reinforcement learning; The standard mutual information between the real reward clusters obtained in real time and the reward clusters obtained by inverse reinforcement learning is calculated; the specific process is: Clustering based on real rewards obtained in real time and reward clustering obtained by inverse reinforcement learning Calculate the mutual information between the two in, in, for the σth category in ; for The εth category in S ' is the real state space; According to the calculated mutual information between the two Calculate the standard mutual information between the two in, The specific process of reward clustering obtained by inverse reinforcement learning is as follows: Clustering of real rewards obtained in real time Sort in descending order according to the size of each element, and select the number of top clusters o in advance j , get the reward cluster set after selection Get the corresponding real state space According to the pollution ratio m z ={m1,m2,…,m |m| }, use the inverse reinforcement learning algorithm to obtain the corresponding reward cluster set r u ,z , clustering is performed in the state space S'; Specifically, the reward cluster is initialized: The reward set obtained according to the given inverse reinforcement learning and the pre-set number of clusters y i , using the clustering nesting algorithm, for a given reward set obtained by inverse reinforcement learning Perform clustering; Reward set obtained by inverse reinforcement learning Each element in is considered as a category. In the i-th iteration, the number of clusters currently selected is y i , calculate the distance between any two categories in the inverse reward set in, There are two rewards and The Euclidean distance between for The xth category in ; for The y-th category in ; Traversal All categories in the dataset are included, and the Euclidean distance between any two categories is calculated, and the calculated distances are sorted in ascending order; Select Category Distance The two smallest categories are merged. New_C u =C α u ∪C β u Among them, C α u , C β u is the distance D obtained in this round al The two smallest categories; New_C u is the inverse clustering subset after this round of merging; New_C u Add , obtain the merged inverse reward set, then traverse all categories in the merged inverse reward set, calculate the distance between any two categories, and sort the calculated multiple distances in ascending order; Using the above merging process, merge the two categories with the smallest category distance to obtain the cluster subset after the merger, and add it to the merged inverse reward set. Repeat the above process until the number of clusters in the merged inverse reward set reaches the currently selected number of clusters y i After that, the clustering is completed. Get the reward clustering obtained by inverse reinforcement learning 2. The evaluation method for reverse engineering of reinforcement learning rewards according to claim 1, characterized in that: The specific process of obtaining the real reward cluster is as follows: Initialize the reward cluster: According to the given real reward set r e ={r1,r2,…,r |r| } and the pre-set number of clusters y={y1,y2,…,y |y| }, using the clustering nesting algorithm, the real reward set r e ={r1,r2,…,r |r| } perform clustering; Real reward collection r e ={r1,r2,…,r |r| Each element in} is considered as a category. In the i-th iteration, the number of clusters currently selected is y i , calculate the distance between any two categories in the real reward set Among them, d(r x ,r y ) are two adjacent real rewards and The Euclidean distance between C i The xth category in ; C i The y-th category in ; Traverse C i All categories in the , and calculate the distance between any two categories, and sort the calculated multiple distances in order from small to large; Select Category Distance The two smallest categories are merged to obtain the cluster subset New_C after this round of merging; New_C=C α ∪C β Among them, C α , C β is the distance D obtained in this round al The two smallest categories; New_C is the cluster subset after this round of merging; Add New_C to C i , obtain the merged reward set, then traverse all categories in the merged reward set, calculate the distance between any two categories, and sort the calculated multiple distances in ascending order; Using the above merging process, merge the two categories with the smallest category distance to obtain the cluster subset after the round of merging, and add it to the merged reward set. Repeat the above process until the number of clusters in the merged real reward set reaches the currently selected number of clusters y i After that, the clustering is completed. Get the true reward clustering 3. The evaluation method for reverse engineering of reinforcement learning rewards according to claim 1, characterized in that: The standard mutual information sequence is generated according to the pre-set range of cluster number, cluster sorting range and contamination trajectory ratio range; the specific process is as follows: The number of clusters is pre-set to (y1,y |y| ), the cluster sorting range is (o1,o |o| ), the pollution trajectory ratio range is (m1,m |m| ), When the number of clusters is y1, the cluster order is o1, the proportion of contaminated trajectories is m1, and the corresponding standard mutual information is obtained; Traverse the range of cluster sorting and the range of pollution trajectory proportion to obtain the corresponding multiple standard mutual information; Each time the number of clusters is changed, the above process is performed to obtain the corresponding multiple standard mutual information; The above multiple standard mutual information are combined into a standard mutual information sequence.

4. The evaluation method for reverse engineering of reinforcement learning rewards according to claim 1, characterized in that: The fourth-order tensor model includes a fourth-order tensor layer, an extraction layer, and an output layer connected in sequence; The fourth-order tensor layer is used to set the required cluster number range, cluster sorting range, and pollution trajectory ratio range according to actual requirements, and to set the three evaluation indicators of MRL, EVD, and C-NMI required to be output by the output layer; The fourth-order tensor layer includes a fourth-order tensor vector Where I = |y|, J = |o|, Z = |m|, K = |g|, the first-order tensor is the number of clusters I, the second-order tensor is the cluster ranking J, the third-order tensor is the proportion of contaminated trajectories Z, and the fourth-order tensor is the index g. The fourth-order tensor g uses three dimensions to represent the three evaluation indicators, expressed as (g1, g2, g3) = (MRL, EVD, C-NMI). The extraction layer is used to extract the corresponding key indicator values ​​from the input standard mutual information sequence according to different evaluation indicators; The extraction layer establishes the extraction matrix B according to the three evaluation indicators of MRL, EVD and C-NMI. (g) : in, represents a multidimensional sequence of MRLs, and represent the multidimensional sequences of EVD and C-NMI, respectively; Based on the interquartile range (IQR), key indicator values ​​of MRL, EVD, and C-NMI were extracted respectively; Specifically, using As a key indicator value for MRL and EVD, while using As the key indicator value of C-NMI, and Respectively represent B i,j,z,g The upper and lower quartiles of , where The output layer is used to output the key indicator values ​​corresponding to the extracted different evaluation indicators.

5. An evaluation system for reverse engineering of reinforcement learning rewards, characterized in that: The system includes: The standard mutual information sequence acquisition module is used to calculate the standard mutual information between the real reward clusters obtained in real time and the reward clusters obtained by inverse reinforcement learning; Generate a standard mutual information sequence based on the pre-set range of cluster number, cluster sorting range and contamination trajectory ratio range; a data processing module, configured to input the standard mutual information into a pre-established fourth-order tensor model to obtain an evaluation result; and The evaluation module is used to evaluate the reward clustering obtained by inverse reinforcement learning based on the evaluation results; The standard mutual information between the real reward clusters obtained in real time and the reward clusters obtained by inverse reinforcement learning is calculated; the specific process is: Clustering based on real rewards obtained in real time and reward clustering obtained by inverse reinforcement learning Calculate the mutual information between the two in, in, for the σth category in ; for The εth category in S ' is the real state space; According to the calculated mutual information between the two Calculate the standard mutual information between the two in, The specific process of reward clustering obtained by inverse reinforcement learning is as follows: Clustering of real rewards obtained in real time Sort in descending order according to the size of each element, and select the number of top clusters o in advance j , get the reward cluster set after selection Get the corresponding real state space According to the pollution ratio m z ={m1,m2,…,m |m| }, use the inverse reinforcement learning algorithm to obtain the corresponding reward cluster set r u ,z , clustering is performed in the state space S'; Specifically, the reward cluster is initialized: The reward set obtained according to the given inverse reinforcement learning and the pre-set number of clusters y i , using the clustering nesting algorithm, for a given reward set obtained by inverse reinforcement learning Perform clustering; Reward set obtained by inverse reinforcement learning Each element in is considered as a category. In the i-th iteration, the number of clusters currently selected is y i , calculate the distance between any two categories in the inverse reward set in, There are two rewards and The Euclidean distance between for The xth category in ; for The y-th category in ; Traversal All categories in the dataset are included, and the Euclidean distance between any two categories is calculated, and the calculated distances are sorted in ascending order; Select Category Distance The two smallest categories are merged. Among them, C α u , C β u is the distance D obtained in this round al The two smallest categories; New_C u is the inverse clustering subset after this round of merging; New_C u Add , obtain the merged inverse reward set, then traverse all categories in the merged inverse reward set, calculate the distance between any two categories, and sort the calculated multiple distances in ascending order; Using the above merging process, merge the two categories with the smallest category distance to obtain the cluster subset after the merger, and add it to the merged inverse reward set. Repeat the above process until the number of clusters in the merged inverse reward set reaches the currently selected number of clusters y i After that, the clustering is completed. Get the reward clustering obtained by inverse reinforcement learning 6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 4.

Images