Reinforcement learning-based power marketing inspection anomaly priority ranking and optimization method and system

Abstract

The application discloses a power marketing inspection abnormal priority sequencing and optimization method and system based on reinforcement learning, and particularly relates to the technical field of power marketing inspection resource scheduling data processing, which is used to solve the problem that the existing offline training reinforcement learning sequencing strategy has a performance blind area of inconsistency between offline evaluation and online performance when being deployed, and causes the sequencing model to fail to safely connect when transitioning from offline training to online inspection closed loop. The application is implemented by performing differentiated masking on state vectors in a sample set of Markov decision process and multiple offline policy evaluations to obtain a value estimation value set, calculating a cognitive uncertainty measure, a coverage density and a substitutability measure, demarcating an offline policy blind area through Pareto non-dominated sequencing, performing a preset risk rule sequencing on state-action pairs falling into the offline policy blind area during online sequencing, and performing reinforcement learning sequencing on state-action pairs not falling into the offline policy blind area.

Description

Technical Field

[0001] This invention relates to the field of power marketing audit resource scheduling data processing technology, and more specifically, to a method and system for prioritizing and optimizing power marketing audit anomalies based on reinforcement learning. Background Technology

[0002] Currently, the electricity marketing audit field has gradually built an anomaly identification system centered on data-driven approaches. This system uses unsupervised learning algorithms to detect outliers in user electricity consumption behavior and business operation data, and calculates risk scores based on dimensions such as business impact and historical frequency to form an initial ranking of abnormal work orders. To further overcome the limitations of static weights and expert-experience-driven ranking, the industry has begun to model audit priority ranking as a Markov decision process. This utilizes reinforcement learning methods, with on-site verification feedback as a reward signal, to drive continuous optimization of the ranking strategy within the business loop, enabling the anomaly handling order to adapt to dynamic changes in the audit situation.

[0003] However, when the audit ranking problem is placed into the Markov decision framework, the agent must rely on historical logs generated by the previous fixed strategy for offline training because the actual business environment prohibits online exploratory decision-making. Under the conditions of sparse abnormal samples and lagging audit feedback, the evaluation methods of the strategy such as importance sampling used in offline training will suffer from variance inflation. The valuation of state-action pairs that have not been fully covered in the historical data is seriously inaccurate. This inconsistency between offline evaluation and online performance makes it impossible to truly perceive the risk of degradation in the actual environment before the strategy is deployed. As a result, there is a performance blind spot when the ranking model transitions from offline training to online audit closed loop, which hinders the safe and effective dynamic optimization of audit resource scheduling strategy. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, this invention provides a method and system for prioritizing and optimizing anomalies in electricity marketing audits based on reinforcement learning, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A reinforcement learning-based method for prioritizing and optimizing anomalies in electricity marketing audits includes: S1: Obtain historical logs of electricity marketing audits, extract risk characteristics and verification results of abnormal work orders, and construct a sample set of Markov decision processes containing state vectors, actions, and rewards; S2: Perform multiple random maskings on the state vectors of each sample in the sample set according to the dimension. Based on the masked sample set obtained after each masking, obtain the value estimate of each state-action pair through offline policy evaluation and form a value estimate set. S3: For each state-action pair in the sample set, calculate the cognitive uncertainty measure of the value estimate set, and calculate the Mahalanobis distance between the state-action pair and the remaining samples in the sample set. Take the reciprocal of the minimum Mahalanobis distance as the coverage density. S4: Calculate the change in value estimate and uncertainty of each state-action pair after the corresponding action is replaced by other optional actions through the set of value estimates, and obtain the substitutability measure of the state-action pair; S5: Using cognitive uncertainty measurement, coverage density and substitutability measurement as target dimensions, perform Pareto non-dominated ranking on all state-action pairs in the sample set, and define state-action pairs whose dominance level exceeds the preset reference level as offline policy blind spots. S6: During online sorting, for the state-action pair corresponding to the current audit decision, if it falls into the offline strategy blind zone, the preset risk rule sorting is executed; otherwise, the reinforcement learning strategy sorting derived from the value estimate set is executed.

[0006] Furthermore, S1 includes: obtaining user electricity consumption behavior data, business operation records, and historical verification results from the electricity marketing audit historical logs; extracting statistical features of user electricity consumption behavior data and business process features of business operation records, and concatenating them into a state vector; taking the audit action of abnormal work orders as the action, determining the reward benchmark value based on whether the historical verification results are violations, and discounting the reward benchmark value in combination with the verification feedback delay time to obtain the reward; combining the state vector, action, and reward into triples to form a sample set of the Markov decision process.

[0007] Further, S2 includes: setting a mask probability distribution corresponding to each dimension of the state vector; independently performing random masking on each dimension of the state vector of each sample in the sample set according to the mask probability distribution to generate a mask sample set; repeating this process multiple times to obtain multiple mask sample sets; on each mask sample set, using the state vector, action, and reward triple as input, calculating the value estimate for each state-action pair appearing in the sample set through offline policy evaluation; and aggregating the multiple value estimates obtained for the same state-action pair under multiple mask sample sets to form a value estimate set.

[0008] Furthermore, S3 includes: for each state-action pair in the sample set, calculating the variance of multiple value estimates in the value estimate set as a measure of cognitive uncertainty; calculating the covariance matrix of all state-action pairs in the sample set; for each state-action pair whose coverage density is to be calculated, calculating the Mahalanobis distance between it and the remaining state-action pairs in the sample set, taking the minimum value among all Mahalanobis distances, and taking the reciprocal of the minimum Mahalanobis distance as the coverage density.

[0009] Further, S4 includes: for each state-action pair in the sample set, extracting the mean of the value estimates under the corresponding action from the value estimate set as the benchmark value estimate; replacing the action of the current state-action pair with other optional actions in the action space of the sample set, obtaining the mean of the value estimates after the replacement action from the value estimate set, calculating the difference between the mean after the replacement action and the benchmark value estimate as the value estimate change; obtaining the variance of the value estimates after the replacement action from the value estimate set, calculating the difference between the variance after the replacement action and the variance before the replacement action as the uncertainty change; and combining the value estimate changes and uncertainty changes corresponding to all other optional actions to obtain the substitutability measure of the state-action pair.

[0010] Furthermore, by combining the changes in value estimate and uncertainty corresponding to all other optional actions, a measure of substitutability for the state-action pair is obtained, including: for each other optional action, calculating the product of the change in value estimate and the change in uncertainty, and taking the maximum value among the products corresponding to all other optional actions as the measure of substitutability for the state-action pair.

[0011] Furthermore, S5 includes: using cognitive uncertainty measure, coverage density, and substitutability measure as three target dimensions, performing pairwise comparisons on all state-action pairs in the sample set; if one state-action pair is not inferior to another state-action pair in all three target dimensions and is strictly dominant in at least one target dimension, then the former is determined to dominate the latter; classifying state-action pairs in the sample set that have never been dominated by any other state-action pair into the first Pareto front; removing the first Pareto front from the sample set and re-identifying a new first Pareto front as the second Pareto front; repeating this process to assign a dominance level to each state-action pair; and defining state-action pairs whose dominance level exceeds the preset reference level as offline policy blind spots.

[0012] Furthermore, the preset reference level is determined in the following way: all state-action pairs in the sample set are arranged in ascending order of dominance level, and the dominance level when the cumulative number percentage reaches the preset safety coverage threshold is used as the preset reference level.

[0013] Furthermore, S6 includes: during online ranking, obtaining the abnormal work order corresponding to the current audit decision, extracting the state vector of the abnormal work order, and determining the state-action pair corresponding to the current audit decision; determining whether the state-action pair corresponding to the current audit decision falls into the offline policy blind zone, that is, determining whether the cognitive uncertainty measure, coverage density, and substitutability measure of the state-action pair are dominated by any state-action pair Pareto within the offline policy blind zone in the three target dimensions; if it falls into the offline policy blind zone, then ranking according to the risk score of the abnormal work order from high to low according to the preset risk rules; if it does not fall into the offline policy blind zone, then calculating the mean of the value estimates in the value estimate set, and ranking according to the mean from high to low according to the reinforcement learning strategy.

[0014] On the other hand, the present invention provides a reinforcement learning-based system for prioritizing and optimizing anomalies in electricity marketing audits, including: The sample construction module is used to obtain historical logs of electricity marketing audits, extract risk characteristics and verification results of abnormal work orders, and construct a sample set of Markov decision processes containing state vectors, actions, and rewards. The value generation module is used to perform multiple random maskings on the state vectors of each sample in the sample set according to the dimension. Based on the masked sample set obtained after each masking, the value estimate of each state-action pair is obtained through offline policy evaluation and a value estimate set is formed. The measurement calculation module is used to calculate the cognitive uncertainty measure of the value estimate set for each state-action pair in the sample set, and to calculate the Mahalanobis distance between the state-action pair and the remaining samples in the sample set, taking the reciprocal of the minimum Mahalanobis distance as the coverage density. The substitutability module is used to calculate the change in value estimate and uncertainty of each state-action pair after the corresponding action is replaced by other alternative actions, based on the set of value estimates, and to obtain a measure of the substitutability of the state-action pair. The blind zone delineation module is used to perform Pareto non-dominated ranking on all state-action pairs in the sample set with cognitive uncertainty measure, coverage density and substitutability measure as target dimensions, and delineate state-action pairs with a dominance level exceeding the preset reference level as offline policy blind zones. The sorting execution module is used to sort the state-action pairs corresponding to the current audit decision online. If the pair falls into the offline strategy blind zone, the module will execute the sorting according to the preset risk rules. Otherwise, it will execute the sorting according to the reinforcement learning strategy derived from the set of value estimates.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The audit priority ranking is constructed as a Markov decision process. By performing differential masking and multiple evaluations on the state vector in the offline stage, a set of value estimates reflecting the degree of cognitive divergence is obtained. Based on this, cognitive uncertainty measure, coverage density and substitutability measure are calculated. The reliability of offline strategy evaluation in each region of the state-action space is characterized from three perspectives: valuation stability, sample support sufficiency and action selectivity. Pareto non-dominated ranking is used to jointly analyze the three measurement dimensions and delineate the offline strategy blind zone. This avoids the boundary distortion introduced by subjective weighting of multi-dimensional indicators or setting thresholds for each item, so that the blind zone boundary fits the true distribution of the sample set.

[0016] 2. During the online ranking phase, the ranking strategy is automatically switched based on whether the current audit decision's corresponding state-action pair falls into the offline strategy blind zone. Within the blind zone, risk-based ranking ensures the lower limit of audit quality, while outside the blind zone, reinforcement learning-based ranking leverages dynamic optimization capabilities. This allows audit resource allocation to fully utilize the adaptive advantages of reinforcement learning within the reliable area of ​​offline assessment, while avoiding the risk of ranking degradation caused by inaccurate valuations within the cognitive blind zone. Under the business constraints that prohibit online exploratory decisions, a secure transition from offline training to online deployment is achieved. Attached Figure Description

[0017] Figure 1 This is a flowchart of the reinforcement learning-based method for prioritizing and optimizing anomalies in electricity marketing audits, as described in this invention. Figure 2 This is a schematic diagram of the structure of the reinforcement learning-based power marketing audit anomaly priority ranking and optimization system of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1: Figure 1 This invention presents a reinforcement learning-based method for prioritizing and optimizing anomalies in electricity marketing audits, including: S1: Obtain historical logs of electricity marketing audits, extract risk characteristics and verification results of abnormal work orders, and construct a sample set of Markov decision processes containing state vectors, actions, and rewards; S2: Perform multiple random maskings on the state vectors of each sample in the sample set according to the dimension. Based on the masked sample set obtained after each masking, obtain the value estimate of each state-action pair through offline policy evaluation and form a value estimate set. S3: For each state-action pair in the sample set, calculate the cognitive uncertainty measure of the value estimate set, and calculate the Mahalanobis distance between the state-action pair and the remaining samples in the sample set. Take the reciprocal of the minimum Mahalanobis distance as the coverage density. S4: Calculate the change in value estimate and uncertainty of each state-action pair after the corresponding action is replaced by other optional actions through the set of value estimates, and obtain the substitutability measure of the state-action pair; S5: Using cognitive uncertainty measurement, coverage density and substitutability measurement as target dimensions, perform Pareto non-dominated ranking on all state-action pairs in the sample set, and define state-action pairs whose dominance level exceeds the preset reference level as offline policy blind spots. S6: During online sorting, for the state-action pair corresponding to the current audit decision, if it falls into the offline strategy blind zone, the preset risk rule sorting is executed; otherwise, the reinforcement learning strategy sorting derived from the value estimate set is executed.

[0020] In a specific implementation of S1, user electricity consumption behavior data, business operation records, and historical verification results are obtained from the electricity marketing audit historical log. The electricity marketing audit historical log is stored in the audit archive database of the marketing business application system and is extracted by time range and audit object type through the standard query interface of the data platform. User electricity consumption behavior data includes time-series data reflecting electricity consumption patterns, such as daily frozen positive active power readings at metering points, voltage and current acquisition curves, transformer area line loss rate, and power factor. Business operation records include log data reflecting the business processing process, such as business expansion application process node flow records, electricity price category change operation records, and metering device replacement records. Historical verification results include verification conclusions from on-site verification work orders, such as on-site confirmation of electricity theft, on-site confirmation of meter malfunction, and on-site confirmation of no abnormalities.

[0021] Statistical features of user electricity consumption behavior data and business process features of business operation records are extracted and concatenated into a state vector. The statistical features are extracted by using a fixed-duration sliding window to capture user electricity consumption behavior data. The window length is set according to the electricity consumption cycle characteristics of the audited object; for example, a 30-day window is set for photovoltaic power generation users, and a 7-day window is set for agricultural drainage users. The mean, standard deviation, month-on-month change rate, and year-on-year change rate of electricity consumption within the window are calculated as statistical features of electricity consumption. The average deviation of voltage from rated voltage and the maximum value of three-phase current imbalance within the window are calculated as statistical features of power quality. The median and interquartile range of power factor within the window are calculated as statistical features of power efficiency.

[0022] The method for extracting business process features is as follows: Process nodes in the business operation records are sorted by operation occurrence time to form a process node sequence. The mean, maximum, and minimum time intervals between adjacent nodes in the process node sequence are calculated, and the frequency of abnormal flow nodes in the process node sequence is statistically analyzed. Abnormal flow nodes refer to nodes whose process node order deviates from the standard process path. The standard process path is obtained from the process template configuration of the marketing business application system. For example, the standard process path for business expansion and installation applications is acceptance, survey, approval, construction, acceptance, and archiving. If a reverse operation occurs in the process node sequence, where archiving is performed before approval, then both the archiving node and the approval node are marked as abnormal flow nodes. The statistical values ​​of time intervals and the frequency of abnormal flow nodes are used as business process features. The above statistical features and business process features are concatenated end-to-end according to a preset feature order. The preset feature order is fixed in advance according to the feature type. For example, electricity consumption statistics, electricity quality statistics, electricity efficiency statistics, and business process features are arranged in sequence to form a multi-dimensional real number array as a state vector. The dimension of the state vector is equal to the sum of the dimensions of the statistical features and the dimensions of the business process features.

[0023] The audit action for abnormal work orders is used as the action. Audit actions include three types: dispatching work orders for on-site verification, temporarily storing them for further observation, and directly archiving them without processing. The reward benchmark value is determined based on whether the historical verification results indicate a violation. When the historical verification result confirms on-site electricity theft or illegal electricity use, the reward benchmark value is positive one. When the historical verification result confirms no abnormalities or normal metering error, the reward benchmark value is negative one. When the historical verification result confirms on-site meter malfunction not attributable to the user, the reward benchmark value is zero. The reward is obtained by discounting the reward benchmark value based on the verification feedback delay time. The discount method is that the reward equals the reward benchmark value multiplied by a discount factor, calculated as: γ=e (-λΔt) Where γ represents the discount factor, λ represents the attenuation coefficient, and Δt represents the verification feedback delay duration. The attenuation coefficient λ is set according to the timeliness requirements of the audit task. The setting method is as follows: determine the tolerance period for delayed feedback in the audit business. The tolerance period refers to the time elapsed after the reward benchmark value attenuates to approximately 37% of the initial value. The attenuation coefficient λ is taken as the reciprocal of the tolerance period. For example, if the tolerance period is set to 20 days, then λ is 0.05. The verification feedback delay duration Δt is in days. The verification feedback delay duration Δt is calculated from the time the audit action is executed until the time the verification conclusion is returned. The time the verification conclusion is returned is based on the timestamp in the marketing business application system where the verification work order status changes to "completed".

[0024] State vectors, actions, and rewards are combined into triples. Each historical audit log corresponds to a state vector, an action, and a reward, forming a triple. All triples are collected to form a sample set of the Markov decision process, which is stored in the form of a data table. Each record contains a state vector field, an action field, and a reward field.

[0025] In a specific implementation of S2, a mask probability distribution corresponding to each dimension of the state vector is set. Random masking is then performed independently on each dimension of the state vector of each sample in the sample set according to the mask probability distribution, generating a mask sample set. This process is repeated multiple times to obtain multiple mask sample sets. The mask probability distribution for each dimension of the state vector is determined based on the degree of variation of the feature values ​​of each dimension in the sample set. Specifically, the coefficient of variation of the feature values ​​of each dimension of the state vector in the sample set is calculated. The coefficient of variation is the ratio of the standard deviation of the feature values ​​to the mean of the feature values. Dimensions with coefficients of variation higher than a preset variation threshold are set to high mask probability, while dimensions with coefficients of variation not higher than the preset variation threshold are set to low mask probability. The preset variation threshold is determined based on the median of the coefficients of variation for all dimensions in the sample set. The determination process is as follows: the coefficient of variation for each dimension of the state vector in the sample set is calculated, and all coefficients of variation for all dimensions are arranged in ascending order. The coefficient of variation value in the middle position after arrangement is taken as the preset variation threshold. If the number of dimensions is even, the average of the two middle coefficients of variation is taken as the preset variation threshold. The high mask probability is set to 0.4 and the low mask probability is set to 0.1. The selection of high and low mask probabilities ensures that the high-variance dimension has a greater probability of being set to zero after masking, thus exposing the degree of influence of this dimension on the value estimation.

[0026] The random masking process is as follows: For each state vector in the sample set, iterate through each dimension of the state vector, generating a random number uniformly distributed between 0 and 1 based on the mask probability corresponding to that dimension. Compare the random number with the mask probability; if the random number is less than the mask probability, set the value of that dimension to zero; otherwise, keep the value of that dimension unchanged. After processing all dimensions, a mask sample is obtained. This process is repeated for each sample in the entire sample set to generate a mask sample set. The mask sample set generation process is repeated, with the number of repetitions determined by the total number of dimensions of the state vector. When the state vector dimension is d, the number of repetitions is d divided by 2 and rounded up. For example, when the state vector dimension is 12, the number of repetitions is 6, resulting in multiple mask sample sets. The reason for using differential mask probability driven by the coefficient of variation instead of uniform mask probability is that different feature dimensions contribute significantly to anomaly detection in the inspection scenario. Dimensions with higher feature value variation tend to carry more outlier discrimination information. Applying higher mask probability to highly variable dimensions can more effectively perturb the input basis of value estimation, so that the discrepancy between the value estimates trained on different mask sample sets fully reflects the cognitive uncertainty when key information is missing in the state space.

[0027] For each masked sample set, using state vectors, actions, and reward triples as input, offline policy evaluation is used to calculate the value estimate for each state-action pair appearing in the sample set. The offline policy evaluation employs the Batch Constrained Q Evaluation algorithm, also known as the Fitted Q-Evaluation algorithm. Its implementation in this scenario is as follows: the triples in the masked sample set are expanded into a state vector sequence, an action sequence, and a reward sequence according to the original order of appearance of the samples in the Markov decision process's sample set. The original order of appearance refers to the order in which the Markov decision process's sample set was constructed, arranged in ascending order according to the audit log timestamps. The number of iterations for value estimation is set to k, where k is the square root of the number of samples in the masked sample set and rounded up to balance computational overhead and convergence sufficiency. The value estimate for each state-action pair is initialized to zero. In each iteration, each state-action pair is evaluated according to B... The Ellman equation updates the value estimate using the formula Q(s,a) = r + γmax × maxa'Q(s',a'), where Q(s,a) represents the value estimate of taking action a under state vector s, r represents the reward, γmax represents the maximum discount factor (0.99), and maxa'Q(s',a') represents the maximum value estimate among all possible actions under the next state vector s' of the current state vector s. The next state vector s' is determined according to the order of the state vector sequence in the masked sample set, i.e., the next adjacent state vector of state vector s in the state vector sequence. After k iterations, the final value estimate of each state-action pair is obtained. The FittedQ-Evaluation process is executed on each masked sample set. Due to the different mask positions, the numerical distribution of the state vectors in each masked sample set changes, causing each FittedQ-Evaluation to converge to a different value estimate, thus generating multiple value estimates with cognitive discrepancies for the same state-action pair. Compared to methods that generate diverse estimates by randomly initializing network parameters, the method based on state mask perturbation of the input space directly affects the audit feature level. It can simulate the real impact on decision value judgment when different feature indicators are missing or collected abnormally in audit operations. This makes there a direct causal relationship between the uncertainty information carried by the value estimate set and the information completeness risk of audit operations.

[0028] Multiple value estimates for the same state-action pair obtained across multiple masked sample sets are aggregated to form a value estimate set. The aggregation method is as follows: Iterate through all state-action pairs appearing in the sample sets of the Markov decision process. A state-action pair refers to a combination of a state vector and an action taken under that state vector. An empty value estimate list is created for each state-action pair. The value estimates for that state-action pair are then extracted sequentially from the FittedQ-Evaluation results corresponding to each masked sample set and added to the value estimate list. After traversing all masked sample sets, the elements in the value estimate list constitute the value estimate set for that state-action pair. The number of value estimates in the value estimate set is equal to the number of times the masked sample set is repeated. The value estimate set provides the data foundation for subsequent calculations of cognitive uncertainty and substitutability measures. The dispersion of the value estimates within the set directly reflects the cognitive reliability of the offline policy evaluation on the corresponding state-action pair.

[0029] In a specific implementation of S3, for each state-action pair in the sample set, the variance of multiple value estimates in the value estimate set is calculated, and the calculated variance serves as a measure of cognitive uncertainty. The value estimate set is obtained by performing multiple random masking operations on the state vectors of each sample in the sample set along their dimensions, and evaluating the masked sample sets obtained after each masking operation using an offline strategy. The number of value estimates in the value estimate set is equal to the number of times the masked sample sets are repeated. The variance is calculated as follows: ; where σ 2 Variance represents a measure of cognitive uncertainty, N represents the number of value estimates in the set of value estimates, and Q represents the variance. i Let represent the i-th value estimate in the set of value estimates, and μ represent the mean of all value estimates in the set. A larger variance indicates a higher degree of dispersion in the value estimates of the same state-action pair under different masking conditions; that is, a more significant cognitive discrepancy in offline policy evaluation of this state-action pair, and stronger cognitive uncertainty. Compared to directly using the range or mean absolute deviation of value estimates, variance is more sensitive to outlier estimates and can more precisely capture extreme valuation deviations caused by a few masking patterns, thus providing a more robust uncertainty signal for subsequent blind spot identification.

[0030] Calculate the covariance matrix of all state-action pairs in the sample set. All state-action pairs in the sample set have been enumerated after the above masking and offline policy evaluation processing of the Markov decision process sample set. Each state-action pair corresponds to a feature vector composed of each dimension of the state vector and the action encoding. One-hot encoding is used for action encoding, mapping the action type to a vector of the same length as the action space, with 1s for the action-related positions and 0s for the remaining positions. The state vector and the one-hot encoded action vector are concatenated end-to-end to obtain the feature vector of the state-action pair. For the feature vectors of all state-action pairs in the sample set, the covariance between any two dimensions is calculated as follows: ;where Cov(X) j X k X represents the covariance between the j-th and k-th dimensions of the eigenvector. j and X k Let μ represent the value sequences of the j-th and k-th dimensions of the eigenvector, respectively. j and μ k Let represent the mean of the j-th dimension and the k-th dimension, respectively. The covariances between all pairs of dimensions form the covariance matrix. When the covariance matrix is ​​not invertible, it is regularized by adding a regularization constant to the diagonal elements of the covariance matrix. The regularization constant is one-thousandth of the mean of the diagonal elements of the covariance matrix to ensure that the inverse matrix exists and is numerically stable.

[0031] For each state-action pair whose coverage density needs to be calculated, the Mahalanobis distance between that state-action pair and all other state-action pairs in the sample set is calculated. The Mahalanobis distance between the feature vector of the current state-action pair and the feature vector of another state-action pair is calculated as follows: ; where D(v p , v q ) represents the Mahalanobis distance between two state-action pairs, v p and v q Let Σ represent the feature vectors of the current state-action pair and the feature vectors of the other state-action pair, respectively. -1Let represent the inverse of the covariance matrix after regularization, and T denote the vector transpose. The minimum Mahalanobis distance among all calculated values ​​is taken, and its reciprocal is used as the coverage density of the current state-action pair. The minimum Mahalanobis distance reflects the local sparsity of the state-action pair in the sample feature space; a larger minimum Mahalanobis distance indicates that the state-action pair is farther from any historical sample, resulting in lower coverage density. Compared to traditional frequency statistics, using the reciprocal of the Mahalanobis distance as a measure of coverage density considers the correlation between feature dimensions, avoiding the false high coverage illusion caused by repeated counting on strongly correlated dimensions. This more accurately reflects the degree to which a certain audit decision-making model has been fully validated in historical data, thus providing a more reliable sample support metric for delineating offline strategy blind spots.

[0032] In a specific implementation of S4, for each state-action pair in the sample set, the mean of the value estimates under the corresponding action is extracted from the value estimate set as the baseline value estimate. The value estimate set stores multiple value estimates for the same state-action pair obtained through offline policy evaluation under multiple masked sample sets. The corresponding action refers to the action that has been executed in that state-action pair. The mean is calculated by adding all value estimates in the value estimate set and then dividing by the number of value estimates. The baseline value estimate reflects the expected value of taking the current action in the current state.

[0033] Replace the action of the current state-action pair with other optional actions from the action space of the sample set. The action space of the sample set consists of all audit action types defined in step S1, including dispatching on-site verification work orders, temporarily storing for further observation, and directly archiving without processing. Other optional actions refer to the actions in the action space other than those already executed in the current state-action pair. Obtain the mean of the value estimate corresponding to the replacement action from the value estimate set. The method is as follows: while keeping the state vector unchanged, replace the action with other optional actions to form a new state-action pair, and extract the mean of the value estimate from the value estimate set corresponding to the new state-action pair. Calculate the difference between the mean of the replacement action and the baseline value estimate. The difference is taken as the value estimate change. A positive value estimate change indicates that the expected value increases after the replacement action, and a negative value estimate change indicates that the expected value decreases.

[0034] The variance of the value estimate corresponding to the replacement action is obtained from the set of value estimates. This is done by calculating the variance from the set of value estimates corresponding to the new state-action pair formed by the replacement action. The difference between the variance after the replacement action and the variance before replacement is calculated. The variance before replacement refers to the variance of the current state-action pair in the set of value estimates, which is the cognitive uncertainty measure calculated in step S3. The difference is taken as the uncertainty change. A positive uncertainty change indicates that the divergence in value estimation increases after the replacement action, while a negative uncertainty change indicates that the divergence decreases. The reason for examining both the value estimation change and the uncertainty change is that substitutability depends not only on whether the replacement action brings higher expected value, but also on the degree of cognitive consistency in value judgment after replacement. Only when the replacement action performs better in both the value and certainty dimensions does it indicate that there is a substitutable space for the decision of the current state-action pair.

[0035] The substitutability measure of a state-action pair is obtained by combining the changes in value estimate and uncertainty corresponding to all other alternative actions. The synthesis method is as follows: for each other alternative action, calculate the product of the change in value estimate and the change in uncertainty, and take the maximum value among all other alternative actions' corresponding products as the substitutability measure of the state-action pair. The product is calculated as: Sa = Va × Ua, where Sa represents the product corresponding to other alternative action a, Va represents the change in value estimate corresponding to other alternative action a, and Ua represents the change in uncertainty corresponding to other alternative action a. The reason for taking the maximum value as the substitutability measure is that the product simultaneously amplifies the combined effect of value enhancement and divergence reduction. The other alternative action with the largest product represents the most competitive alternative for the current state-action pair; the larger the product value, the easier it is for the decision of the current state-action pair to be replaced by other actions.

[0036] The substitutability measure, together with the cognitive uncertainty measure and coverage density calculated in step S3, constitutes a three-dimensional metric space. The substitutability measure supplements the dimension of offline policy blind spot judgment from the perspective of action options. When the substitutability measure of a state-action pair is high, even if the cognitive uncertainty is low and the coverage density is high, there may still be better alternative decisions. Being cautious about such areas during policy deployment helps to avoid getting trapped in local optima ranking strategies.

[0037] In a specific implementation of S5, Pareto non-dominated ranking is performed on all state-action pairs in the sample set using three target dimensions: cognitive uncertainty measure, coverage density, and substitutability measure. The cognitive uncertainty measure is obtained by calculating the variance from the set of value estimates in step S3; the coverage density is calculated in step S3 using the reciprocal of the minimum Mahalanobis distance; and the substitutability measure is calculated in step S4 by combining changes in value estimates and changes in uncertainty. All three target dimensions are indicators for evaluating the reliability of offline strategy assessments. A lower cognitive uncertainty measure indicates more reliable value estimates; a higher coverage density indicates more sufficient neighbor support for the state-action pair in the sample space; and a higher substitutability measure indicates that the current action is more easily replaced by other actions. Pareto non-dominated ranking directly compares the raw values ​​of the three target dimensions without requiring dimensional unification or normalization. The ranking result depends only on the comparison between the dimensions, avoiding the subjective bias introduced by weight settings in traditional weighted scoring methods.

[0038] The dominance determination rule in Pareto's non-dominated ranking is as follows: For all state-action pairs in the sample set, each pair is compared. If state-action pair A is not inferior to state-action pair B in all three objective dimensions and is strictly dominant in at least one objective dimension, then state-action pair A is considered to dominate state-action pair B. "Not inferior" means that the cognitive uncertainty metric of state-action pair A is not greater than that of state-action pair B, and simultaneously, the coverage density metric of state-action pair A is not less than that of state-action pair B, and the substitutability metric of state-action pair A is not less than that of state-action pair B. "Strictly dominant" means that at least one of the above three conditions is strictly inequal, i.e., the cognitive uncertainty metric is strictly less than, or the coverage density metric is strictly greater than, or the substitutability metric is strictly greater than. The dominance determination rule treats the cognitive uncertainty metric as a negative indicator and the coverage density and substitutability metrics as positive indicators for unified comparison, ensuring that the comparison direction of the three objective dimensions in the dominance relationship is consistent with the direction of their respective impact on the reliability of offline strategy evaluation.

[0039] The process of assigning dominance levels is as follows: state-action pairs in the sample set that have never been dominated by any other state-action pair are assigned to the first Pareto front, which corresponds to the first dominance level; all state-action pairs assigned to the first Pareto front are removed from the sample set, and state-action pairs that have never been dominated by any other remaining state-action pairs are re-identified from the remaining state-action pairs and assigned to the second Pareto front, which corresponds to the second dominance level; the process of removing state-action pairs assigned to the front and identifying new fronts from the remaining state-action pairs is repeated until all state-action pairs in the sample set are assigned to a dominance level, resulting in a dominance level number for each state-action pair. The smaller the dominance level number, the better the overall performance of the state-action pair in the three target dimensions.

[0040] The preset reference level is determined as follows: all state-action pairs in the sample set are arranged in ascending order of dominance level. Starting from the first dominance level, the number of state-action pairs is accumulated layer by layer. The cumulative number is calculated as a percentage of the total number of state-action pairs in the sample set. When the cumulative number percentage first reaches or exceeds the preset security coverage threshold, the corresponding dominance level is taken as the preset reference level. The preset security coverage threshold is set according to the requirements of the audit business for the coverage of the offline strategy blind spot. The setting method is as follows: the audit business experts determine the minimum proportion of state-action pairs that are expected to be sorted according to preset risk rules within the offline strategy blind spot. This proportion reflects the upper limit of the sorting decision risk that the audit business can tolerate due to the deviation of offline strategy evaluation. The preset security coverage threshold is equal to the decimal value of this proportion after percentage conversion. For example, if the audit business experts determine that the minimum proportion is 60%, then the preset security coverage threshold is 0.6. The preset reference level divides the dominance level into two regions: state-action pairs with a dominance level no higher than the preset reference level constitute the trusted region, and state-action pairs with a dominance level higher than the preset reference level constitute the offline strategy blind spot.

[0041] State-action pairs whose dominance level exceeds a preset reference level are designated as offline policy blind zones. A dominance level exceeding the preset reference level refers to a state-action pair whose dominance level number is greater than the dominance level number corresponding to the preset reference level. These state-action pairs exhibit poor overall performance across the three target dimensions, manifested as high cognitive uncertainty, insufficient coverage density, or high substitutability, leading to insufficient reliability in offline policy evaluation for their value estimation. Using Pareto non-dominated ranking to define offline policy blind zones, rather than setting hard thresholds for each target dimension, automatically adapts to the nonlinear relationships between the three target dimensions. The dependencies between target dimensions are implicitly preserved in the geometric structure of the Pareto front through the three-dimensional joint comparison in the dominance determination rule, avoiding boundary distortion caused by dimensional coupling. This makes the division of offline policy blind zones more closely resemble the true distribution of state-action pairs in the sample set.

[0042] In a specific implementation of S6, during online sorting, abnormal work orders corresponding to the current audit decision are obtained. These abnormal work orders come from the queue of pending work orders pushed in real time by the power marketing audit system. The state vector of the abnormal work order is extracted in the same way as when extracting the state vector in step S1 when constructing the sample set of the Markov decision process: statistical features and business process features are extracted from the user electricity consumption behavior data and business operation records associated with the abnormal work order. The statistical features include the average electricity consumption, standard deviation of electricity consumption, month-on-month change rate of electricity consumption, year-on-year change rate of electricity consumption, average deviation of voltage from rated voltage, maximum value of three-phase current imbalance, median and interquartile range of power factor, and the business process features include the average, maximum, and minimum time intervals between adjacent nodes in the process node sequence in the business operation record, as well as the frequency of occurrence of abnormal flow nodes. The statistical features and business process features are concatenated end to end according to a preset feature order to form a state vector. Determine the state-action pair corresponding to the current audit decision, and combine the extracted state vector with the action selected by the current audit decision for the abnormal work order. The action selected by the current audit decision is selected from the action space of the sample set. The action space of the sample set includes three action types: dispatching on-site verification work orders, temporarily storing for further observation, and directly archiving without processing. This yields the state-action pair corresponding to the current audit decision.

[0043] To determine whether the current audit decision's corresponding state-action pair falls into the offline policy blind zone, the criteria are whether the cognitive uncertainty measure, coverage density, and substitutability measure of the current state-action pair are dominated by any state-action pair within the offline policy blind zone in three target dimensions. The cognitive uncertainty measure is calculated online as follows: using the same mask probability distribution and offline policy evaluation method as in step S2, a set of value estimates is generated for the current state-action pair, and the variance of multiple value estimates in the set is calculated as the cognitive uncertainty measure. The coverage density is calculated online as follows: using the covariance matrix of all state-action pairs in the sample set calculated in step S3, the Mahalanobis distance between the feature vector of the current state-action pair and the feature vectors of all other state-action pairs in the sample set is calculated, and the reciprocal of the smallest Mahalanobis distance is taken as the coverage density. The substitutability measure is calculated online as follows: following the method in step S4, the product of the change in value estimate and the change in uncertainty after replacing the current state-action pair's action with other optional actions is calculated, and the maximum value among all other optional actions' corresponding products is taken as the substitutability measure. The Pareto dominance determination is performed according to the dominance determination rule in step S5: If there is at least one state-action pair within the offline policy blind zone whose cognitive uncertainty measure is not greater than the cognitive uncertainty measure of the current state-action pair, whose coverage density is not less than the coverage density of the current state-action pair, and whose substitutability measure is not less than the substitutability measure of the current state-action pair, and at least one of the three conditions is met, then it is determined that the state-action pair within the offline policy blind zone dominates the current state-action pair, and the current state-action pair falls into the offline policy blind zone. The offline policy blind zone is the set of state-action pairs defined in step S5 by Pareto non-dominance sorting and a preset reference hierarchy.

[0044] When the state-action pair corresponding to the current audit decision falls into the offline strategy blind zone, the abnormal work orders are sorted according to their risk scores from high to low using preset risk rules. The risk scores of abnormal work orders are obtained from the electricity marketing audit system. These scores are calculated based on an unsupervised learning algorithm that performs outlier detection on user electricity consumption behavior and business operation data, combined with business impact and historical frequency dimensions. The preset risk rule sorting is executed as follows: each abnormal work order corresponding to the state-action pair falling into the offline strategy blind zone is arranged in descending order of its risk score, forming a work order processing sequence. Auditors dispatch on-site verification work orders in the order of the work order processing sequence, prioritizing the abnormal work orders with the highest risk scores.

[0045] When the state-action pair corresponding to the current audit decision does not fall into the offline policy blind zone, the mean of the value estimates in the value estimate set is calculated, and the reinforcement learning policy is executed to sort them from high to low according to the mean. The value estimate set is the set of value estimates generated online when determining the attribution of the offline policy blind zone. The mean is calculated by adding all value estimates in the value estimate set and then dividing by the number of value estimates. The reinforcement learning policy sorting is executed as follows: the abnormal work orders corresponding to the state-action pairs that do not fall into the offline policy blind zone are arranged in descending order of the mean of the value estimates to form a work order processing sequence. Auditors dispatch on-site verification work orders in the order of the work order processing sequence, with the abnormal work orders with the highest mean being dispatched first.

[0046] During the online ranking process, for each state-action pair corresponding to the current audit decision, an offline strategy blind zone attribution judgment is first performed, and then a preset risk rule ranking or a reinforcement learning strategy ranking is selected based on the judgment result. The switching boundary between the two ranking strategies is determined by the geometric boundary of the offline strategy blind zone. The offline strategy blind zone remains unchanged after being defined in step S5 in the offline stage. In the online stage, only three metrics of the current state-action pair need to be calculated and Pareto dominance judgment is performed to complete the ranking strategy selection. The online computational complexity is only linearly related to the sample set size, which meets the requirements of the power marketing audit system for online ranking response time.

[0047] Example 2: Figure 2 A schematic diagram of the reinforcement learning-based power marketing audit anomaly priority ranking and optimization system of the present invention is provided. The reinforcement learning-based power marketing audit anomaly priority ranking and optimization system includes: The sample construction module is used to obtain historical logs of electricity marketing audits, extract risk characteristics and verification results of abnormal work orders, and construct a sample set of Markov decision processes containing state vectors, actions, and rewards. The value generation module is used to perform multiple random maskings on the state vectors of each sample in the sample set according to the dimension. Based on the masked sample set obtained after each masking, the value estimate of each state-action pair is obtained through offline policy evaluation and a value estimate set is formed. The measurement calculation module is used to calculate the cognitive uncertainty measure of the value estimate set for each state-action pair in the sample set, and to calculate the Mahalanobis distance between the state-action pair and the remaining samples in the sample set, taking the reciprocal of the minimum Mahalanobis distance as the coverage density. The substitutability module is used to calculate the change in value estimate and uncertainty of each state-action pair after the corresponding action is replaced by other alternative actions, based on the set of value estimates, and to obtain a measure of the substitutability of the state-action pair. The blind zone delineation module is used to perform Pareto non-dominated ranking on all state-action pairs in the sample set with cognitive uncertainty measure, coverage density and substitutability measure as target dimensions, and delineate state-action pairs with a dominance level exceeding the preset reference level as offline policy blind zones. The sorting execution module is used to sort the state-action pairs corresponding to the current audit decision online. If the pair falls into the offline strategy blind zone, the module will execute the sorting according to the preset risk rules. Otherwise, it will execute the sorting according to the reinforcement learning strategy derived from the set of value estimates.

[0048] All calculations involved in the embodiments are performed using dimensionless numerical values, and the preset parameters and thresholds in the calculations can be set by those skilled in the art according to actual conditions.

[0049] This technical solution can be flexibly deployed, for example, as embedded software running on device hardware, or installed on personal computers or other smart terminals with user interfaces, thus adapting to various hardware environments and usage requirements.

[0050] The above solutions can be implemented in software, hardware, firmware, or a combination thereof. When implemented in software, they are presented, in whole or in part, as a computer program product, containing one or more computer instructions or programs. When these instructions or programs are loaded and executed on a computer, results are produced corresponding to the processes or functions of the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one medium to another via wired or wireless means, such as from a website, server, or data center via wired means such as fiber optic cables, twisted-pair cables, or coaxial cables, or wireless means such as infrared or microwaves to another site. A computer-readable storage medium refers to any usable medium that a computer can access or a data storage device such as a server or data center that contains one or more usable media, including magnetic media such as floppy disks, hard disks, and magnetic tapes, optical media such as DVDs, and semiconductor media such as solid-state drives.

[0051] The specific working process of the system, device and module can be found in the method embodiment, and will not be repeated here.

[0052] The disclosed systems, devices, and methods can be implemented in other ways. The device embodiments are for illustrative purposes only, and the module division is only a logical division. In practice, different divisions can be implemented, such as merging or integrating multiple modules or components, or omitting some features. Coupling, direct coupling, or communication connections between the components can be achieved through interfaces, while indirect coupling or communication connections can take electrical, mechanical, or other forms.

[0053] The modules described as separate components may or may not be physically separated. The components shown as modules can be physical hardware or software, and can be deployed centrally or distributed across multiple network nodes. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0054] The functional modules in each embodiment can be integrated into one processing module, or they can exist independently, or two or more modules can be integrated into one.

[0055] If the functionality is implemented as a software module and used as an independent product, it can be stored in a computer-readable storage medium. Under this understanding, the substantial contribution of the technical solution of this application can be embodied in the form of a software product. This computer software product is stored in a storage medium and contains instructions to cause a computer device, such as a personal computer, server, or network device, to execute all or part of the steps of the methods in the embodiments of this application. The storage medium includes any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory, random access memory, magnetic disk, or optical disk.

[0056] The above are merely specific embodiments of this application, and the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should fall within the scope of protection of this application.

Claims

1. A reinforcement learning-based method for prioritizing and optimizing anomalies in electricity marketing audits, characterized in that... include: S1: Obtain historical logs of electricity marketing audits, extract risk characteristics and verification results of abnormal work orders, and construct a sample set of Markov decision processes containing state vectors, actions, and rewards; S2: Perform multiple random maskings on the state vectors of each sample in the sample set according to the dimension. Based on the masked sample set obtained after each masking, obtain the value estimate of each state-action pair through offline policy evaluation and form a value estimate set. S3: For each state-action pair in the sample set, calculate the cognitive uncertainty measure of the value estimate set, and calculate the Mahalanobis distance between the state-action pair and the remaining samples in the sample set. Take the reciprocal of the minimum Mahalanobis distance as the coverage density. S4: Calculate the change in value estimate and uncertainty of each state-action pair after the corresponding action is replaced by other optional actions through the set of value estimates, and obtain the substitutability measure of the state-action pair; S5: Using cognitive uncertainty measurement, coverage density and substitutability measurement as target dimensions, perform Pareto non-dominated ranking on all state-action pairs in the sample set, and define state-action pairs whose dominance level exceeds the preset reference level as offline policy blind spots. S6: During online sorting, for the state-action pair corresponding to the current audit decision, if it falls into the offline strategy blind zone, the preset risk rule sorting is executed; otherwise, the reinforcement learning strategy sorting derived from the value estimate set is executed.

2. The method for prioritizing and optimizing anomalies in electricity marketing audits based on reinforcement learning as described in claim 1, characterized in that, S1 includes: User electricity consumption behavior data, business operation records, and historical verification results are obtained from the historical logs of electricity marketing audits. Statistical features of user electricity consumption behavior data and business process features of business operation records are extracted and concatenated into a state vector. The audit action of abnormal work orders is used as the action. The reward benchmark value is determined based on whether the historical verification results are violations. The reward benchmark value is discounted in combination with the verification feedback delay time to obtain the reward. The state vector, action, and reward are combined into triples to form a sample set of Markov decision process.

3. The method for prioritizing and optimizing anomalies in electricity marketing audits based on reinforcement learning as described in claim 1, characterized in that, S2 includes: Define the mask probability distribution corresponding to each dimension of the state vector, and independently perform random masking on each dimension of the state vector of each sample in the sample set according to the mask probability distribution to generate a mask sample set. Repeat this process multiple times to obtain multiple mask sample sets. On each mask sample set, use the state vector, action, and reward triple as input, and calculate the value estimate for each state-action pair appearing in the sample set through offline policy evaluation. Combine the multiple value estimates obtained for the same state-action pair under multiple mask sample sets to form a value estimate set.

4. The method for prioritizing and optimizing anomalies in electricity marketing audits based on reinforcement learning as described in claim 1, characterized in that, S3 includes: for each state-action pair in the sample set, calculating the variance of multiple value estimates in the value estimate set as a measure of cognitive uncertainty; calculating the covariance matrix of all state-action pairs in the sample set; for each state-action pair whose coverage density is to be calculated, calculating the Mahalanobis distance between it and the remaining state-action pairs in the sample set, taking the minimum value among all Mahalanobis distances, and taking the reciprocal of the minimum Mahalanobis distance as the coverage density.

5. The method for prioritizing and optimizing anomalies in electricity marketing audits based on reinforcement learning as described in claim 1, characterized in that, S4 includes: for each state-action pair in the sample set, extracting the mean of the value estimates under the corresponding action from the value estimate set as the benchmark value estimate; replacing the action of the current state-action pair with other optional actions in the action space of the sample set, obtaining the mean of the value estimates after the replacement action from the value estimate set, calculating the difference between the mean after the replacement action and the benchmark value estimate as the value estimate change; obtaining the variance of the value estimates after the replacement action from the value estimate set, calculating the difference between the variance after the replacement action and the variance before the replacement action as the uncertainty change; and combining the value estimate changes and uncertainty changes corresponding to all other optional actions to obtain the substitutability measure of the state-action pair.

6. The method for prioritizing and optimizing anomalies in electricity marketing audits based on reinforcement learning as described in claim 5, characterized in that, By combining the changes in value estimate and uncertainty corresponding to all other alternative actions, a measure of substitutability for the state-action pair is obtained, including: for each other alternative action, calculating the product of the change in value estimate and the change in uncertainty, and taking the maximum value of the product corresponding to all other alternative actions as the measure of substitutability for the state-action pair.

7. The method for prioritizing and optimizing anomalies in electricity marketing audits based on reinforcement learning as described in claim 1, characterized in that, S5 includes: using cognitive uncertainty measure, coverage density, and substitutability measure as three target dimensions, performing pairwise comparisons on all state-action pairs in the sample set; if one state-action pair is not inferior to another state-action pair in all three target dimensions and is strictly dominant in at least one target dimension, then the former is determined to dominate the latter; classifying state-action pairs in the sample set that have never been dominated by any other state-action pair into the first Pareto front, removing the first Pareto front from the sample set, and re-identifying a new first Pareto front as the second Pareto front, repeating this process to assign a dominance level to each state-action pair; and defining state-action pairs whose dominance level exceeds the preset reference level as offline policy blind spots.

8. The method for prioritizing and optimizing anomalies in electricity marketing audits based on reinforcement learning as described in claim 7, characterized in that, The preset reference level is determined as follows: all state-action pairs in the sample set are arranged in ascending order of dominance level, and the dominance level at which the cumulative number percentage reaches the preset safety coverage threshold is used as the preset reference level.

9. The method for prioritizing and optimizing anomalies in electricity marketing audits based on reinforcement learning as described in claim 1, characterized in that, S6 include: During online sorting, obtain the abnormal work order corresponding to the current audit decision, extract the state vector of the abnormal work order, and determine the state-action pair corresponding to the current audit decision; determine whether the state-action pair corresponding to the current audit decision falls into the offline strategy blind zone, that is, determine whether the cognitive uncertainty measure, coverage density and substitutability measure of the state-action pair are dominated by any state-action pair Pareto in the offline strategy blind zone in the three target dimensions. If the abnormal work order falls into the offline policy blind zone, it is sorted according to the risk score from high to low by the preset risk rules; if it does not fall into the offline policy blind zone, the mean of the value estimates in the value estimate set is calculated, and the reinforcement learning policy is sorted according to the mean from high to low.

10. A reinforcement learning-based system for prioritizing and optimizing anomalies in electricity marketing audits, used to implement the reinforcement learning-based method for prioritizing and optimizing anomalies in electricity marketing audits as described in any one of claims 1-9, characterized in that, include: The sample construction module is used to obtain historical logs of electricity marketing audits, extract risk characteristics and verification results of abnormal work orders, and construct a sample set of Markov decision processes containing state vectors, actions, and rewards. The value generation module is used to perform multiple random maskings on the state vectors of each sample in the sample set according to the dimension. Based on the masked sample set obtained after each masking, the value estimate of each state-action pair is obtained through offline policy evaluation and a value estimate set is formed. The measurement calculation module is used to calculate the cognitive uncertainty measure of the value estimate set for each state-action pair in the sample set, and to calculate the Mahalanobis distance between the state-action pair and the remaining samples in the sample set, taking the reciprocal of the minimum Mahalanobis distance as the coverage density. The substitutability module is used to calculate the change in value estimate and uncertainty of each state-action pair after the corresponding action is replaced by other alternative actions, based on the set of value estimates, and to obtain a measure of the substitutability of the state-action pair. The blind zone delineation module is used to perform Pareto non-dominated ranking on all state-action pairs in the sample set with cognitive uncertainty measure, coverage density and substitutability measure as target dimensions, and delineate state-action pairs with a dominance level exceeding the preset reference level as offline policy blind zones. The sorting execution module is used to sort the state-action pairs corresponding to the current audit decision online. If the pair falls into the offline strategy blind zone, the module will execute the sorting according to the preset risk rules. Otherwise, it will execute the sorting according to the reinforcement learning strategy derived from the set of value estimates.

Images