A mutual evaluation cheating identification method and device, an electronic device, and a medium

Abstract

The application discloses a mutual evaluation cheating identification method and device based on short loop blocking and timing score graph anomaly detection, electronic equipment and medium, the method comprises the following steps: constructing a candidate evaluation relationship graph before distribution, calculating a multi-dimensional risk weight, detecting a short loop and solving a low short loop risk evaluation distribution scheme by mixed integer optimization; after mutual evaluation, a directed weighted timing score graph is constructed, seven types of abnormal patterns such as high score closed loop, mutual benefit cluster and common suppression are detected after correction in a robust style, abnormal edges are determined by fusing strong rules and continuous evidence, and after continuous interpolation replacement, counterfactual ranking recalculation is performed, and risk score and structured evidence chain are output. The application blocks the formation of cheating closed loop from the source, can effectively identify multi-person collaborative relationship type cheating, quantifies the influence of score anomaly on ranking, and improves review efficiency.

Description

Technical Field

[0001] This invention relates to the field of enterprise office automation technology, and in particular to a method, device, electronic device and storage medium for identifying mutual evaluation cheating based on short loop blocking and time-series scoring graph anomaly detection. Background Technology

[0002] With the widespread adoption of office automation (OA) systems, peer review mechanisms in performance appraisals have become a crucial tool for internal competency assessment. In scenarios like 360-degree performance evaluations, superiors, peers, subordinates, and external collaborators provide multi-dimensional scores to the evaluated individual, resulting in a more comprehensive competency profile than a single evaluation source. However, the openness of peer review mechanisms also creates opportunities for cheating: reviewers may form mutually beneficial loops (giving each other high scores), collude (multiple individuals jointly raising or lowering scores for specific individuals), or engage in organized score manipulation through synchronized submissions and plagiarism. If these cheating behaviors are not effectively identified and addressed, they will severely distort performance rankings and damage the credibility of the evaluation system.

[0003] In the prior art, patent document CN115713270A discloses a method and apparatus for detecting and correcting anomaly scores in peer review. This method transforms the scoring data into a two-dimensional matrix, performs one-way anomaly detection, consistency detection, and two-way anomaly detection, and uses information entropy to correct the anomaly data. One-way detection includes neighborhood distance and density judgment based on KD-trees, while two-way detection includes correlation coefficient, difference consistency, dispersion rate, principal component analysis, and calculation of the degree of local anomalies. This scheme mainly focuses on anomaly detection and result correction after scoring is completed. It does not address structural risk control before the allocation of review tasks, nor does it directly model mutually beneficial clusters formed by multiple collaborators using directed short loops or community structures, and it does not quantify the probability of anomalies and the impact of anomaly scores on the final ranking.

[0004] Patent document CN107622461A discloses a method for peer review of subjective questions in MOOC systems. It establishes a weighted bipartite graph based on the assignment content, knowledge points, and reviewer attributes, assigns assignment weights to reviewers, and averages multiple scores for the same assignment. Another patent document, CN107146006A, calculates a grading ability value based on the deviation between the reviewer's score and the standard score, and uses this ability value for assignment allocation and score weighting. These solutions primarily address issues of task matching, allocation fairness, and scoring quality in peer review, but do not consider historical relationships between reviewers, reciprocal loops, or multi-person collusion for cheating as allocation constraints or anomaly detection targets.

[0005] Furthermore, schemes for identifying anomalous groups, such as CN118797465A, can utilize data on devices, locations, communications, and access behaviors to build identification models, and establish relationship graphs, divide connected subgraphs, and calculate graph density coefficients based on historical anomalous data; CN120235623A involves the identification of fraudulent groups based on knowledge graphs. However, these schemes are mainly aimed at scenarios involving financial fraud or debt evasion risks, and their entity types, relationship meanings, anomalous labels, and risk objectives differ significantly from the directed scoring relationships, review role coverage, scoring deviations, and ranking impacts in enterprise peer review.

[0006] At the academic research level, Jecmen et al. investigated whether reviewer-author collusion loops could be identified solely based on conference paper bidding data. Their results showed that colluders could manipulate paper allocation while evading commonly used detection algorithms, making reliable identification of collusion relationships difficult using only bidding data. More complex methods and additional metadata such as text similarity were needed. Zhao Mingming et al. implemented a sentinel assignment with pre-scored teacher scores, calculating reputation weights based on reviewers' performance on these assignments and using threshold truncation to filter maliciously high or low scores. However, this method primarily targeted individual reviewer reputation and scoring bias, failing to identify multi-person collaborative cheating using a relationship graph structure, and also neglecting to incorporate multimodal behavioral evidence and ranking impact assessment after scoring. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this invention provides a method, apparatus, electronic device, and storage medium for identifying mutual review cheating based on short-loop blocking and time-series scoring graph anomaly detection. This invention effectively solves the technical problems in existing OA mutual review systems, such as the lack of prevention against cheating relationship structures during the allocation stage, the blindness to multi-person collaborative relationship-based cheating in post-review detection, the coexistence of false positives and false negatives due to individual reviewer style differences, the inability to quantify whether abnormal scores truly affect ranking results, and the lack of verifiable evidence chains for risk conclusions. These problems include: failure to prevent cheating relationship structures during allocation in existing OA mutual review systems; failure to detect cheating in multi-person collaborative relationships after the event; the coexistence of false positives and false negatives due to individual reviewer style differences; the inability to quantify whether abnormal scores truly affect ranking results; and the lack of verifiable evidence chains for risk conclusions.

[0008] The technical solution adopted by this invention to solve its technical problem is:

[0009] In a first aspect, the present invention provides a method for identifying mutual review cheating based on short-loop blocking and time-series scoring graph anomaly detection, comprising: acquiring mutual review task data and organizational relationship data, and performing entity parsing and normalization processing to obtain a normalized entity dataset; constructing a candidate review relationship graph based on the normalized entity dataset, the candidate review relationship graph including multiple types of nodes and multiple types of edges; calculating risk weights for candidate review edges in the candidate review relationship graph to obtain risk weight values ​​for each candidate review edge; performing short-loop detection on the candidate review relationship graph based on the risk weight values ​​to identify short-loop structures containing a preset number of nodes; and minimizing the risk associated with the short-loop structure as the objective. The review allocation optimization solution is obtained by applying the results of the short loop detection to obtain the target review allocation scheme. After the mutual review is completed, the multimodal scoring data generated by the mutual review is collected. Based on the target review allocation scheme and the multimodal scoring data, a directed weighted time series scoring graph is constructed. The scoring data in the directed weighted time series scoring graph is subjected to style correction processing based on robust statistics to eliminate the style bias of the reviewers and the level bias of the reviewed objects, and the corrected scoring data is obtained. Based on the corrected scoring data, multiple preset abnormal patterns are detected hierarchically to obtain the detection results of each abnormal pattern. The detection results of each abnormal pattern are fused to determine the comprehensive abnormal score of each edge in the directed weighted time series scoring graph.

[0010] Optionally, the various types of nodes include employee nodes. Departmental nodes Project Nodes and historical mission nodes The various types of edges include reviewable edges. same department Joint project superior-subordinate side Approval chain edge Historical peer review Collaborating with history .

[0011] Optionally, the calculation of risk weights for candidate review edges includes:

[0012] For each reviewable edge Extract a six-dimensional feature vector, the dimensions of which include the historical mutual evaluation strength. Organizational closeness Project overlap Strength of superior-subordinate relationship Strength of the approval chain and historical reciprocity ;

[0013] Through convex combination functions The six-dimensional feature vectors are aggregated into reviewers. For the reviewed object The risk weight values ​​of the candidate review edges, where the aggregate weight satisfy and .

[0014] Optionally, the short cycle detection of the candidate review relationship graph includes: coarsely screening the candidate review relationship graph by exponentiation of the adjacency matrix to delineate the candidate node set, and then performing a depth-limited depth-first search on the candidate node set to accurately enumerate basic simple cycles to obtain a candidate cycle set; in the candidate cycle set, detecting binary mutual review cycles, ternary mutual review cycles, quaternary mutual review cycles, and historical reciprocal cycles constructed based on reciprocal relationships identified in historical mutual review task data.

[0015] Optionally, the evaluation allocation optimization solution adopts a mixed integer programming model, which includes:

[0016] Hard blocking constraint: for any short loop structure detected The number of review edges actually allocated in this short-loop structure is constrained to satisfy... ,in It is a short ring structure The set of candidate edges, Let the edge be represented by the binary decision variable. Whether or not it is selected for the review and allocation scheme This represents the number of nodes in the short ring structure.

[0017] Soft penalty mechanism: For any short loop structure detected Introducing a violation indicator variable And a penalty term is added to the objective function of the mixed integer programming model. ,in For the ring length and the penalty coefficient satisfying ;

[0018] Historical reciprocal cycle constraint: For short cycles containing historical fixed edges Set the current set of edges to be decided With historical fixed edge set Separate, constrained If the right side is negative, then all current candidate edges will be forcibly disconnected;

[0019] Load balancing: The load imbalance of reviewers is approximated by piecewise linearization by introducing auxiliary variables;

[0020] Reviewer shortage fallback: When the mixed-integer programming model has no feasible solution, a gap slack variable is introduced. satisfy And a pre-set large constant penalty weight is applied to the gap relaxation variable in the objective function. If there are still problems after solving... If all of these fail, the approach will be to recursively relax the professional matching threshold, switch from hard blocking to soft penalty, increase the reviewer's task limit, or expand the candidate reviewer pool. If all of these fail, the approach will be to manually assign the reviewers.

[0021] Optionally, the style correction processing based on robust statistics includes: using median absolute deviation for double correction: the first correction subtracts the reviewer's median score from the scoring data and divides it by the reviewer's median absolute deviation to eliminate reviewer style bias; the second correction subtracts the reviewee's median score from the data after the first correction and divides it by the reviewee's median absolute deviation to eliminate the reviewee's level bias.

[0022] When calculating the normalized denominator, the first level of correction uses the median absolute deviation of the reviewer's historical scores as the normalization benchmark, and the second level of correction uses the median absolute deviation of the first level of correction values ​​of the reviewed object as the normalization benchmark.

[0023] For reviewers whose historical scoring data volume is lower than the preset threshold, the statistical data is replaced step by step in the order of reviewer themselves, same position or department, and overall; for the reviewed object, the statistical data is replaced step by step in the order of same batch, same position or project, historical scores, and overall.

[0024] Optionally, the multiple exception modes include the following seven exception modes:

[0025] High-resolution closed-loop anomaly mode: The set of short-loop structures actually formed after review, allocation, and optimization. The set of short-loop structures that are selected and retained in the short-loop detection By merging, we obtain the actual set of short cycles. For each short loop in the actual short loop set, the mean value of the correction deviation on the loop is calculated as the loop score intensity. When the loop score intensity exceeds the preset average threshold and the correction deviation of each edge on the loop exceeds the preset minimum deviation threshold, the loop is marked as a high-scoring closed loop by AND joint determination.

[0026] Anomalous patterns of reciprocal clusters: The directed weighted time series scoring graph is symmetrically transformed into an undirected reciprocal weighted graph, and the Louvain community detection algorithm is used to identify reciprocal clusters;

[0027] Cluster risk assessment anomaly pattern: Based on the identified mutually beneficial clusters, the difference between the average deviation of the score inside the cluster and the average deviation of the score outside the cluster is calculated, and the mutual edge density inside the cluster and the cluster mutuality rate are combined for joint judgment to identify high-risk clusters;

[0028] Collusive suppression of abnormal patterns: Detects patterns where multiple reviewers give the same reviewer significantly lower scores than the reviewer's historical average, and calculates a suppression score. The suppression score is determined based on the deviation of the score and the number of reviewers involved in the suppression.

[0029] Synchronous scoring anomaly mode: The kernel density estimation method is used to detect the synchronicity of the scoring behavior of multiple reviewers in the time dimension. The set of reviewers whose maximum point-by-point ratio of the kernel density estimation of the group's submission time to the baseline kernel density estimation exceeds the preset synchronization threshold is judged as synchronous scoring anomaly.

[0030] Text similarity anomaly mode: SimHash local sensitive hashing, MinHash minimum hashing, and semantic text similarity calculation based on the pre-trained language model SBERT are used to calculate the text content of the review. The similarity obtained by the three methods is weighted and fused to obtain the comprehensive text similarity. When the average comprehensive text similarity within the group exceeds the preset similarity threshold, it is judged as a text similarity anomaly.

[0031] Long-term mild reciprocal anomaly pattern: using exponentially weighted moving distance The reciprocal rating behavior among reviewers in multiple peer review tasks, where the score difference is consistently below a preset severe threshold, is considered a long-term mild reciprocal abnormality when the index-weighted moving distance exceeds a preset mild threshold in a consecutive preset number of peer review tasks.

[0032] Optionally, the detection results of the various abnormal patterns are fused, including:

[0033] For each edge Calculate rule-based anomaly scores respectively Outlier scores based on continuous statistical models The maximum of the two values ​​is taken as the anomaly score for that edge.

[0034]

[0035] in , , , The three strong rules—high-score closed loop, reciprocal cluster, and joint suppression—have a single vote weight that is greater than the preset anomaly threshold. , , For the binary indicator function corresponding to strong rules; It is obtained by weighted summation of the detection results based on the continuous statistical model among the seven abnormal modes.

[0036] Optionally, the method further includes an anomaly edge processing step:

[0037] For edges in the directed weighted time series scoring graph where the abnormal comprehensive score exceeds a preset abnormal threshold, the scoring data of the abnormal edge is replaced with the scoring data of the normal edge through continuous interpolation. A hierarchical audit label is added to the replaced abnormal edge. The hierarchical audit label is marked as an approximate deletion adjustment, a replacement-based adjustment, or a weight reduction-based adjustment according to the interval where the abnormal comprehensive score is located.

[0038] In the subsequent score fusion calculation, a trust weight is introduced for each edge. The trust weight is linearly negatively correlated with the anomaly score of that edge, i.e. .

[0039] Optionally, the method further includes a counterfactual ranking distortion assessment, which includes the following indicators, each of which is normalized and then weighted and fused through a convex combination to obtain a comprehensive ranking distortion:

[0040] Kendall Distance: Calculate the proportion of inconsistent pairs in the ranking sequence of the reviewed object before and after removing the edge whose abnormal comprehensive score exceeds the preset abnormal threshold, relative to the total number of pairs;

[0041] Top-K Jaccard distance: Ranked before and after removing outlier edges. The Jaccard distance of a set of reviewed objects, wherein the Jaccard distance is... Subtract the Jaccard similarity between the two sets;

[0042] Boundary Crossing XOR Index: The number of reviewed objects that cross the preset ranking boundary before and after removing abnormal edges is counted and then normalized by dividing by the total number of reviewed objects.

[0043] Maximum ranking change: Record the maximum change in ranking of a single reviewed object before and after removing the abnormal edge, and normalize it by dividing it by the total number of reviewed objects;

[0044] Contribution rate of abnormal edges: Calculate the proportion of the average score adjustment of abnormal edges to the score range, and truncate it to the preset upper limit.

[0045] Optionally, the method further includes a comprehensive risk assessment:

[0046] Based on the detection results of each side under the multi-dimensional anomaly pattern, a comprehensive risk score is calculated. The normalized comprehensive risk score ranges from [value range missing]. ;

[0047] The comprehensive risk score is mapped to a five-level risk level, which is divided into low risk, low-medium risk, medium-high risk, high risk, and extremely high risk according to the risk score range.

[0048] For each review relationship that is judged to be abnormal and whose risk level reaches the preset level, a structured evidence chain in JSON format is generated.

[0049] Optionally, the method further includes a hierarchical feedback learning process, which includes:

[0050] Edge-level softening layer: Only weights consecutive evidence are fused. Edge anomalies are comprehensively analyzed using the softening indicator function and the softening maximum value operator. The problem is transformed into a differentiable approximation. The cross-entropy loss is calculated using manually verified edge-level labels as supervision signals on the edge-level pseudo-probabilities, and then updated using stochastic gradient descent. During deployment and inference, the softened operator is reverted to hard computation;

[0051] Calibration layer: The components of ranking distortion and mutual evaluation cheating risk are used as calculated feature vectors and input into the independent calibration layer. The task-level cross-entropy loss is calculated using manually verified task-level labels, and only the convex combination weights are updated. and scalar parameters The gradient is not backpropagated to the sorting, Top-K, XOR operation, and set counting stages;

[0052] Threshold adaptive layer: For each non-differentiable threshold parameter, based on near-differentiable threshold parameters... The exponential moving average of the optimal F1 value on the second-reviewed sample is used for adaptive updating, where This is the preset number of sliding window samples;

[0053] Offline hyperparameter optimization layer: Aggregates the risk weights from the discrete solution of mixed-integer programming into parameters. and regularization coefficient The Bandit algorithm, Bayesian optimization, or grid search are used for offline periodic tuning. The time decay coefficient and smoothing constant are fixed offline on the validation set before deployment.

[0054] Secondly, the present invention provides a mutual evaluation cheating identification device based on short-loop blocking and time-series scoring graph anomaly detection, comprising:

[0055] The data acquisition and normalization module is used to acquire mutual evaluation task data and organizational relationship data, and perform entity parsing and normalization processing to obtain a normalized entity dataset.

[0056] The candidate review relationship graph construction module is used to construct a candidate review relationship graph based on the normalized entity dataset. The candidate review relationship graph includes multiple types of nodes and multiple types of edges.

[0057] The risk weight calculation module is used to calculate the risk weight of the candidate review edges in the candidate review relationship graph to obtain the risk weight value of each candidate review edge.

[0058] The short loop detection module is used to perform short loop detection on the candidate review relationship graph based on the risk weight value, and identify short loop structures containing a preset number of nodes;

[0059] The review allocation optimization module is used to optimize the review allocation based on the results of the short loop detection, with the goal of minimizing the risk associated with the short loop structure, and to obtain the target review allocation scheme.

[0060] The scoring graph construction module is used to collect multimodal scoring data generated by mutual evaluation after the mutual evaluation is completed, and construct a directed weighted time series scoring graph based on the target evaluation allocation scheme and the multimodal scoring data.

[0061] The style correction module is used to perform style correction processing on the score data in the directed weighted time series score graph based on robust statistics, to eliminate the style bias of the reviewer and the level bias of the reviewed object, and to obtain the corrected score data.

[0062] The abnormal pattern hierarchical detection module is used to perform hierarchical detection on a variety of preset abnormal patterns based on the corrected scoring data, and obtain the detection results of each abnormal pattern.

[0063] An anomaly fusion module is used to fuse the detection results of the various anomaly modes to determine the comprehensive anomaly score of each edge in the directed weighted time series scoring graph.

[0064] Thirdly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements any of the methods described in the first aspect above.

[0065] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods described in the first aspect above.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] (1) Reduce the probability of forming reciprocal cheating relationship structures from the source. By constructing candidate review relationship graphs and detecting short-loop structures during the review task allocation stage, a review allocation scheme with low short-loop risk is solved using mixed integer optimization. During the allocation stage, binary, ternary, and quaternary mutual review loops and historical reciprocal loops are preventively blocked, preventing cheating relationship structures before the review task is issued. This fundamentally changes the technical approach of existing schemes that only perform post-event detection.

[0068] (2) Significantly improves the detection capability of multi-person collaborative relationship-based cheating patterns. By reconstructing actual short loops, the candidate short loops before allocation are bridged with the actual scoring short loops after mutual evaluation. Based on the correction of deviation, six types of cheating patterns are detected in layers: high-score closed loops, reciprocal clusters, joint suppression, abnormal synchronous submissions, and text templates. This makes up for the fundamental defect of existing single-point threshold schemes that are almost blind to relationship-based cheating.

[0069] (3) It can identify long-term mild reciprocity patterns that are difficult to detect by traditional methods. By using exponentially weighted moving average deviation (EWMD) to aggregate multi-period score sequences across periods, it can include hidden reciprocity behaviors that only slightly increase the score in each period, have a significant cumulative effect but extremely weak single-period signals in the detection range, thus solving the problem that the recall rate of existing single-point threshold and single-period statistical methods for this type of pattern is almost zero.

[0070] (4) Effectively distinguish between individual style differences of reviewers and style-embedded cheating. By using a dual robust style correction based on median absolute deviation (MAD), the median and dispersion of the reviewer's own historical scores are used to replace the global mean-variance, thereby eliminating the style bias of the reviewer and the level bias of the reviewed object. The two levels of correction use the reviewer's own MAD and the MAD of the reviewed object's z-score as normalized denominators, respectively, so that long-term lenient reviewers are no longer systematically falsely reported, while the hidden reciprocal score boosting by strict reviewers can also be effectively captured.

[0071] (5) Achieve quantitative assessment of the impact of abnormal scores on ranking results. By recalculating counterfactual rankings in conjunction with Kendall's τ distance, Top-K Jaccard distance, and boundary crossing XOR indicators, it is possible to clearly distinguish between two distinct business scenarios: "abnormal scores do not affect rankings" and "abnormal scores cross reward / promotion boundaries." This provides direct quantitative evidence for risk control decisions and makes up for the shortcomings of existing solutions in answering the key question of "whether abnormal scores have truly changed the results."

[0072] (6) Outputting a structured multidimensional evidence chain improves review efficiency and interpretability. By generating a JSON-formatted evidence chain that includes abnormal paths, suspected gang members, time synchronization evidence, text similarity evidence, ranking change comparison, and disposal suggestions, manual review is transformed from reviewing original scoring records one by one into targeted review based on structured evidence, which significantly reduces the time spent on reviewing a single case and improves the consistency of review conclusions.

[0073] (7) It has cross-scenario universality and scalability. This method is not bound to specific business scenarios—it can be applied to various OA peer review scenarios such as performance peer review, project peer review, promotion review, and training assessment, only requiring adjustment of the corresponding feature weights and threshold parameters. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0075] Figure 1 This is a flowchart of the mutual evaluation cheating identification method provided in the embodiments of the present invention;

[0076] Figure 2 This is a flowchart of the first stage of the present invention – pre-allocation short-loop blocking process;

[0077] Figure 3 This is a flowchart of the anomaly detection process for the time series scoring graph after mutual evaluation, provided in an embodiment of the present invention.

[0078] Figure 4 This is an abnormal fusion flowchart provided in an embodiment of the present invention;

[0079] Figure 5 This is a schematic diagram of the structure of the mutual evaluation cheating identification device provided in the embodiment of the present invention. Detailed Implementation

[0080] The prior art and the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0081] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed description. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0082] The technical problems to be solved by this invention include the following five aspects:

[0083] First, during the task allocation phase of the review, how can we reduce the probability of forming binary, ternary, and quaternary mutual review loops as well as historical reciprocal loops from the source, and identify and deal with potential cheating relationship structures before the review tasks are issued?

[0084] Second, after peer review is completed, how to identify multi-person collaborative relationship-based cheating patterns, including hidden forms such as mutually beneficial high-score closed loops, mutually beneficial clusters, joint suppression, abnormal synchronous submissions, templated evaluation texts, and long-term mild reciprocity.

[0085] Third, how to correct for individual style differences among reviewers in anomaly detection, distinguishing between the general preferences of lenient or strict reviewers and "style-embedded cheating", thereby reducing the false alarm rate while increasing the detection rate of cheating hidden in style.

[0086] Fourth, how to quantitatively determine whether suspected abnormal scores truly affected the final ranking, including changes in the Top-K list, crossing of reward or promotion boundaries, and the proportion of contribution to the total score gap, etc.

[0087] Fifth, how to output risk conclusions in the form of a verifiable multi-dimensional chain of evidence, including key information such as abnormal paths, suspected gang members, time synchronization evidence, text similarity evidence, ranking change comparison, and handling recommendations.

[0088] This invention breaks down the identification of mutual review cheating into two stages: pre-allocation short-loop blocking and post-allocation time-series scoring graph anomaly detection. Before allocation, it constructs a candidate reviewer relationship graph, calculates multi-dimensional risk weights, detects short-loop structures, and uses mixed-integer optimization to solve for a reviewer allocation scheme with low short-loop risk, reducing the probability of mutually beneficial closed loops from the source. After mutual review, it eliminates scoring style bias through robust style correction, detects seven types of anomaly patterns hierarchically, determines the set of anomaly edges using a combination of strong rule weighting and continuous evidence fusion, quantifies the actual impact of anomaly scores on ranking through counterfactual ranking, and finally outputs a structured evidence chain. The two stages are bridged by actual short-loop reconstruction—pre-allocation candidate short loops and post-allocation actual scoring short loops converge to form a closed loop.

[0089] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0090] A preferred embodiment of the present invention provides a method for identifying mutual evaluation cheating based on short loop blocking and time-series scoring graph anomaly detection. Figure 1 This is a flowchart illustrating the overall process of the peer review cheating detection method provided in this embodiment of the invention. The method described in this embodiment is divided into two phases: Phase 1 (pre-assignment prevention) and Phase 2 (post-peer review detection), specifically including the following steps:

[0091] Step S110: Obtain mutual evaluation task data and organizational relationship data, and perform entity parsing and normalization processing to obtain a normalized entity dataset.

[0092] Specifically, this step collects raw data from the following multi-source heterogeneous data systems:

[0093] The data for peer review tasks includes: task number, peer review period, number of reviewers N required for each reviewed object, maximum workload M for each reviewer, scoring dimensions, and scoring range.

[0094] The candidate reviewer data includes: employee ID, department, position, collection of projects participated in, number of reviews in history, and historical risk records.

[0095] The data of the reviewed individuals includes: employee ID, department, projects participated in, job level, reward threshold, and promotion threshold.

[0096] Organizational relationship data includes: relationships within the same department, relationships between superiors and subordinates, relationships related to joint projects, relationships in the approval chain, historical collaboration frequency, and historical mutual evaluation relationships.

[0097] Current peer review results data (collected only after peer review is completed): reviewer ID, reviewed user ID, rating value, sub-item rating, submission time, evaluation text, device ID, IP address, page dwell time, number of edits, copy and paste behavior;

[0098] Historical scoring profile data: historical average score, historical median score, historical median absolute deviation (MAD), percentage of highest score, percentage of lowest score, historical appeal rate, and historical review confirmation rate.

[0099] Since the above data usually comes from multiple independent subsystems such as HR system, performance system, project management system, and approval system, employees, departments, projects, and historical tasks often use different identifiers in different systems. This step first performs entity resolution and normalization to map the same entity in different systems to a unified identifier, giving a unified set of entities and a set of relationships.

[0100] Step S120: Based on the normalized entity dataset, construct a candidate review relationship graph, which includes multiple types of nodes and multiple types of edges.

[0101] Specifically, the normalized entity dataset obtained in step S110 is assembled into a multi-relation candidate review relation graph:

[0102]

[0103] Among them, the node set These correspond to four types of nodes: employees, departments, projects, and historical tasks (employee nodes). Departmental nodes Project Nodes and historical mission nodes ); edge set Includes seven types of edges: reviewable edges same department Joint project superior-subordinate side Approval chain edge Historical peer review Collaborating with history Each edge carries an edge type label and a normalized relation strength value.

[0104] Step S130: Calculate the risk weight of each candidate review edge in the candidate review relationship graph to obtain the risk weight value of each candidate review edge.

[0105] Preferably, a six-dimensional feature vector is extracted for each reviewable edge, wherein the dimensions of the six-dimensional feature vector include the historical mutual review strength. Organizational closeness Project overlap Strength of superior-subordinate relationship Strength of the approval chain and historical reciprocity The six-dimensional feature vectors are aggregated into risk weight values ​​for the candidate review edges of the reviewed object by the reviewer through a convex combination function.

[0106] Specifically, for each potential review edge (i.e., reviewers) The entity to be evaluated The risk weights for cheating are defined as a convex combination of six-dimensional features:

[0107]

[0108] in , The six characteristics cover historical peer review, organizational closeness, project overlap, hierarchy, approval chain, and historical reciprocity:

[0109] (1) Historical mutual evaluation intensity This reflects the frequency of reviewers' historical evaluations of the reviewed entity, using an exponential saturation function:

[0110]

[0111] in For the past Within a cycle Reviewed Number of times, Saturation coefficient (typically taken as) The number of historical peer reviews is This item is at that time As the number of times accumulates, it gradually tends towards saturation. .

[0112] (2) Organizational closeness This reflects the distance between two employees in the organizational hierarchy, using a reciprocal decay function:

[0113]

[0114] in To find the shortest distance on the organization tree, within the same department... Cross-level departments And so on.

[0115] (3) Overlap of common projects This reflects the degree of overlap between the reviewer's and the reviewed entity's project sets, using the Jaccard similarity score:

[0116]

[0117] When neither party is involved in any project, it is agreed to take This indicates there is no risk of project collaboration, thus avoiding [the risk of such risks]. Undefined.

[0118] (4) Strength of superior-subordinate relationship This reflects the strength of the relationship between two employees in the management chain:

[0119]

[0120] in To manage distances on the chain, For attenuation cutoff distance (typically taken) ),when hour ,make sure Furthermore, it continuously decays to zero at the distance from the boundary.

[0121] (5) Strength of the approval chain This reflects the frequency of association between two employees in the approval process. This embodiment provides two equivalent implementation methods (choose one when applying):

[0122] Method 1 (min-max normalization):

[0123]

[0124] in Within the statistics window and The number of approval associations between different departments. When the total number of approval associations in the statistics window is 0 (e.g., a newly established department, a sub-organization without an approval process), the following is agreed upon: To avoid 0 / 0 being undefined.

[0125] Method 2 (saturation function, avoiding the sensitivity of min-max to extreme values):

[0126]

[0127] in This is the saturation coefficient, typically taking a value of 0.3. This form is in... The value is 0 when the number of approvals is saturated and tends to 1 as the number of approvals increases, thus avoiding the problem in Method 1 where a single extreme approval user lowers the min-max normalization result of the other logarithms.

[0128] (6) Historical reciprocity This reflects whether there are records of mutual evaluations between two employees, with each giving the other high scores. A geometric mean with two-way gating is used.

[0129]

[0130] in For historical peer review Give Normalized mean ( ); For history The number of ratings; As an indicator function, it is only taken when there is at least one historical rating in both directions. —One-way absence means the whole is This fundamentally avoids miscalculating a unidirectional high score as "reciprocity." The geometric mean is more rigorous than the arithmetic mean: in any direction... The whole is Time decay term , Typical .

[0131] In conclusion, , Indicates no risk. This indicates an extremely high risk.

[0132] Step S140: Based on the risk weight value, perform short loop detection on the candidate review relationship graph to identify short loop structures containing a preset number of nodes.

[0133] Preferably, the candidate review relationship graph is coarsely screened by exponentiation of the adjacency matrix to delineate the candidate node set, and then a depth-limited depth-first search is performed on the candidate node set to accurately enumerate basic simple cycles, thus obtaining the candidate cycle set.

[0134] Specifically, short loop detection employs a two-stage method combining depth-first search (DFS) and adjacency matrix exponentiation.

[0135] Phase 1 – Coarse screening using matrix exponentiation. Constructing a directed adjacency matrix based on potential review edges. ,in If and only if there exists a candidate review edge .calculate (typical ),Will As a coarse screening indicator, the set of candidate nodes is narrowed down:

[0136]

[0137] It needs to be made clear that, The statistics are from Departure process Step back The total number of paths (including non-simple paths) cannot be used to directly determine the total number of paths. At a certain length On a simple ring, this stage only delineates candidate nodes.

[0138] The second stage – DFS exact enumeration. (For...) Each node in the process performs a depth-limited DFS (depth reaches 100%). (Truncation upon return), pseudocode is as follows:

[0139]

[0140] The current path is connected end to end and has depth The time-recording ring structure is used, and canonical encoding is performed to remove duplicates based on the smallest lexicographical rotation. The output length does not exceed [a certain value]. The set of directed rings Each It is an ordered sequence of nodes.

[0141] In the candidate ring set, binary mutual evaluation rings were detected respectively. and Simultaneous existence), ternary mutual evaluation cycle ( Furthermore, the quaternary mutual evaluation loop incorporates existing reverse evaluation edges from the historical cycle into the adjacency matrix. --Right now In history Reviewed —By re-executing the two-stage method described above, historical reciprocal cycles spanning multiple cycles can be identified. During detection, the current set of edges to be decided is distinguished. With historical fixed edge set This provides input for applying precise linearization constraints to the historical reciprocal loop in step S150.

[0142] In an alternative implementation, short loop detection can also be replaced by the Johnson algorithm instead of depth-constrained DFS. The Johnson algorithm (Johnson DB, “Finding all the elementary circuits of a directedgraph”, SIAM Journal on Computing, 1975) can... Enumerate all basic cycles in time, where Let be the number of basic cycles. For sparse graphs, Johnson's algorithm is more efficient than depth-constrained DFS and can directly replace the DFS enumeration steps described above.

[0143] Step S150: With the goal of minimizing the risk associated with the short loop structure, the review allocation optimization solution is performed based on the results of the short loop detection to obtain the target review allocation scheme.

[0144] After detecting candidate short cycles, the core problem becomes: how to select a review allocation scheme that minimizes the selection of high-risk candidate edges and disrupts the short cycle structure, while satisfying basic constraints such as requiring a sufficient number of reviewers for each reviewed object and preventing each reviewer from being overloaded. This is a typical constrained combinatorial optimization problem.

[0145] Figure 2 This invention provides a flowchart for Phase One – Short-Loop Blocking Before Allocation. This step constructs a mixed-integer optimization model, defining the decision variables... Indicate whether to include the reviewer Assigning evaluations to the review subjects The optimization model in this step is based on... The specific writing methods differ, falling into two forms: Integer Linear Programming (ILP, Scheme A) or Mixed Integer Quadratic Programming (MIQP, Scheme B), collectively referred to as Mixed Integer Optimization Model (MIP).

[0146] The mixed integer programming model includes:

[0147] Hard blocking constraint: for any short loop structure detected The number of review edges actually allocated in this short-loop structure is constrained to satisfy... ,in It is a short ring structure The set of candidate edges, Let the binary decision variables represent the edges Whether or not it is selected for the review and allocation scheme This represents the number of nodes in the short ring structure.

[0148] Soft penalty mechanism: For any short loop structure detected Introducing a violation indicator variable And a penalty term is added to the objective function of the mixed integer programming model. ,in For the ring length and the penalty coefficient satisfying ;

[0149] Historical reciprocal cycle constraint: For short cycles containing historical fixed edges Set the current set of edges to be decided With historical fixed edge set Separate, constrained If the right side is negative, then all current candidate edges will be forcibly disconnected;

[0150] Load balancing: The load imbalance of reviewers is approximated by piecewise linearization by introducing auxiliary variables;

[0151] Reviewer shortage fallback: When the mixed-integer programming model has no feasible solution, a gap slack variable is introduced. satisfy And a pre-set large constant penalty weight is applied to the gap relaxation variable in the objective function. If there are still problems after solving... If all of these fail, the approach will be to recursively relax the professional matching threshold, switch from hard blocking to soft penalty, increase the reviewer's task limit, or expand the candidate reviewer pool. If all of these fail, the approach will be to manually assign the reviewers.

[0152] The following is a more detailed description of this step:

[0153] The objective function balances three considerations simultaneously: minimizing the total risk cost of edges in the allocation scheme, maximizing the professional match between reviewers and reviewed entities, and maintaining a balanced workload among reviewers.

[0154]

[0155] in The candidate review edge risk weights calculated in step S130 As reviewer right Professional matching degree (calculated based on cosine similarity of job sequence, technology stack, and past review experience vectors). As a weighting factor, This is a penalty for unbalanced task load.

[0156] Short-loop constraints are the core of this invention, distinguishing it from existing allocation methods. Short-loop constraints have two implementation methods: "hard blocking" and "soft penalty," which are protected simultaneously as parallel embodiments in this invention.

[0157] Hard blocking constraint (Implementation Method 1): For each length of Potential short loop (The set of directed edges on a node sequence is denoted as ) The number of review edges actually allocated in this short-loop structure is constrained to satisfy:

[0158]

[0159] That is, at most on the ring The allocation of edges necessarily preserves at least one break, structurally preventing the formation of a complete mutual evaluation loop. This implementation does not introduce indicator variables. The global feasible solution space is tighter, but when the candidate review pool is small and the professional matching threshold is high, it may lead to no feasible solution, and it needs to be used in conjunction with a fallback mechanism.

[0160] Soft penalty mechanism (Implementation Method 2): For each potential short loop Introducing violation indicator variables The constraints are relaxed to:

[0161]

[0162] Substituting into the specific form: the constraint of the binary ring is ( ); ternary link sequence The constraints are The same logic applies to quaternions. A penalty term is added to the objective function. Since binary rings are more likely to form reciprocal relationships than ternary rings, and ternary rings have more concentrated risk than quaternary rings, the penalty coefficient satisfies... Hard blocking corresponds to Special cases; soft penalties are suitable for scenarios where the pool of reviewers is limited and strong constraints easily lead to infeasible solutions, through a trade-off. and Feasibility of softening solution.

[0163] Precise linearization of historical reciprocal cycles is another key design aspect—it's crucial to distinguish between currently decision-making edges (whose allocation is controllable) and historically fixed edges (which have already occurred and are considered constant). This involves considering the relationships between cycles containing historically occurring edges. Let the set of edges to be decided be denoted as . The historical fixed edge set is Historical edge is regarded as constant Then the constraint is:

[0164]

[0165] If the right side This indicates that the historical fixed edges have independently formed a complete risk structure. At this point, all current candidate edges should be forcibly disconnected, or the process should be switched to manual allocation. The final short-loop penalty term in Implementation Method Two is... .

[0166] Common constraints include: each reviewed object must... Individual reviewers ( The maximum number of tasks per reviewer is [number missing]. ( ); cannot self-evaluate ( ); Professional matching degree is below the threshold Edge pre-filtering ( when ); The decision variable is binary ( ).

[0167] Load balancing is achieved by introducing auxiliary variables. Achieve piecewise linear approximation:

[0168]

[0169] but Equivalent to The entire model remains linear and can be solved directly by the standard ILP solver. As an alternative embodiment, Alternatively, a double-relaxation MIQP approach can be used. Option A is more sensitive to extreme load differences but has higher computational complexity. Both options are within the scope of this invention, and the deployer can freely choose based on the solver's capabilities and load balancing sensitivity requirements.

[0170] The solver employs the branch-and-bound method or commercial ILP solvers (CPLEX, Gurobi). For large-scale problems (with more than [number of employees]), [further details are needed]. The solution can be obtained by first relaxing the short-cycle constraints to find the initial solution, and then iteratively repairing the violated short-cycle constraints.

[0171] In an alternative implementation, when the number of employees exceeds 50,000, the allocation optimization can be replaced by a greedy algorithm with simulated annealing heuristic solver instead of ILP. The specific steps are as follows: (1) Greedy initialization: according to (1) Greedy allocation in ascending order; (2) Simulated annealing local search: Randomly swap two allocation schemes and decide whether to accept them according to the Metropolis criterion; (3) Short cycle repair: After each iteration, detect allocations that violate short cycle constraints and forcibly repair them. This scheme can handle 100,000-level review tasks in 1 minute. In addition, the Gurobi solver takes about 5-30 seconds to solve a single problem when the number of employees does not exceed 10,000, and the optimal parameters are selected. (Accept a 1% optimality gap to speed up the solution) Second, For ultra-large-scale problems, the Lagrange relaxation addition generation strategy is enabled.

[0172] The output includes: review allocation plan The set of blocked short loops, the total risk score, the task load balancing degree, and the professional matching degree.

[0173] A gap-filling mechanism for the number of reviewers ensures that the algorithm can still provide interpretable outputs under boundary conditions, rather than interrupting the process. When a mixed-integer programming model becomes infeasible due to reasons such as an excessively high matching threshold, insufficient task capacity, or overly strong hard-loop constraints, a gap slack variable is introduced. Indicates the object being reviewed The shortfall in the number of reviewers has not been met:

[0174]

[0175] And in the objective function Apply a pre-set large constant penalty weight (The value is much greater than) If there are still [problems] after solving... The following priorities will be applied recursively to relax restrictions:

[0176] (1) Lower the professional matching threshold ( );

[0177] (2) Switch from hard blocking to soft punishment;

[0178] (3) Improve (Reviewer task limit) or expand the pool of candidate reviewers;

[0179] (4) If all of the above fail, the assignment will be transferred to manual assignment and recorded in assignment_fallback_log.

[0180] Through steps S110 to S150 above, the short-loop blocking prevention before allocation in Phase 1 is completed—the mutual review task allocation is abstracted as a risk minimization constraint optimization problem on the candidate review relationship graph. Cheating relationship structures are detected and blocked before the review task is issued, outputting a review allocation scheme with low short-loop risk. The reason for intervening during the allocation phase rather than only detecting after peer review is that once a cheating relationship is formed and a scoring result is generated, subsequent corrections cannot completely eliminate its distorting effect on the final ranking and reward decisions—prevention is better than cure. After the review allocation scheme is implemented, the second phase, post-peer review multimodal anomaly detection, is initiated. The following steps S210 to S340 describe the complete process of the second phase of this embodiment.

[0181] Step S210: After the mutual evaluation is completed, collect the multimodal scoring data generated by the mutual evaluation, and construct a directed weighted time series scoring graph based on the target evaluation allocation scheme and the multimodal scoring data.

[0182] After the peer review task is completed, the system enters the post-event detection phase. In this phase, the scope of data collection expands from the organizational relationship data in step S110 to actual scoring behavior data—not only recording who gave whom how many points, but also recording multimodal information such as submission time, evaluation text, and device fingerprints, because important clues to relationship-based cheating are hidden in these behavioral dimensions of data.

[0183] The system collects the following records for each rating: The rating record includes the reviewer ID, the target ID, the score, the dimension scores, the submission time, the review text, the device ID, the IP address, the page stay time, the number of edits, and the copy-paste behavior. The latter five items (device, IP, stay time, number of edits, and copy-paste behavior) are used for subsequent synchronous submission detection and device correlation analysis.

[0184] These collected data are assembled into a directed weighted graph with time attributes, allowing rating behavior to be formally analyzed within the graph structure:

[0185]

[0186] in For a set of employee nodes; Let be a set of directed scoring edges, each edge Indicates the reviewer For the person being reviewed An actual rating; It is a set of edge attributes, containing the original score. Submission time Evaluation text Text similarity vector Equipment correlation and IP correlation, organizational relationship strength and historical relationship strength.

[0187] Step S220: Perform style correction processing based on robust statistics on the scoring data in the directed weighted time series scoring graph to eliminate the style bias of the reviewer and the level bias of the reviewed object, and obtain the corrected scoring data.

[0188] The style correction process based on robust statistics includes:

[0189] A double correction is performed using the median absolute deviation: the first correction is to subtract the reviewer's median score from the scoring data and divide by the median absolute deviation of the reviewer's score to eliminate reviewer style bias; the second correction is to subtract the reviewee's median score from the data after the first correction and divide by the median absolute deviation of the reviewee's score to eliminate the reviewee's level bias.

[0190] When calculating the normalized denominator, the first level of correction uses the median absolute deviation of the reviewer's historical scores as the normalization benchmark, and the second level of correction uses the median absolute deviation of the first level of correction values ​​of the reviewed object as the normalization benchmark.

[0191] For reviewers whose historical scoring data volume is lower than the preset threshold, the statistical data is replaced step by step in the order of reviewer themselves, same position or department, and overall; for the reviewed object, the statistical data is replaced step by step in the order of same batch, same position or project, historical scores, and overall.

[0192] Specifically, before proceeding with anomaly detection, a fundamental issue needs to be addressed: different reviewers naturally have different scoring habits—some are accustomed to giving high scores (lenient), while others are accustomed to giving low scores (strict). If the deviation is directly determined by the mean-variance of the same batch, lenient reviewers will generally be misjudged as cheating, while even if strict reviewers participate in cheating or mutual benefit, their "score boosting" will only fall within the normal high score range, thus hiding in the blind spot of the mean-variance. This means that if the differences in scoring styles are not eliminated first, all subsequent anomaly detection will be based on a biased foundation.

[0193] Therefore, this step uses a robust z-score based on median absolute deviation (MAD) for style correction, while simultaneously removing reviewer personal bias and object-inherent level bias. For each reviewer... Calculate the individual style center and dispersion from their historical scores:

[0194]

[0195]

[0196]

[0197] in As reviewer In history The set of all objects rated within a period, a constant Used to convert MAD into values ​​such as standard deviation under the Gaussian distribution assumption. The lower limit of dispersion (typically taking the range of the scoring scale) For example, taking a percentage system ), used to prevent (When reviewers' historical scores are highly consistent) this leads to subsequent calculations being divided by zero.

[0198] For each reviewed object Calculate batch center and dispersion from the ratings of others in the same batch:

[0199]

[0200] Synonymous with the above, used to prevent the denominator from approaching zero when the reviewers' scores are concentrated.

[0201] The deviation correction was calculated using a two-stage MAD-z correction.

[0202] Level 1 correction – eliminating reviewer style bias. For each scoring line… As the reviewer Historical median rating Absolute deviation from historical median rating Perform the first-order z-fractional transformation:

[0203]

[0204] Among them, the reviewers The style statistics are defined as follows:

[0205]

[0206]

[0207]

[0208] As reviewer In history The set of all objects rated within a given period. (Constant) Used to convert MAD into a value equivalent to standard deviation under the Gaussian distribution assumption; The lower limit of dispersion (typically taking the range of the scoring scale) For example, taking a percentage system ),prevent Divide by zero.

[0209] Second-level correction—eliminating the horizontal bias of the reviewed object. Values ​​after first-level correction. There may still be inherent differences in the level of the reviewed subjects (some subjects naturally score higher or lower), therefore, for each reviewed subject... Calculate the median and dispersion of its first-level correction values:

[0210]

[0211]

[0212]

[0213] in To review the subject The set of reviewers who will give the scores, calculate and Exclude the current reviewer Self (Leave one method), avoid The scoring contaminates its own calibration benchmark.

[0214] The second-level correction subtracts the object bias from the first-level result and divides by the object dispersion:

[0215]

[0216] The two-level correction uses the MAD (style variance) of both the reviewer and the reviewed entity as the normalized denominator, avoiding the erroneous inclusion of the reviewer's style variance in the significance determination of the reviewed entity's dimension. At that time, the deviation was considered significant.

[0217] Preferably, the style correction also includes cold start rollback and small sample rollback mechanisms. When the reviewer's historical scoring sample size is insufficient... (typical When there are insufficient new employees or review samples, and Unstable, the statistical regression is based on the median level of the reviewer's own position / department → overall level, and... This holds true. However, if the number of reviewers for the current batch of the reviewed entity is insufficient... (typical In the case of the same batch → same position / project → historical average → global median, the statistical data is backed up by level.

[0218] Step S230: Based on the corrected scoring data, perform stratified detection on multiple preset abnormal patterns to obtain the detection results of each abnormal pattern.

[0219] After style correction, each rating edge obtains a normalized deviation that removes personal style bias. The next step is to search for anomalous signals in parallel across multiple dimensions based on this unified metric. Figure 3 This is a flowchart of the second stage of the present invention—post-mutual evaluation time-series scoring graph anomaly detection. In this step, the seven anomaly modes operate independently, each generating candidate anomaly edges based on its own evidence: - S231 High-score closed-loop detection, S232 Reciprocal cluster detection, S233 Cluster risk assessment, S234 Joint suppression detection, and S237 Long-term mild reciprocity detection directly based on correction deviation. Run; - S235 Abnormal Synchronous Submission Detection and S236 Text Similarity Detection run directly based on submission time data and evaluation text data, without relying on the output of other modes. If the current set of reviewers is not marked as a suspected group in S233, then the synchronization component corresponding to each edge within that set... Or text similarity component Pick This does not trigger the exception determination for this mode, but the detection calculation is still performed on all edges.

[0220] S231: High-resolution closed-loop anomaly mode detection. This involves determining the actual set of short-loop structures after review, allocation, and optimization. The set of short-loop structures that are selected and retained in the short-loop detection By merging, we obtain the actual set of short cycles. For each short loop in the actual short loop set, the mean value of the correction deviation on the loop is calculated as the loop score intensity. When the loop score intensity exceeds the preset average threshold and the correction deviation of each edge on the loop exceeds the preset minimum deviation threshold, the loop is marked as a high-scoring closed loop by AND joint determination.

[0221] Specifically, step S140 enumerates the set of rings on the candidate review relationship graph before allocation. This only indicates potential short-loop structures and does not necessarily mean that all of them have actually been scored. Furthermore, because ILP allows for the retention of a portion of candidate short loops (violating instructions) under soft-constraint mode. (It can still form an edge at times), actual scoring chart The directed short loop in it may be A subset may also appear due to actual adjustments to the review task. Uncovered new short loops. Therefore, the actual set of short loops must first be reconstructed to bridge the candidate short loops before assignment with the actual score map after peer review.

[0222] First, construct the adjacency matrix of the actual ratings. ,in Then, the actual score short loop set is constructed by merging two paths:

[0223]

[0224] Filtering Path From the candidate short cycles in step S140, retain the subset where all edges on the cycle have actually been scored; reconstruct the path. Process those located in But any The rating edges that are not fully covered, in The DFS enumeration is re-executed. After the two paths merge, Simultaneously covering both "short loops identified but not blocked during the allocation phase" and "short loops not identified during the allocation phase but actually occurring", and deduplicating by the smallest rotation in lexicographical order.

[0225] right Each short ring Calculate the closed-loop strength of the scoring:

[0226]

[0227] because The value can be positive or negative. A simple average may mask the anomalies of some high-scoring edges due to the cancellation of positive and negative values—for example, an edge on a cycle deviating from its normal value. And another deviation On average only It's not very noticeable, but actually that line The edge is already highly suspicious. Therefore, the AND joint condition is used for judgment:

[0228]

[0229] Typical threshold (Ring average deviation from threshold) (Minimum deviation from the threshold for each edge on the ring). When both conditions are met, the ring is marked as a high-scoring closed loop, and all scored edges on the ring are recorded. Set, and through Contributed to subsequent .

[0230] S232: Detection of abnormal patterns in mutually beneficial clusters. The directed weighted time series scoring graph is symmetrically transformed into an undirected mutually beneficial weighted graph, and the Louvain community detection algorithm is used to identify mutually beneficial clusters.

[0231] Specifically, high-score loop detection can only find precisely matching loop structures, but in reality, cheating groups may form more complex reciprocal networks than simple loops—for example, five people rating each other in pairs, forming a dense reciprocal subgraph rather than a few isolated loops. To capture this more covert clustered cheating, this invention symmetrically transforms the scoring graph into a reciprocal weight graph and uses a community detection algorithm to automatically identify groups with reciprocal clusters.

[0232] To ensure that the community discovery algorithm is directly sensitive to "two-way reciprocal score boosting" while maintaining the mathematical rigor of standard Newman modularity, a symmetric reciprocal weight graph is first constructed. Its edge weight is defined as the smaller of the positive values ​​of the two directional deviations:

[0233]

[0234] This definition has three key properties: only when the reviewer... and When giving each other positive deviation scores One-way high scores ("star-shaped score boosting") are automatically reset to zero because the contribution in the other direction is zero—thus avoiding misjudging situations where one person gives high scores to multiple people but does not receive a reward as a reciprocal cluster; To ensure the symmetry of the reciprocal weight graph, the undirected modularity form can be directly applied.

[0235] Applying standard Newman modularity to this reciprocal weight graph and iteratively optimizing it using the Louvain algorithm:

[0236]

[0237] in For nodes Strength on the reciprocal weight graph (due to symmetry, out strength equals in strength); The total edge weight of the reciprocal weighted graph; For nodes Community tags; This refers to the Kroneckerdelta function.

[0238] because With zero model terms Both are first-order quantities, and their subtraction mathematically rigorously represents "the excess of the observed reciprocity strength within the community relative to the baseline of the stochastic graph," avoiding the direct use of second-order quantities. Replacement of first order This results in a mismatch in the units of measurement. It should be noted that this step can also use the Leiden algorithm to ensure community connectivity, and the modularity form can be generalized to directed modularity. and on the border Reciprocal gating is retained. All the above variations are within the scope of protection of this invention. The Leiden algorithm (Traag et al., Scientific Reports, 2019) corrects the problem of disconnected communities that the Louvain algorithm might produce, and has better stability for reciprocal group detection. It can be directly combined with the reciprocal modularity of this invention. Complete the community division.

[0239] A fallback option for an empty reciprocal weight graph: when the total edge weight... When there are no two-way positive deviation edges in the mutual evaluation, If no community is defined, skip this step and output an empty community partition. Subsequent cluster-related detections will automatically use an empty set, and the corresponding risk components will be taken as... .

[0240] S233: Cluster Risk Assessment Anomaly Pattern Detection. Based on the identified mutually beneficial clusters, the difference between the average deviation of the scores within the cluster and the average deviation of the scores outside the cluster is calculated. This difference is then combined with the density of mutually beneficial edges within the cluster and the cluster mutuality rate to jointly determine and identify high-risk clusters.

[0241] Specifically, after the Louvain algorithm outputs community divisions, not every community is a cheating group—some communities may simply be close relationships formed naturally by their organizational structures. Therefore, a multi-dimensional risk assessment is needed for each community.

[0242] For each community calculate:

[0243] Average deviation within the cluster ;

[0244] Average deviation outside the cluster (That is, whether community members also gave high scores to non-members—if the difference between those inside and outside the cluster is not significant, then the cluster may simply be a natural gathering of lenient reviewers rather than a cheating group.)

[0245] Cluster internal edge density ;

[0246] Cluster reciprocity rate The proportion of bidirectional scoring edges within a cluster is used as the percentage of unordered pairs, with the denominator being the total number of node pairs within the cluster, excluding self-loops; values ​​are taken as follows. .

[0247] In particular, when the community When none of the members have any rating edges for non-members, The format is undefined. This invention stipulates that at this point, a suspected group determination will not be triggered independently; the community will only be marked as "insufficient external comparison" and retained in the evidence chain JSON as a candidate community for manual review, but will not be entered into the formal process. of Calculations are made to prevent the community from being defaulted to a rating system due to a lack of external ratings. After processing, it was mistakenly identified as a gang.

[0248] A cluster is marked as a suspected reciprocal group when the following joint conditions are met:

[0249]

[0250] Typical threshold , , .

[0251] S234: Detection of Abnormal Patterns of Collaborative Suppression. This detects patterns where multiple reviewers give significantly lower scores to the same reviewer than the reviewer's historical average score, and calculates a suppression score accordingly. The suppression score is determined based on the deviation of the score and the number of reviewers involved in the suppression.

[0252] The aforementioned high-score closed loop and reciprocal cluster detection methods target "mutual score boosting," but another common cheating pattern is "collusive suppression"—several reviewers conspire to give a lower score to an external competitor in order to lower their position in the final ranking. In this pattern, the suppressors often also have a reciprocal score-boosting relationship (i.e., they...). The above forms a dense derived subgraph, creating a dual characteristic of "internal reciprocity + external suppression".

[0253] For each reviewed object Construct its reviewer set Select a subset of reviewers whose scores are significantly lower than the batch median:

[0254]

[0255] typical The edge weights of a reciprocal weighted graph are derived from robust z-scores, which theoretically can exceed [a certain threshold]. (In extreme cases of reciprocal score boosting, it can even reach) Therefore, truncation normalization must be performed first; otherwise, the sum of edge weights will exceed the upper bound of density. Normalized edge weights are introduced.

[0256]

[0257] in For reciprocal deviation cutoff threshold (typically taken) Corresponding to the z-distribution (Truncation). In Calculate the normalized edge density on the derived subgraph:

[0258]

[0259] in Indicates in Take unordered pairs from the middle; denominator The total number of node pairs is determined by... By counting unordered pairs, we can obtain Strictly valid. Definition of suppression / inhibition score:

[0260]

[0261] Depend on and , This is strictly true. The continuous form provides a smoother discriminative surface compared to the original indicator function definition, facilitating subsequent threshold adjustment and gradient learning.

[0262] when (typical )and At that time, mark This is a joint suppression model.

[0263] S235: Synchronous Scoring Anomaly Detection. A kernel density estimation method is used to detect the synchronicity of scoring behavior among multiple reviewers over time. A set of reviewers whose maximum point-by-point ratio of the kernel density estimate submitted by the group to the baseline kernel density estimate exceeds a preset synchronization threshold is identified as having an abnormal synchronous scoring pattern.

[0264] The first four anomaly patterns focus on the rating values ​​themselves, but cheating also leaves traces in the time dimension—if several people submit mutual reviews in a short period of time, while normal mutual review submission times should be spread across the entire mutual review window, this time clustering is itself a suspicious signal. This step performs a global time synchronization analysis on the submission times of all rating edges.

[0265] For each suspected group output by S233 And for all reviewers, collect submission time data from members within the corresponding set, and calculate Gaussian kernel density estimates. This includes identifying suspected groups. :

[0266]

[0267] in Standard Gaussian kernel, bandwidth Apply Silverman's rule of thumb and impose a lower bound:

[0268]

[0269] in The lower limit of bandwidth (typically taken as the mutual evaluation window) For example, peer review cycle Hours (minutes), used to prevent group submissions from being concentrated at the same time, resulting in... Consequently, the bandwidth becomes zero. The KDE for the entire scoring time in the same period is denoted as... The same kernel and the same bandwidth are used for estimation, and the same time unit and the same evaluation grid must be used to ensure the consistency of the point-by-point ratio.

[0270] After KDE normalization, it satisfies ,therefore It is a pure geometric quantity unrelated to any group. If we simply copy... This simplistic definition leads to a situation where, with a small sample size and narrow bandwidth, both the numerator and denominator shift together, resulting in a loss of discriminative power "relative to the baseline." Therefore, this invention defines synchronization as the maximum value of the point-by-point ratio of the group's time distribution to the baseline distribution at its peak:

[0271]

[0272] That is, at all times Calculate the pointwise ratio of the two KDEs and then take the maximum value. This definition ensures that the numerator and denominator are measured at the same time point and have consistent dimensions; when the density of the group within a certain time window is much higher than the overall baseline, Significantly greater than Apply a lower limit to the denominator. (Pick Maximum value over all time periods To avoid numerical instability caused by the baseline density approaching zero. (typical When this happens, all rating edges of the group are marked as abnormal synchronous submissions. If the current set is not marked as a suspected group by S233, then the corresponding edges in this set... Components This does not trigger the synchronous scoring anomaly detection.

[0273] S236: Text Similarity Anomaly Pattern Detection. The review text content is calculated using SimHash Local Sensitive Hashing, MinHash Minimum Hashing, and semantic text similarity based on the pre-trained language model SBERT. The similarities obtained by the three methods are weighted and fused to obtain a comprehensive text similarity score. When the average comprehensive text similarity within the group exceeds a preset similarity threshold, it is determined to be an anomaly in text similarity.

[0274] Beyond ratings and time, the evaluation text forms a third traceable chain of evidence—if several people submit highly similar or even identical evaluation texts, it strongly suggests they may have coordinated the evaluation content. For each suspected group output by S233... Evaluation text of each side The following steps are performed sequentially: (a) text preprocessing (removing stop words, punctuation, and Unicode); (b) calculating the SimHash signature. (c) Calculate the MinHash signature (d) Calculate semantic vectors based on Sentence-BERT embeddings to estimate Jaccard similarity; .

[0275] For suspected gangs Any two edges inside The similarity is calculated by combining the three categories. Here's a detail: SimHash's Hamming distance normalization and MinHash's Jaccard estimation both naturally fall within the range... However, the standard cosine similarity of Sentence-BERT embeddings Direct weighting will Beyond The interval. Therefore, the cosine similarity is first normalized: .by Replace the original Participation in weighted fusion:

[0276]

[0277] Typical weights , , ,satisfy All three components are in Combined with convex, Strictly established. When the average within the gang The time stamp is used to cheat by text template.

[0278] Empty text and short text processing: When evaluating text Empty, fewer than 10 characters (typical When the edge contains only templated fields (such as "no rating" or "good"), it is agreed that the edge will not participate in the text similarity calculation (marked as unknown) and will not be included in the denominator of the average text similarity within the group. When the number of known (non-empty and non-short) text edges within the group is insufficient... When the text similarity component of the article is mentioned, it is considered that the gang... Marked as unknown. When a certain group... When all edges are judged as empty / short text, Pick This does not trigger the text template determination. Furthermore, in step S240... There are no other comparable sides within the group ( That is, the size of the isolated gang or cluster is (Undefined, this is a convention in this embodiment) .

[0279] In an alternative implementation, when evaluating a massive amount of text (millions or more), an approximate nearest neighbor search using LSH (Locality Sensitive Hash) can be performed first to quickly recall candidate similar text pairs. Then, a precise semantic comparison can be conducted on the recalled set, reducing the time complexity from... Down to .

[0280] If the current set is not marked as a suspected group by S233, then the edges corresponding to each edge in the set... Components This will not trigger the text similarity anomaly detection.

[0281] S237: Detection of long-term mild reciprocal anomaly patterns. Utilizes exponentially weighted moving distance. The reciprocal rating behavior among reviewers in multiple peer review tasks, where the score difference is consistently below a preset severe threshold, is considered a long-term mild reciprocal abnormality when the index-weighted moving distance exceeds a preset mild threshold in a consecutive preset number of peer review tasks.

[0282] The first six anomaly patterns were detected within a single peer review period, but one type of cheating is quite subtle: two reviewers only slightly raise each other's scores by a few points in each period. This appears perfectly normal in a single period, but accumulates over multiple periods to form a systematic reciprocal preference. If analyzed only within a single period, this "mild" signal would be masked by random fluctuations. This step amplifies this cumulative effect through exponentially weighted aggregation across periods.

[0283] For each pair of employees From history Collect all in each cycle Deviation sequence of rating edge (in For the first Within a cycle If there is no scoring edge for that period, it is considered missing. Calculate the Exponentially Weighted Moving Deviation (EWMD, distinguished from "Exponentially Weighted Cumulative" to avoid misunderstanding as a summation):

[0284]

[0285] in The attenuation coefficient is typical. The weighting of recent cycles is greater.

[0286] The reciprocity symmetry index introduces two layers of gating simultaneously:

[0287]

[0288] in Minimum number of observation periods. Two-way count gating ensures that the historical score count in both directions reaches the minimum. The indicator function is taken To avoid situations where there are sufficient one-way samples but no reverse samples. Because the denominator is Undefined. Nonnegative truncation. Force the negative results to be zero to prevent accidental signals of long-term negative underperformance (which is inconsistent with the semantics of "long-term mild reciprocal score improvement") from being propagated as positive risks.

[0289] The calculation ensures that the indicator is significant only when there is a long-term positive deviation in both directions. (typical When ), it is marked as a long-term mild reciprocal pair.

[0290] In an alternative implementation, long-term mild reciprocity detection can be replaced by the Holt-Winters time series model instead of EWMD. The Holt-Winters model can simultaneously capture three types of time series characteristics: trend, seasonality, and level, and is more robust to multi-period reciprocity data with seasonal fluctuations.

[0291] Step S240: The detection results of each abnormal pattern are fused to determine the comprehensive abnormal score of each edge in the directed weighted time series scoring graph.

[0292] The fusion of detection results for each abnormal pattern includes:

[0293] For each edge Calculate rule-based anomaly scores respectively Outlier scores based on continuous statistical models The maximum of the two values ​​is taken as the anomaly score for that edge. ,in , , , The three strong rules—high-score closed loop, reciprocal cluster, and joint suppression—have a single vote weight that is greater than the preset anomaly threshold. , , For the binary indicator function corresponding to strong rules; It is obtained by weighted summation of the detection results based on the continuous statistical model among the seven abnormal modes.

[0294] The following is a more detailed description of this step.

[0295] Step S230 presents the detection results for each edge under different modes from seven dimensions—some are triggered by strong rules (such as high-scoring closed-loop hits, suspected gang hits, and joint suppression hits), which belong to hard evidence that "hitting the mark should be judged as anomaly"; others are continuous statistics (such as deviation magnitude, synchronization strength, and text similarity level), which belong to soft evidence that "cumulatively crosses the line". How to integrate these two types of signals with vastly different properties into a unified anomaly score for each edge is the core problem to be solved in this chapter.

[0296] Figure 4 This is a flowchart of an anomaly fusion process provided in an embodiment of the present invention. There are two interrelated issues here. If linear weighting is directly performed... Even if any strong rule is hit, its overall score is still subject to weighting. Limitations (e.g.) A single rule hit only contributes It cannot exceed the threshold. This directly conflicts with the semantics of "strong rules should be judged as abnormal". If it were changed to... In binary form, the edge composite score of all hit strong rules is compressed into Therefore, it becomes impossible to distinguish the severity between strong rule edges—three rules hitting simultaneously is obviously more serious than only one hitting.

[0297] To address both issues simultaneously, a rule-weighted approach is introduced. :

[0298]

[0299] in For the "one-vote passing weight" of each strong rule, a typical example is... (High-scoring closed loop is the most serious) (Suspected gangs are the next most likely culprits) (A joint crackdown again). By Hitting any strong rule will make Crossing the threshold and entering At the same time, due to Different strong rules establish comparable priorities.

[0300] The definition of continuous evidence fusion remains unchanged:

[0301]

[0302] Component definitions (all components are uniformly normalized to) ):

[0303] (Based on robust z-distribution) (Normalization deviation of truncation)

[0304] (Whether it is located on the high-score closed loop of the actual scoring, value) (Strong rules)

[0305] (Value) (Strong rules)

[0306] (Value) (Strong rules)

[0307] ,in

[0308] This represents the point-by-point ratio of the edge at the corresponding time point on the synchronization surface of its respective group;

[0309] ,when Time Agreement ;

[0310] .

[0311] Typical weights , .

[0312] Finally, the maximum value of the two edge anomaly comprehensive scores is taken:

[0313]

[0314] By each , From the properties of convex combinations, we can obtain , Therefore Strictly true. This composite rule preserves both rule interpretability and the orderability of strong rule edges (according to...). The size of the rule distinguishes the severity, avoiding the contradiction of "strong rule hits but not exceeding the threshold" under pure linear weighting, and also avoiding binary... The problem of sorting between different strong rules.

[0315] Anomaly scoring edge sets are determined using a uniform threshold:

[0316]

[0317] in Typical .because Strong rule hit (i.e., any rule hit) ) will inevitably make Therefore, edges with strong rules are automatically assigned to the anomaly set; edges with consecutive evidence accumulating beyond the line are also assigned to the anomaly set. Any two paths that succeed are immediately assigned to the anomaly set.

[0318] As an alternative implementation variation, when the business side wants the overall score to increase further when multiple strong rules are hit simultaneously, a Noisy-OR composite form can be used:

[0319]

[0320] This variant remains The range and the semantics of "multiple rules superposition → increased comprehensive score" fall within the scope of protection of this invention. In an alternative embodiment, the anomaly pattern fusion can also use GBDT or graph neural networks to replace the above-mentioned weighted summation method: taking various anomaly features as input, and using GBDT models such as LightGBM and XGBoost, or graph neural networks such as GraphSAGE and GAT to directly output the results. Probability. This end-to-end learning approach can automatically capture non-linear interactions between features, making it suitable for scenarios with ample labeled data.

[0321] Through steps S210 to S240 above, the entire process from constructing the actual scoring graph and robust style correction to hierarchical detection and fusion of seven types of anomaly patterns is completed, resulting in the comprehensive anomaly score for each scoring edge and the set of anomaly scoring edges. However, identifying anomalous edges is only the first step—whether these anomalous edges actually changed the ranking results, the extent of their impact, and what information the reviewers need to make a final judgment all need to be addressed in subsequent steps. The following steps, S310 to S340, further process the anomalous edges, assess their impact on rankings, generate risk conclusions, and establish a feedback learning loop.

[0322] Step S310: Abnormal edge handling steps.

[0323] For edges in the directed weighted time-series scoring graph whose abnormal comprehensive scores exceed a preset abnormal threshold, the scoring data of the abnormal edges is replaced with the scoring data of normal edges through continuous interpolation. A tiered audit label is added to the replaced abnormal edges, and this label is classified as an approximate deletion adjustment, a replacement-based adjustment, or a weight reduction-based adjustment based on the interval in which the abnormal comprehensive score falls. In subsequent score fusion calculations, a trust weight is introduced for each edge. The trust weight is linearly negatively correlated with the anomaly score of that edge, i.e. .

[0324] Specifically, when faced with a batch of rating edges marked as abnormal, the simplest approach is to delete them directly—but this raises two issues: first, how to handle situations where deletion results in insufficient ratings for the reviewed object; and second, hard deletion can lead to inconsistent processing at the boundary where the abnormal scores just cross the threshold (where one edge has abnormal scores). It was directly deleted. (While the differences between the two are negligible, they are completely preserved.) Therefore, this step employs a unified continuous interpolation replacement strategy, using two values, "trust weight + adjustment score," to represent the processing result, thus avoiding strategy discontinuity caused by tiered breakpoints.

[0325] First, define the explicit convention for non-anomaly edges: for edges not in... The rating edge in the middle, , , In other words, non-abnormal edges are not adjusted and have full trust weight. This convention allows subsequent counterfactual ranking recalculations to be applied uniformly to all scored edges. There is no need to perform branching for abnormal / non-abnormal cases.

[0326] right For each abnormal edge in the algorithm, a trust weight is introduced:

[0327]

[0328] The original scores are weighted and interpolated using trust weights and a baseline score:

[0329]

[0330] in To remove outlier raters, the remaining raters' opinions... The robust center. The set of reviewers after removing outliers is denoted as . ,but:

[0331]

[0332] in (both upper and lower sections) ),when The time degenerates to the median as a fallback, to avoid the loss of statistical significance of the trim operation in small samples. Enter alone when all members are abnormal. Branches: By When historical ratings are available, use the median of the historical ratings for the same subject. When there are no historical ratings but there are comparisons with similar positions / projects, take the corresponding median; when neither is available, take the global median. The system retrieves values ​​hierarchically and triggers a "full-staff exception" flag to initiate manual review and reassignment. Simultaneously, this flag is recorded in the assignmentfallbacklog to ensure... Even in extreme cases, there are still strict definitions and complete audit trails in the review process.

[0333] In cases where the abnormal score is extremely high ( ), , Only the original scores are retained. This is approximately, but not strictly, equivalent to "deleting the edge and replacing it with the baseline score"—the difference being that this invention still retains a very small proportion of the original score to maintain the continuity of the adjustment curve and avoid [further issues]. A policy shift occurs at this point. The anomaly score approaches the threshold. hour, The original rating accounted for Benchmark score This is equivalent to "weighted fusion based on trust weights". That is, the original three-tiered processing of "deletion / replacement / demotion" is automatically unified under continuous interpolation.

[0334] If the business side needs to retain explicit hierarchical labels (such as in an audit scenario), it can be done in the following ways: In addition, a tiered action field is added, used only for audit identification and does not change the status quo. The calculation formula is as follows: Marked as DROP-LIKE (approximate deletion adjustment). Mark it as REPLACE (replacement is the main function). Marked as DOWNWEIGHT (primarily a reduction in weight). This action label is used to distinguish between the evidence chain and the manual review interface.

[0335] Through step S310 above, the abnormal rating edges are replaced with adjusted ratings weighted by trust weights using continuous interpolation, and each edge obtains... and Two processing results. The next core question is: did these adjustments actually change the business decision-making outcome? If the original ranking and the adjusted ranking are basically the same, it means that although abnormal scoring exists, it has not reversed the overall situation; if an employee's ranking falls from the Top-10 to outside the reward boundary after the adjustment, immediate action is required. The following step, S320, quantitatively compares the rankings before and after the adjustment.

[0336] Step S320: Counterfactual ranking distortion assessment.

[0337] Let the original ranking be The processed ranking is To determine whether abnormal scores actually affect the final result, the "change before and after ranking" needs to be quantified into comparable values ​​across five dimensions.

[0338] During ranking recalculation, in step S310 Already pressed The inner-layer weighted interpolation from "raw score to benchmark score" has been completed, therefore When summarizing, you can no longer use... As an outer layer weight, otherwise the same source of uncertainty would be attenuated twice. This step directly applies... Take the arithmetic mean:

[0339]

[0340] according to Descending order generation When there are tied scores, stable tie-breaking is performed based on the priority of "total historical score → earliest submission time → employee number" to ensure that the ranking is reproducible and auditable.

[0341] As a variation of the implementation, it is also possible to omit the inclusion of [the relevant information]. Inner interpolation, while directly interpolating... Make a single layer Weighted:

[0342]

[0343] It needs to be made clear that, and They are not numerically equivalent point by point. (Main solution) Low trust ratings The proportion continuously shrinks to a robust benchmark score Then, an equal-weighted average is applied, and the adjusted total score is anchored to the benchmark score, corresponding to the manual review semantic of "low credibility of this edge → replace with the group consensus value". Variant scheme Instead of introducing a benchmark score as a substitute, the outer weight of low-trust scorers in the aggregation is directly reduced. When a side is completely untrustworthy, it is equivalent to excluding it from the weighted average (the weight is...). The semantic bias "treats low-trust raters as invalid reviewers." The two are only consistent at two endpoints:

[0344] All edges are normal. (Time) degrades to ;

[0345] All edges are completely abnormal ( When the fallback process is triggered, it degenerates into .

[0346] Except for these two endpoints, the two are generally not equivalent, especially in In cases of partial edge anomalies, "replace first, then average equally" and "weight by trust weight" will yield different results—only when... The two values ​​will only overlap if they happen to be exactly equal to the weighted average of the non-abnormal scores.

[0347] Regarding monotonicity: the main scheme Regarding either side The changes follow the "towards" The direction of convergence—if An increase in anomaly degree causes a decrease in the contribution of that edge, and vice versa; its monotonic direction depends on and The relative size of the two cannot be simply described as "monotonic in the same direction".

[0348] Specific selection or The decision rests with the deployer based on interpretability requirements: the former facilitates explanations based on the "adjusted score," while the latter facilitates directly displaying the "credibility weight of the rater's opinion," both falling within the scope of this invention. When using... And appeared In the extreme case (all members are abnormal), it is agreed to degenerate into .

[0349] To determine whether abnormal ratings actually affect the final result, the "change in ranking" needs to be quantified into a comparable numerical value. After obtaining two rankings, this embodiment quantifies the ranking distortion from five dimensions:

[0350] (1) Kendall Distance – a measure of overall order change:

[0351]

[0352] That is, calculating the Kendall ranking sequence of the reviewed object before and after removing the edges whose abnormal comprehensive scores exceed the preset abnormal threshold. The proportion of inconsistent pairs to the total number of logarithms.

[0353] (2) Top-K Jaccard distance – a measure of change in the Top-K list:

[0354]

[0355] That is, calculate the ranking before and after removing abnormal edges. The Jaccard distance of a set of reviewed objects, wherein the Jaccard distance is... Subtract the Jaccard similarity between the two sets.

[0356] (3) Boundary crossing rate – measures the crossing of reward / promotion boundaries, strictly distinguishing between inside and outside the boundary using XOR logic (statistically counting the number of reviewed objects that crossed the preset ranking boundary before and after removing abnormal edges and normalizing by dividing by the total number of reviewed objects):

[0357]

[0358] in This represents the logical exclusive OR (XOR) operation. This is a Boolean condition for "being within the reward boundary". If the item is within the boundary before processing but falls outside the boundary after processing (or vice versa), XOR is true, and the crossover is counted. This formula strictly distinguishes between inside and outside the boundary, avoiding the original... Form in Take This leads to ambiguity in the judgment. Similarly, if promotion boundaries exist simultaneously... Parallelism can be defined. Multiple boundary crossing rates can be linearly combined according to business weights.

[0359] (4) Maximum ranking change (record the maximum change in ranking of a single reviewed object before and after removing the abnormal edge and normalize it by dividing by the total number of reviewed objects) - Normalized maximum single-point ranking change:

[0360]

[0361] (5) Contribution rate of abnormal edges (calculate the proportion of the average score adjustment magnitude of abnormal edges to the score range, and truncate to the preset upper limit) — measures the proportion of the cumulative disturbance of abnormal score adjustment magnitude to the score range:

[0362]

[0363] in , The upper and lower bounds of the scoring range (on a 100-point scale) , ), To prevent Small constants that divide by zero (typically taken as...) ),like but The denominator was changed to "the number of abnormal edges multiplied by the scoring range" so that the ratio would strictly fall within the range even under extreme conditions such as joint suppression. Inside, guarantee .

[0364] The five components are weighted and fused using convex combination:

[0365]

[0366] Typical weights , , .

[0367] In an alternative implementation, the ranking distortion can also be replaced by the Spearman footrule distance instead of Kendall's. Spearman footrule distance Faster calculation ( ), suitable for real-time ranking monitoring; Kendall distance It is more sensitive to significant changes in local sorting. One or a weighted combination of the two can be chosen depending on the specific business scenario.

[0368] Through the above step S320, the ranking distortion degree was obtained. This quantifies the actual impact of anomaly scores on the final ranking. The next step, S330, integrates the ranking distortion with the anomaly detection results from the aforementioned dimensions into a unified task-level risk score and generates a structured chain of evidence.

[0369] Step S330: Comprehensive risk assessment.

[0370] The aforementioned steps provide quantitative assessments from seven dimensions: short-loop risk, gang anomalies, scoring deviation, synchronous submission, text similarity, historical reciprocity, and ranking distortion. However, these dimensions have different dimensions and magnitudes, and need to be unified and normalized to be merged into a single risk score that can be used for decision-making.

[0371] Preferably, in this embodiment, the seven risk components are convexly combined to form the final task-level mutual evaluation cheating risk score, that is, the comprehensive risk score is calculated based on the detection results of each side under the multi-dimensional anomaly mode. :

[0372]

[0373] All components are uniformly normalized to For each component, we distinguish between two cases:

[0374] • Confirmed non-existent: If, after inspection, it is confirmed that the corresponding structure does not exist (e.g., no text field, no historical mutual evaluation records, no corresponding edge type in the graph), the component is taken as... ;

[0375] • Data missing: Evidence not collected or sample size insufficient (e.g., text fields not enabled, insufficient historical periods). When behavior logs are not collected, the component is marked as unknown and does not participate in the weighted summation. The corresponding convex combination weights are renormalized according to the weights of the remaining known components.

[0376] Simultaneously calculate evidence coverage. The proportion of the sum of the known component weights to the sum of all weights:

[0377]

[0378] when Below the preset threshold (typically taken) When this occurs, add a "insufficient evidence" marker to the chain of evidence.

[0379] The components are defined as follows:

[0380] That is, the actual rating chart The middle is located in the short loop of the actual score. Or actual rating history reciprocal loop The proportion of the scoring edges on the scale to the total number of actual scoring edges. Reconstructed from step S231, For a real short loop containing historical fixed edges, when Time to take If precautionary risks need to be preserved during the allocation phase, a separate provision will be made. Candidate short loops detected in step S140 that were not blocked As a supplementary audit indicator, it is not included. .

[0381] Based on robust z-distribution Truncation normalization, when Or retrieve when the community is empty .

[0382] Values .

[0383] Normalized and truncated using a synchronization threshold, when no group exists, take... .

[0384] Take when it is confirmed that no gang exists. When a group exists but there is no comparable text within the group (text fields are not enabled or are all empty / short text), the component is marked as unknown.

[0385] With a long-term, moderate, reciprocal threshold Normalization and truncation, when the historical observation period is insufficient When this happens, the component is marked as unknown and does not participate in the weighting.

[0386] Typical weights , . By each component With convex combination weights, Multiply Strictly fall into .

[0387] The above weighted summation is performed only on known components, and the weights are renormalized according to the known components:

[0388]

[0389] in For the first Normalized values ​​of each risk component. Evidence coverage. The risk score is output along with the risk score, allowing reviewers to assess the reliability of the conclusions.

[0390] The overall risk score is mapped to a five-level risk rating, with each level corresponding to a specific business action: Low risk (normally included in the results) For low to medium risk (mark observations, record evidence), For medium to high risk (reduced weighting or subject to random inspection and review) High risk (results temporarily suspended, subject to manual review) This is considered extremely high risk (reassignment of review or initiation of audit investigation).

[0391] The ultimate value of risk assessment lies not in providing a numerical score, but in supporting the judgment and decision-making of reviewers. Therefore, for each review relationship deemed abnormal and reaching a preset risk level, a structured chain of evidence in JSON format is generated, including: the abnormal path (the sequence of nodes on the actual short loop), a list of suspected gang members, details of the abnormal edge (original score, adjusted score, trust weight, overall abnormal score, audit label), a list of detected abnormal pattern types, details of ranking distortion (comparison of original and adjusted rankings, boundary crossing situations), model version and confidence level, and evidence coverage. A list of unknown dimensions (listing the names and reasons for the risk components marked as unknown) and recommended actions.

[0392] Through step S330 above, the entire process of peer review cheating detection has output risk scores and evidence chains. However, cheating methods are constantly evolving, and the static rule system inevitably decays over time. In order for the system to continuously absorb human review feedback and adapt to the evolution of cheating patterns, the following step S340 establishes a layered and decoupled feedback learning closed loop.

[0393] Step S340: Layered feedback learning process.

[0394] Manual review provides labels ( This indicates that there was no cheating. After confirming cheating, how to update the parameters requires understanding the differentiability boundaries of the entire process. Here's an often overlooked fact: the complete peer review process does not constitute an end-to-end differentiable link— Kendall is in it. (Counting inversions), Top-K Jaccard (intersection and union of sets), Discrete operations such as (XOR) There is (Set counting based on hard thresholds) These methods cannot naturally backpropagate. If end-to-end SGD is applied indiscriminately, the gradient will generate incorrect update directions when passing through these discrete steps.

[0395] Therefore, in this embodiment, the parameters are divided into six groups according to the "gradient reachable path length," and each group adopts a corresponding update mechanism. Parameters at lower levels are closer to the gradient and are updated more frequently; parameters at higher levels pass through more non-differentiable stages and can only be optimized offline. The hierarchical strategy for each parameter group is as follows:

[0396]

[0397] First layer (edge-level softening layer) – Edge-level softening training (for use only) ):

[0398] Only continuous evidence is fused into weights Edge anomalies are comprehensively analyzed using the softening indicator function and the softening maximum value operator. The problem is transformed into a differentiable approximation. The cross-entropy loss is calculated using manually verified edge-level labels as supervision signals on the edge-level pseudo-probabilities, and then updated using stochastic gradient descent. During deployment inference, the softened operator is reverted to hard computation. Specifically:

[0399] (Continuous evidence fusion weights) only apply to a single scoring edge. Only on the link Point-state operations such as indicator functions can be softened into differentiable approximations. This is achieved by introducing softening operators.

[0400] and

[0401] in Temperature coefficient (typically taken during training) (The higher the temperature, the closer it is to hard computation). The soft version is used for computation during the training phase. Obtain the edge-level pseudo-probability:

[0402]

[0403] The given boundary labels are manually reviewed. Calculate the edge-level cross-entropy loss: .

[0404] Reparameterization via softmax Eliminating simplex constraints, in Space-based SGD execution: , Typical Temperature during inference deployment The softening operator converges back to the hard threshold and This ensures that training and inference are consistent at the limits.

[0405] Second layer (calibration layer) – Calibration layer convex combination update (used for…) ):

[0406] (Distortion and Risk Fusion Weights) Quantity (Kendall) Top-KJaccard )and Quantity (including) (Equal to set counting) depends on sorting and set operations, and is not backpropagated in the SGD link. Therefore, these components are treated as pre-calculated feature vectors and fed into a separate calibration layer:

[0407]

[0408] in, , The calibration layer treats each component as a given input and only updates the convex combination weights. as well as No sorting, Top-K, XOR, Discrete operations are then backpropagated.

[0409] Task-level tags based on manual review Calculate the task-level binary classification cross-entropy loss:

[0410]

[0411] Similarly, use softmax reparameterization + SGD update. and .

[0412] This layering has clear semantic boundaries: the first layer answers "whether the discrepancy of a single scoring edge is accurate," and the second layer answers "given that the components have already been calculated, how to weight them to obtain a reasonable task-level risk." Neither layer involves backpropagation of the sorting results, set size, or ILP solution, fundamentally avoiding the dishonest practice of end-to-end SGD breaking down in the discrete stage.

[0413] The third layer (threshold adaptive layer) – adaptive adjustment of the threshold parameter:

[0414] For each non-differentiable threshold parameter, based on near The exponential moving average of the optimal F1 value on the second-reviewed sample is used for adaptive updating, where The preset number of sliding window samples, specifically:

[0415] Threshold set All appear in hard indicator functions In the middle, the deployment phase Since it's not differentiable, gradient descent cannot be directly applied. Therefore, an exponential moving average based on the F1 optimum of historical verification samples is used instead:

[0416]

[0417] in The coefficient of the exponential moving average (typical) ), Typical , From near The F1 curve was obtained by scanning the sample, with a scanning interval of [value missing]. .

[0418] The fourth layer (offline hyperparameter optimization layer) – offline parameter tuning based on ILP discrete solutions or Lagrange tradeoffs: aggregating parameters based on risk weights obtained from mixed-integer programming discrete solutions. and regularization coefficient Offline periodic tuning is performed using the Bandit algorithm, Bayesian optimization, or grid search. The time decay coefficient and smoothing constant are fixed offline on the validation set before deployment. Specifically:

[0419] The following parameters, due to the discrete decision-making process via the ILP solver on the link, cannot have their gradients backpropagated and can only be tuned offline:

[0420] RiskEdge Weighting The system employs contextual bandit (LinUCB / Thompson Sampling) or Bayesian optimization (Gaussian process + EI sampling function), using "post-hoc anomaly recall rate of the allocation scheme + Top-K Jaccard stability" as a black-box reward signal for iterative updates. Each peer review cycle is treated as an episode, and batch updates are performed after accumulating a sufficient number of episodes offline. You can also directly in Perform grid search over a historical period.

[0421] Lagrange trade-off It is optimized through dual decomposition or offline grid search and does not participate in online updates.

[0422] Time decay With smoothing constant : Solidify offline after validation on the holdout set.

[0423] Through these three separate update mechanisms, differentiable and non-differentiable links are strictly distinguished, avoiding errors in the update direction caused by directly substituting non-differentiable parameters from the discrete ILP solution into the gradient formula. The complete feedback learning loop can be summarized as follows: manual review → data feedback → updates of the first and second layers (online SGD), the third layer (online EMA), and the fourth layer (offline Bandit / grid search) respectively → effective in the next peer review cycle.

[0424] In an alternative implementation, online learning can also use contextual bandit instead of gradient descent: the feature vector of each rating edge is regarded as the context, and the disposal action (pass / mark / review / reassign) is regarded as the arm. The optimal policy is learned using algorithms such as LinUCB or Thompson Sampling, which can automatically balance exploration and exploitation.

[0425] To facilitate deployment and implementation, the following summarizes the empirical values ​​of the threshold parameters and weights involved in each step of this embodiment:

[0426]

[0427] Through the above steps S110 to S340, the present invention constitutes an end-to-end data processing closed loop, which can be broken down into seven stages:

[0428] (1) Prevention before allocation: The allocation of mutual review tasks is abstracted into a directed graph of candidate review between employee nodes. By short cycle detection (DFS + matrix power coarse screening) and constraint optimization (MIP, providing two implementations of hard blocking and soft penalty), the probability of binary, ternary and quaternary mutual benefit structures being formed before the review task is issued is reduced.

[0429] (2) Reconstruction of actual short loops after peer review: After the peer review is completed, a directed weighted time-series scoring graph is constructed based on the actual scoring log, submission time log and evaluation text log. The actual scoring short cycle set is obtained by combining the two approaches: "step S140 candidate short cycle screening + DFS re-enumeration". This connects the two stages before allocation and after mutual evaluation.

[0430] (3) Robust style correction: Two-level MAD-z correction is adopted. The first level uses the median of the reviewer’s historical scores and MAD to eliminate style bias. The second level uses the median of the first-level correction value of the reviewed object and MAD to eliminate level bias, so as to obtain the correction deviation after eliminating the reviewer’s individual style and the object’s inherent level.

[0431] (4) Multimodal anomaly identification: Based on residual deviation and multimodal behavioral data, seven parallel branches are used to identify seven types of relational anomalies: high-scoring closed loops, reciprocal clusters, cluster risk assessment, joint suppression, synchronous submission, text similarity, and long-term mild reciprocity. Then, through... Integrating strong rule weighting with continuous evidence (wherein) and This ensures that a single vote is passed while also allowing for the sorting of strong rules, thus forming a set of abnormal scoring edges. .

[0432] (5) Continuous interpolation adjustment: for The edge of the middle Perform continuous interpolation replacement to avoid breakpoints in different ranges.

[0433] (6) Counterfactual ranking recalculation: using Kendall The impact of outlier edges on the final ranking is quantified using three metrics: distance, Top-K Jaccard distance, and boundary crossing rate (XOR form).

[0434] (7) Evidence chain output and feedback learning: Generate an evidence chain containing abnormal paths, abnormal edges, ranking changes, and handling suggestions; and simultaneously train and calibrate the edge-level softening layer (online SGD, only Threshold EMA (online), offline Bandit / grid search (for...) The three-tiered mechanism continuously incorporates feedback from manual review.

[0435] The seven steps together form a closed loop: "pre-allocation prevention → post-peer review and detection → adjustment of abnormal sides → ranking impact assessment → manual review and feedback." The technical effectiveness of this closed loop can be evaluated from the following seven dimensions:

[0436]

[0437] To further verify the technical effects of the peer review cheating identification method provided in this embodiment, the beneficial effects that this application embodiment can achieve are explained below in conjunction with specific experimental data and comparison results.

[0438] The experiment was conducted on a controlled simulated peer review dataset. Cheating samples were injected using a known process and obtained ground truth labels accordingly. Unless otherwise specified, all results are... The mean of repeated experiments with 1 independent random seed, and the standard deviation is given. Confidence intervals and paired significance tests.

[0439] Construction of the experimental dataset. First, generate the organizational world: a set of employee nodes. Furthermore, a department tree, job sequence, project participation bipartite diagram, hierarchical chain, and approval chain are generated on it, corresponding to the risk weight of the candidate review edge in step S130. Features to The source of truth. Secondly, generate the ideal score in the cheat-free case using an additive model isomorphic to step S220:

[0440]

[0441] in As a global benchmark (out of 100) ); Due to the reviewer's style bias, it was categorized as lenient. Strict type Neutral type Three types of mixtures are generated, with a mixing ratio of [missing information]. , , ; The inherent true level of the subject being reviewed. This true level is also used as the gold standard ranking for evaluating the accuracy of the five evaluation rankings. For random residuals, Scoring cutoff to range And round down.

[0442] It should be noted that this embodiment specifically incorporates the reviewer's style. The cheating injection was set as two independent random processes to cleanly measure the false alarm rate of "misjudging reviewer individual style as cheating" in Effect 4.

[0443] On the aforementioned cheat-free baseline dataset, cheating was injected according to seven patterns and assigned a ground truth label. Among them, the single-cycle score increase of long-term mild reciprocity (M7) was deliberately set to a small value to accurately reflect the detection difficulty of this type of covert cheating.

[0444]

[0445] Overall composition of the dataset: number of employee nodes Number of departments The number of projects is Number of mutual evaluation cycles Number of reviewers per subject and the upper limit of reviewers' tasks Total number of edges in the full cycle score Inject cheat edge count Items (percentage) ), indicating the number of gang cases Marking the long-term mild reciprocal logarithm Number of style test reviewers (relaxed) name Strict type (None of the names were obtained through cheating).

[0446] Comparison Method. To avoid comparing only with a weak baseline, a rule-based baseline and a general graph anomaly detection strong baseline without scene customization are also set.

[0447] A0 Random assignment Baseline allocation A1 Assigned according to department / project rules Baseline allocation A2 Rule-based allocation + load balancing (without blocking short loops) Baseline allocation A★ This invention relates to short-cycle constrained mixed integer optimization (HARD / SOFT two embodiments). This invention D0 No anomaly detection performed Detection baseline D1 Global mean-variance single-point threshold alarm Detection baseline D2 Historical rating profile threshold alert Detection baseline D3 General graph anomaly detection (reciprocity rate + standard modularity, no scenario customization) Detect strong baseline D★ The entire process from steps S220 to S330 of this invention This invention

[0448] Evaluation metrics and statistical methods. Boundary level detection uses truth labels. For this purpose, precision, recall, and F1 score are defined as follows:

[0449]

[0450] Given that the proportion of cheating samples is only about This is a class imbalance problem; therefore, the primary report is F1 and PR-AUC (area under the precision-recall curve), and the secondary report is ROC-AUC. For long-term mild reciprocity... ( (Taking the number of true positive examples). The effectiveness of short-loop blocking is measured by the decrease in the number of A0s relative to random assignments: The false alarm rate is based on the total number of cheat-free scoring edges: .

[0451] In terms of statistical methods, each configuration is based on Each independent random seed is repeated, and the reported mean is... Standard deviation and bootstrap-based )of Confidence intervals; differences between this invention and each baseline were assessed using paired Wilcoxon signed-rank tests, with Holm-Bonferroni correction for multiple comparisons, and Cliff's effect size was also reported. All thresholds and weights are selected on the validation set, and the test set is evaluated only once during training. verify The tests are periodically segmented and mutually exclusive with the group to prevent data leakage.

[0452] Technical Effect 1: Reduces the probability of forming reciprocal cheating relationships at the source. As shown in Table 1-1 below, compared to random allocation of A0, the short-loop constraint allocation (HARD embodiment) of this invention reduces the number of binary, ternary, and quaternary mutual evaluation loops respectively. , , Historical reciprocity cycle decline The rule assignment A1 only reduces the binary ring's... It is almost ineffective for three- and four-membered rings. Meanwhile, the average professional matching degree of this invention only decreases relative to A1. The gap relaxation trigger rate is This indicates that short-loop blocking did not come at the cost of significantly sacrificing allocation quality.

[0453]

[0454] This effect stems directly from the prevention of cheating relationship structures by short loop detection in step S140 and short loop constraints in step S150 before the review task is issued.

[0455] Technical Effect 2: Significantly improves the detection rate of multi-person collaborative relationship-based cheating. As shown in Table 1-2 below, for relationship-based cheating such as ternary high-score closed loops and reciprocal clusters, the edge-level F1 of the traditional mean-variance single-point threshold D1 is only... (its recall rate is only) This confirms that single-point thresholds are almost blind to relational cheating (the F1 score of the abnormally strong baseline D3 in the general graph is...). The F1 of the present invention D★ reaches Recall rate PR-AUC The F1 improvement of this invention relative to D3 is significant (after correction) according to the paired Wilcoxon test. Cliff's This effect stems from the MAD-z style correction in step S220, the actual short-loop reconstruction and high-resolution loop closure determination in step S231, the symmetric reciprocal weighted graph cluster detection in step S232, and the strong rule weighting and continuous evidence fusion in step S240. synergy.

[0456] Table 1-2 Group Cheating Detection (Side Level)

[0457] D1 Mean-Variance 0.610 0.092 0.160 0.121 0.612 D2 Historical Portrait 0.550 0.141 0.224 0.188 0.667 D3 General Plot Anomaly (Strong Baseline) 0.706 0.572 0.632 0.661 0.842 D★ This invention 0.823 0.847 0.835 ± 0.018 0.871 0.932

[0458] Further breakdown of the recall rate of this invention by cheating mode: M1 binary loop M2 high-resolution closed loop M3 reciprocal cluster M4 and other forces jointly suppress M5 simultaneous submission M6 text template M7 is mild in the long term. The detection rate of the explicit pattern is relatively high, while the detection rate of the deliberately mild long-term reciprocal pattern is relatively low, which is consistent with the objective law that "the weaker the signal, the more difficult it is to detect".

[0459] Technical Effect 3: It can identify long-term, mild reciprocal patterns that are difficult to detect using traditional methods. As shown in Tables 1-3 below, for long-term, mild reciprocal patterns that only slightly increase in score per cycle and accumulate over multiple cycles, traditional methods... only The present invention achieves EWMD multi-cycle polymerization in step S237. As can be seen from the binning based on the intensity of the lift, when the lift in a single cycle drops to... Recall rate Rise to Shida This indicates that the detection capability increases monotonically with the intensity of cheating.

[0460]

[0461] Technical effect four: Significantly reduces the false positive rate for normal rating styles. As shown in Table 1-4 below, for Long-term loose type and With a team of long-term, rigorous reviewers, none of whom have cheated, the overall false alarm rate of the mean-variance method D1 is [missing value]. (Among them, the loose-fitting Gundam) (Systematic false positives), while the present invention undergoes MAD-z dual correction in step S220 (based on the full history residual dispersion). After setting the denominator, the overall false alarm rate decreased to Furthermore, the lenient and strict approaches tend to be balanced. This effect demonstrates that the present invention can distinguish between individual reviewer style and style-embedded cheating.

[0462] Table 1-4 Style Correction False Alarm Rate (No cheating detected in any of the subjects)

[0463] D1 Mean-Variance 19.8% 6.6% 13.2% D3 General Diagram Anomaly 11.4% 4.8% 8.1% D★ This invention (MAD-z) 3.9% 2.9% 3.4% ± 0.6%

[0464] Technical benefit five: The actual impact of outlier scores on the final ranking can be quantified. This is based on the true level generated by the model. As the gold standard ranking, the ranking distortion output in step S320 of this invention... The Pearson correlation coefficient with the actual ranking perturbation is The Spearman correlation coefficient is After continuous interpolation replacement and counterfactual recalculation in steps S310 and S320, the ranking of the reviewed object relative to the gold standard Kendall From the original ranking Upgraded to This indicates that the present invention can not only identify anomalies, but also distinguish between two types of situations: "abnormal scores that do not affect rankings" and "abnormal scores that cross reward / promotion boundaries".

[0465] Technical benefit six: Improves the interpretability and review efficiency of risk control conclusions. In a study conducted by... Number of reviewers, cross-design approach, coverage In the review study of the case, when using the multimodal evidence chain output by step S330 of the present invention, the average review time per case is reduced compared to when there is no evidence chain. Minutes down minutes (approximately) (times), and at the same time, the consistency of the review conclusions was verified. Cohen's Depend on Upgraded to It should be noted that this is a human factor experiment, and its values ​​are affected by individual differences among reviewers. Therefore, a crossover design was used to offset individual differences and report the sample size.

[0466] Technical Effect Seven: Versatility, Scalability, and Operational Efficiency. As shown in Tables 1-5 below, this invention achieves [the following benefits]. The end-to-end time per employee per cycle is The time consumption is in seconds, with mixed integer optimization being the main time-consuming step. (seconds); when the number of employees exceeds When the time is 10,000, switch to a greedy algorithm using the alternative solution. Simulated annealing heuristic solution, End-to-end time with 10,000 employees Seconds, the average gap of the optimal objective value in relative integer linear programming is This method is not tied to a specific business scenario and can be applied to scenarios such as performance peer review, project peer review, and promotion evaluation. Only the corresponding feature weights and threshold parameters need to be adjusted.

[0467] Table 1-5 Operating Efficiency (Different Scales, Single Cycle)

[0468] 1,000 Integer linear programming 4.1 s 0.6 GB 5,000 Integer linear programming 63.0 s 2.3 GB 20,000 Greedy algorithm + simulated annealing 41.2 s 4.8 GB 100,000 Greedy algorithm + simulated annealing 287 s 9.7 GB

[0469] Ablation Experiment: To verify that each key technical feature does indeed contribute substantially to the above effects, using the entire link of this invention as a reference, individual technical features were removed sequentially, and the degradation of the corresponding indicators was observed. The results are shown in Tables 1-6. It is evident that each removed feature led to a significant degradation of at least one indicator: removing short-loop blocking reduced the binary loop's false positive rate from 97.6% to 44.1%; removing MAD-z correction increased the false positive rate from 3.4% to 12.7%; removing the symmetric reciprocal weight graph (replacing it with ordinary modularity) reduced the gang F1 score from 0.835 to 0.766; removing EWMD reduced the long-term mild reciprocal recall rate from 0.647 to 0.068; and removing the max composite rule (replacing it with pure linear weighting) reduced the gang recall rate from 0.847 to 0.701. This indicates that this invention is not a simple stacking of known algorithms, and that there is an irreplaceable synergistic relationship between the various technical features.

[0470]

[0471] Based on the experiments described above, this invention significantly reduces the probability of reciprocal cheating relationships at the source during the allocation phase. In the post-evaluation phase, it significantly outperforms existing single-point thresholding and general graph anomaly methods against multi-person collaborative relationship-based cheating, long-term mild reciprocity, and style-embedded cheating. Furthermore, it can map graph-level anomalies back to the ranking impact at the business level, while improving interpretability, review efficiency, and scalability. Ablation experiments further demonstrate the irreplaceable nature of each key technical feature. It should be objectively noted that the above values ​​are taken from a simulated dataset with known process-injected cheating, and its signal clarity is higher than real business data. In a real anonymized environment, due to the more concealed nature of cheating and the greater difficulty in labeling, detection metrics are expected to decline somewhat (the F1 score for group detection is expected to be approximately...). to The long-term moderate reciprocal recall rate is expected to be approximately to The data in this embodiment is only used to verify the relative contribution of each technical feature of the present invention and the feasibility of the method.

[0472] On the other hand, embodiments of the present invention also provide a mutual evaluation cheating identification device based on short loop blocking and time-series scoring graph anomaly detection, such as... Figure 5 As shown, the device includes:

[0473] The data acquisition and normalization module 100 is used to acquire mutual evaluation task data and organizational relationship data, and perform entity parsing and normalization processing to obtain a normalized entity dataset. For a detailed description, please refer to step S110.

[0474] The candidate reviewer relationship graph construction module 200 is used to construct a candidate reviewer relationship graph based on the normalized entity dataset. The candidate reviewer relationship graph includes multiple types of nodes and multiple types of edges. For a detailed description, please refer to step S120.

[0475] The risk weight calculation module 300 is used to calculate the risk weight of each candidate review edge in the candidate review relationship graph. For a detailed explanation, please refer to the description of step S130.

[0476] The short loop detection module 400 is used to perform short loop detection on the candidate review relationship graph based on the risk weight value, and identify short loop structures containing a preset number of nodes. For a detailed description, please refer to step S140.

[0477] The review allocation optimization module 500 is used to optimize the review allocation based on the results of the short loop detection, with the goal of minimizing the risk associated with the short loop structure, to obtain the target review allocation scheme. For a detailed description, please refer to step S150.

[0478] The scoring graph construction module 600 is used to collect the multimodal scoring data generated by the mutual evaluation after the mutual evaluation is completed, and construct a directed weighted time series scoring graph based on the target evaluation allocation scheme and the multimodal scoring data. For a detailed description, please refer to step S210.

[0479] The style correction module 700 is used to perform style correction processing on the scoring data in the directed weighted time series scoring graph based on robust statistics, eliminating the style bias of the reviewers and the level bias of the reviewed objects, and obtaining the corrected scoring data. For a detailed description, please refer to step S220.

[0480] The anomaly pattern stratified detection module 800 is used to perform stratified detection on multiple preset anomaly patterns based on the corrected scoring data, and obtain the detection results for each anomaly pattern. For a detailed description, please refer to step S230.

[0481] The anomaly fusion module 900 is used to fuse the detection results of the various anomaly modes to determine the comprehensive anomaly score of each edge in the directed weighted time series scoring graph. For a detailed description, please refer to step S240.

[0482] In terms of deployment and implementation, a typical architecture of this invention is as follows: the offline training module uses Python 3.9 + PyTorch 2.0 + NetworkX 3.0 + Gurobi 10.0 and runs on a GPU cluster; the online inference module uses Java / Go microservices and integrates ONNX Runtime to infer trained models; the graph storage uses Neo4j or JanusGraph to store candidate review relationship graphs and time-series scoring graphs; the scoring log uses Kafka + ClickHouse to support fast querying of historical scores over multiple periods; the evidence chain storage uses MongoDB and is indexed by task number; and the review feedback interface uses REST API + WebSocket to push data to the reviewers' workbench in real time.

[0483] In addition, embodiments of the present invention also provide an electronic device for identifying peer review cheating. This electronic device can be a terminal and may include a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the electronic device provides computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the electronic device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a peer review cheating identification method. The display screen of the electronic device can be a liquid crystal display screen or an e-ink display screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0484] Those skilled in the art will understand that the structure of the above-described electronic device is only a partial structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than the aforementioned components, or combine certain components, or have different component arrangements.

[0485] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the mutual evaluation cheating identification method as described in the embodiments of this application.

[0486] In an exemplary embodiment, a storage medium is also provided, which, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to execute the mutual evaluation cheating identification method in the embodiments of this application.

[0487] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the mutual evaluation cheating identification method in the embodiments of this application.

[0488] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0489] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0490] The above description is only a preferred embodiment of the present invention. It should be understood that the above description of the embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, etc. made within the idea and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying mutual evaluation cheating based on short-loop blocking and time-series scoring graph anomaly detection, characterized in that, include: Acquire peer review task data and organizational relationship data, and perform entity parsing and normalization to obtain a normalized entity dataset; Based on the normalized entity dataset, a candidate review relationship graph is constructed, which includes multiple types of nodes and multiple types of edges. Risk weights are calculated for the candidate review edges in the candidate review relationship graph to obtain the risk weight value of each candidate review edge. Based on the risk weight value, short loop detection is performed on the candidate review relationship graph to identify short loop structures containing a preset number of nodes; With the goal of minimizing the risk associated with the short loop structure, the review allocation optimization solution is obtained based on the results of the short loop detection to obtain the target review allocation scheme; After the peer review is completed, the multimodal scoring data generated by the peer review is collected, and a directed weighted time series scoring graph is constructed based on the target review allocation scheme and the multimodal scoring data. The scoring data in the directed weighted time series scoring graph is subjected to style correction processing based on robust statistics to eliminate the style bias of the reviewers and the level bias of the reviewed objects, and the corrected scoring data is obtained. Based on the corrected scoring data, a variety of preset abnormality patterns are detected in a stratified manner to obtain the detection results of each abnormality pattern. The detection results of each abnormal pattern are fused to determine the comprehensive abnormal score of each edge in the directed weighted time series scoring graph.

2. The method according to claim 1, characterized in that, The various types of nodes include employee nodes. Departmental nodes Project Nodes and historical mission nodes The various types of edges include reviewable edges. same department Joint project superior and subordinate sides Approval chain edge Historical peer review Collaborating with history .

3. The method according to claim 2, characterized in that, The calculation of risk weights for candidate review edges includes: For each reviewable edge Extract a six-dimensional feature vector, the dimensions of which include the historical mutual evaluation strength. Organizational closeness Project overlap Strength of superior-subordinate relationship Strength of the approval chain and historical reciprocity ; Through convex combination functions The six-dimensional feature vectors are aggregated into reviewers. For the reviewed object The risk weight values ​​of the candidate review edges, where the aggregate weight satisfy and .

4. The method according to claim 1, characterized in that, The short-loop detection of the candidate review relationship graph includes: The candidate review relationship graph is coarsely screened by exponentiation of the adjacency matrix to delineate the candidate node set. Then, a depth-limited depth-first search is performed on the candidate node set to accurately enumerate the basic simple cycles, thus obtaining the candidate cycle set. In the candidate ring set, binary mutual evaluation rings, ternary mutual evaluation rings, quaternary mutual evaluation rings, and historical mutual evaluation rings constructed based on the mutual relationships identified in historical mutual evaluation task data are detected respectively.

5. The method according to claim 4, characterized in that, The evaluation allocation optimization solution adopts a mixed integer programming model, which includes: Hard blocking constraint: for any short loop structure detected The number of review edges actually allocated in this short-loop structure is constrained to satisfy... ,in It is a short ring structure The set of candidate edges, Let the binary decision variables represent the edges Whether or not it is selected for the review and allocation scheme This represents the number of nodes in the short ring structure. Soft penalty mechanism: For any short loop structure detected Introducing a violation indicator variable And a penalty term is added to the objective function of the mixed integer programming model. ,in For the ring length and the penalty coefficient satisfying ; Historical reciprocal cycle constraint: For short cycles containing historical fixed edges Set the current set of edges to be decided With historical fixed edge set Separate, constrained If the right side is negative, then all current candidate edges will be forcibly disconnected; Load balancing: The load imbalance of reviewers is approximated by piecewise linearization by introducing auxiliary variables; Reviewer shortage fallback: When the mixed-integer programming model has no feasible solution, a gap slack variable is introduced. satisfy And a pre-set large constant penalty weight is applied to the gap relaxation variable in the objective function. If there are still problems after solving... If all of these fail, the approach will be to recursively relax the professional matching threshold, switch from hard blocking to soft penalty, increase the reviewer's task limit, or expand the candidate reviewer pool. If all of these fail, the approach will be to manually assign the reviewers.

6. The method according to claim 1, characterized in that, The style correction process based on robust statistics includes: A double correction is performed using the median absolute deviation: the first correction is to subtract the reviewer's median score from the scoring data and divide by the median absolute deviation of the reviewer's score to eliminate reviewer style bias; the second correction is to subtract the reviewee's median score from the data after the first correction and divide by the median absolute deviation of the reviewee's score to eliminate the reviewee's level bias. When calculating the normalized denominator, the first level of correction uses the median absolute deviation of the reviewer's historical scores as the normalization benchmark, and the second level of correction uses the median absolute deviation of the first level of correction values ​​of the reviewed object as the normalization benchmark. For reviewers whose historical scoring data volume is lower than the preset threshold, the statistical data is replaced step by step in the order of reviewer themselves, same position or department, and overall; for the reviewed object, the statistical data is replaced step by step in the order of same batch, same position or project, historical scores, and overall.

7. The method according to claim 1, characterized in that, The various exception modes include the following seven exception modes: High-resolution closed-loop anomaly mode: The set of short-loop structures actually formed after review, allocation, and optimization. The set of short-loop structures that are selected and retained in the short-loop detection By merging, we obtain the actual set of short cycles. ; For each short loop in the actual short loop set, the mean value of the correction deviation on the loop is calculated as the loop score intensity. When the loop score intensity exceeds the preset average threshold and the correction deviation of each edge on the loop exceeds the preset minimum deviation threshold, the loop is marked as a high-scoring closed loop by AND joint determination. Anomalous patterns of reciprocal clusters: The directed weighted time series scoring graph is symmetrically transformed into an undirected reciprocal weighted graph, and the Louvain community detection algorithm is used to identify reciprocal clusters; Cluster risk assessment anomaly pattern: Based on the identified mutually beneficial clusters, the difference between the average deviation of the score inside the cluster and the average deviation of the score outside the cluster is calculated, and the mutual edge density inside the cluster and the cluster mutuality rate are combined for joint judgment to identify high-risk clusters; Collusive suppression of abnormal patterns: Detects patterns where multiple reviewers give the same reviewer significantly lower scores than the reviewer's historical average, and calculates a suppression score. The suppression score is determined based on the deviation of the score and the number of reviewers involved in the suppression. Synchronous scoring anomaly mode: The kernel density estimation method is used to detect the synchronicity of the scoring behavior of multiple reviewers in the time dimension. The set of reviewers whose maximum point-by-point ratio of the kernel density estimation of the group's submission time to the baseline kernel density estimation exceeds the preset synchronization threshold is judged as synchronous scoring anomaly. Text similarity anomaly mode: SimHash local sensitive hashing, MinHash minimum hashing, and semantic text similarity calculation based on the pre-trained language model SBERT are used to calculate the text content of the review. The similarity obtained by the three methods is weighted and fused to obtain the comprehensive text similarity. When the average comprehensive text similarity within the group exceeds the preset similarity threshold, it is judged as a text similarity anomaly. Long-term mild reciprocal anomaly pattern: using exponentially weighted moving distance The reciprocal rating behavior among reviewers in multiple peer review tasks, where the score difference is consistently below a preset severe threshold, is considered a long-term mild reciprocal abnormality when the index-weighted moving distance exceeds a preset mild threshold in a consecutive preset number of peer review tasks.

8. The method according to claim 7, characterized in that, The fusion of the detection results of the various abnormal patterns includes: For each edge Calculate rule-based anomaly scores respectively Outlier scores based on continuous statistical models The maximum of the two values ​​is taken as the anomaly score for that edge. ,in , , , The three strong rules—high-score closed loop, reciprocal cluster, and joint suppression—have a single vote weight that is greater than the preset anomaly threshold. , , For the binary indicator function corresponding to strong rules; It is obtained by weighted summation of the detection results based on the continuous statistical model among the seven abnormal modes.

9. The method according to claim 1, characterized in that, It also includes abnormal edge handling steps: For edges in the directed weighted time series scoring graph where the abnormal comprehensive score exceeds a preset abnormal threshold, the scoring data of the abnormal edge is replaced with the scoring data of the normal edge through continuous interpolation. A hierarchical audit label is added to the replaced abnormal edge. The hierarchical audit label is marked as an approximate deletion adjustment, a replacement-based adjustment, or a weight reduction-based adjustment according to the interval where the abnormal comprehensive score is located. In the subsequent score fusion calculation, a trust weight is introduced for each edge. The trust weight is linearly negatively correlated with the anomaly score of that edge, i.e. .

10. The method according to claim 1, characterized in that, It also includes a counterfactual ranking distortion assessment, which comprises the following indicators, each of which is normalized and then weighted and fused through a convex combination to obtain a comprehensive ranking distortion: Kendall Distance: Calculate the proportion of inconsistent pairs in the ranking sequence of the reviewed object before and after removing the edge whose abnormal comprehensive score exceeds the preset abnormal threshold, relative to the total number of pairs; Top-K Jaccard distance: Ranked before and after removing outlier edges. The Jaccard distance of a set of reviewed objects, wherein the Jaccard distance is... Subtract the Jaccard similarity between the two sets; Boundary Crossing XOR Index: The number of reviewed objects that cross the preset ranking boundary before and after removing abnormal edges is counted and then normalized by dividing by the total number of reviewed objects. Maximum ranking change: Record the maximum change in ranking of a single reviewed object before and after removing the abnormal edge, and normalize it by dividing it by the total number of reviewed objects; Contribution rate of abnormal edges: Calculate the proportion of the average score adjustment of abnormal edges to the score range, and truncate it to the preset upper limit.

11. The method according to claim 1, characterized in that, The method also includes a comprehensive risk assessment: Based on the detection results of each side under the multi-dimensional anomaly pattern, a comprehensive risk score is calculated. The normalized comprehensive risk score ranges from [value range missing]. ; The comprehensive risk score is mapped to a five-level risk level, which is divided into low risk, low-medium risk, medium-high risk, high risk, and extremely high risk according to the risk score range. For each review relationship that is judged to be abnormal and whose risk level reaches the preset level, a structured evidence chain in JSON format is generated.

12. The method according to claim 1, characterized in that, It also includes a hierarchical feedback learning process, which includes: Edge-level softening layer: Only weights consecutive evidence are fused. Edge anomalies are comprehensively analyzed using the softening indicator function and the softening maximum value operator. The problem is transformed into a differentiable approximation. The cross-entropy loss is calculated using manually verified edge-level labels as supervision signals on the edge-level pseudo-probabilities, and then updated using stochastic gradient descent. During deployment and inference, the softened operator is reverted to hard computation; Calibration layer: The components of ranking distortion and mutual evaluation cheating risk are used as calculated feature vectors and input into the independent calibration layer. The task-level cross-entropy loss is calculated using manually verified task-level labels, and only the convex combination weights are updated. and scalar parameters The gradient is not backpropagated to the sorting, Top-K, XOR operation, and set counting stages; Threshold adaptive layer: For each non-differentiable threshold parameter, based on near-differentiable threshold parameters... The exponential moving average of the optimal F1 value on the second-reviewed sample is used for adaptive updating, where This is the preset number of sliding window samples; Offline hyperparameter optimization layer: Aggregates the risk weights from the discrete solution of mixed-integer programming into parameters. and regularization coefficient The Bandit algorithm, Bayesian optimization, or grid search are used for offline periodic tuning. The time decay coefficient and smoothing constant are fixed offline on the validation set before deployment.

13. A device for identifying mutual evaluation cheating based on short-loop blocking and time-series scoring graph anomaly detection, characterized in that, include: The data acquisition and normalization module is used to acquire mutual evaluation task data and organizational relationship data, and perform entity parsing and normalization processing to obtain a normalized entity dataset. The candidate review relationship graph construction module is used to construct a candidate review relationship graph based on the normalized entity dataset. The candidate review relationship graph includes multiple types of nodes and multiple types of edges. The risk weight calculation module is used to calculate the risk weight of the candidate review edges in the candidate review relationship graph to obtain the risk weight value of each candidate review edge. The short loop detection module is used to perform short loop detection on the candidate review relationship graph based on the risk weight value, and identify short loop structures containing a preset number of nodes; The review allocation optimization module is used to optimize the review allocation based on the results of the short loop detection, with the goal of minimizing the risk associated with the short loop structure, and to obtain the target review allocation scheme. The scoring graph construction module is used to collect multimodal scoring data generated by mutual evaluation after the mutual evaluation is completed, and construct a directed weighted time series scoring graph based on the target evaluation allocation scheme and the multimodal scoring data. The style correction module is used to perform style correction processing on the scoring data in the directed weighted time series scoring graph based on robust statistics, to eliminate the style bias of the reviewer and the level bias of the reviewed object, and to obtain the corrected scoring data. The abnormal pattern hierarchical detection module is used to perform hierarchical detection on a variety of preset abnormal patterns based on the corrected scoring data, and obtain the detection results of each abnormal pattern. An anomaly fusion module is used to fuse the detection results of the various anomaly modes to determine the comprehensive anomaly score of each edge in the directed weighted time series scoring graph.

14. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the method as claimed in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 12.

Images