Expert abnormal score screening method and system

Abstract

The invention relates to the technical field of expert score exception screening, and particularly discloses an expert exception score screening method and system, and the method comprises the steps: obtaining expert scores of a single bidding and tendering activity, constructing a score matrix, calculating the score mean value and standard deviation of each expert for scoring different evaluated enterprises, and a standardized score corresponding to each score, and arranging the standardized scores of the scores corresponding to the experts from small to large, forming a corresponding data set, calculating a first quantile, a second quantile and a quantile distance, setting an abnormal value judgment boundary according to the quantile distance, and judging the score corresponding to the standardized score exceeding the abnormal value judgment boundary as an abnormal score. According to the method, a statistical discrete model principle is adopted, and an abnormal value deviating from the overall scoring trend is identified according to the scoring deviation degree of the same expert for different bidders, so that scoring supervision is changed from experience judgment to data support, and a quantitative basis for checking and reviewing processes is provided for supervision departments.

Description

Technical Field

[0001] This invention relates to the field of expert rating anomaly screening technology, and in particular to a method and system for screening expert rating anomalies. Background Technology

[0002] In bidding processes, individual experts may score a particular bidder in a way that deviates significantly from the standard scoring criteria or differs substantially from the scores given by other experts, without a reasonable explanation. Such behavior clearly violates the principles of fairness and impartiality and may even involve the transfer of benefits or improper interference. However, currently, there is no unified standard in China for identifying "abnormally high or low scores." Furthermore, expert scoring is often influenced by personal preferences, scoring habits, and professional competence, and the review process is limited by time and conditions, making it difficult to identify such abnormal scoring behavior in a timely manner through visual observation alone.

[0003] To mitigate potential unfairness from manual scoring, some organizations have begun developing intelligent scoring models. These models are first trained using extensive data, and then automatically score bid documents based on the training results. While this approach helps reduce human bias, saves labor costs, and improves scoring efficiency, its effectiveness is highly dependent on the model structure, parameter settings, and training process. Furthermore, because bid evaluation often involves a large amount of information that is difficult to quantify, model scores can easily deviate from the actual scores that bid documents should receive, potentially reducing scoring accuracy and contradicting the principles of fairness and precision sought in bidding processes.

[0004] Therefore, based on the above issues, how to objectively and accurately identify abnormal scores from experts in bidding activities is an urgent problem to be solved. Summary of the Invention

[0005] To address the technical problems in the prior art, this invention provides a method and system for screening expert anomaly scores.

[0006] In a first aspect, the present invention provides a method for screening expert anomaly scores, comprising the following steps:

[0007] Obtain expert scores for a single bidding event and construct a scoring matrix;

[0008] Based on the scoring matrix, the mean and standard deviation of the scores given by each expert to different participating companies are calculated.

[0009] The standardized score corresponding to each score is calculated using the mean and the standard deviation.

[0010] The standardized scores for each expert are sorted from smallest to largest to form a corresponding dataset;

[0011] The first and second quantiles of the data set are calculated using a preset quantile order and preset first and second position parameters.

[0012] The difference between the first quantile and the second quantile is calculated as the quantile distance;

[0013] An outlier determination boundary is set based on the quantile, and scores corresponding to standardized scores that exceed the outlier determination boundary are determined as outlier scores.

[0014] Furthermore, setting outlier determination boundaries based on the quantile distance includes:

[0015] The outlier detection boundary is set as follows:

[0016] ;

[0017] The upper limit value of the boundary is: The lower boundary value is: ; The standardized score, This is the second quantile. This is the first quantile. The threshold coefficient is, and >0, The interval is the distance between the quantiles; , The number of experts; , The number of participating companies.

[0018] Furthermore, obtain expert scores for each bidding event and construct a scoring matrix, including:

[0019] Obtain information on experts, participating companies, and the scoring information provided by experts for the bids submitted by the participating companies;

[0020] Construct an expert information set based on the aforementioned expert information. And, based on the information of the participating companies, construct a set of company information. ;

[0021] A scoring matrix is ​​constructed based on the scoring information, the expert information set, and the enterprise information set. ;

[0022] in, ,element For experts For enterprises The ratings given.

[0023] Furthermore, based on the aforementioned scoring matrix, the mean and standard deviation of the scores given by each expert to different participating companies are calculated, including:

[0024] Computational experts The average score of all scores given to the participating companies is ;

[0025] Calculate the standard deviation, for ;

[0026] The step of calculating the standardized score corresponding to each rating using the mean and the standard deviation includes: .

[0027] Furthermore, the standardized scores for each expert's rating are sorted from smallest to largest, forming a corresponding dataset, including:

[0028] Summary Expert Standardized score corresponding to the rating To form the initial dataset ;

[0029] The standardized scores in the dataset are sorted and transformed into the dataset. ;in, .

[0030] Furthermore, by using a preset quantile order and preset first and second positional parameters, the first and second quantiles corresponding to the data set are calculated, including:

[0031] The first quantile corresponding to the data set is calculated in the following manner. Second quantile :

[0032] ;

[0033] In the formula, Position parameters; position parameters Include For the first position parameter, The second positional parameter is used to calculate the first quantile. Second quantile ; and The value is based on the quantile order. , preset position and The value is calculated from this; It means to take integer values, , as well as Belonging to the data set ; Represents an integer;

[0034] The calculation of the difference between the first quantile and the second quantile as the quantile distance includes: through... Calculate the quantile.

[0035] Furthermore, the preset quantile order is 4, and the first positional parameter The second position parameter That is, the first quantile The first quartile and the second quartile It is the third quartile.

[0036] Furthermore, the methods also include:

[0037] Calculate the relative degree to which the abnormal score exceeds the boundary, and generate a reminder message after matching the relative degree with a preset level.

[0038] Furthermore, calculating the relative degree to which the abnormal score exceeds the boundary includes:

[0039] Calculate the degree of abnormality for abnormally high and abnormally low values;

[0040] The method for calculating the degree of abnormality of the abnormally high value is as follows: ;

[0041] The method for calculating the degree of abnormality of the abnormally low value is as follows: ;

[0042] The preset levels include:

[0043] when At that time, the level is "abnormal". When the level is "significantly abnormal", At that time, the level was "extremely abnormal".

[0044] Secondly, this invention proposes an expert anomaly scoring and screening system, the system comprising an information acquisition module, a scoring calculation module, and an anomaly screening module, wherein:

[0045] The information acquisition module is connected to the scoring calculation module, and the information acquisition module is used to acquire the expert scores for a single bidding activity.

[0046] The scoring calculation module is connected to the information acquisition module and the anomaly screening module. The scoring calculation module is used to construct a scoring matrix based on the expert scores; and, based on the scoring matrix, calculate the mean and standard deviation of the scores given by each expert to different participating companies; calculate the standardized score corresponding to each score using the mean and standard deviation; arrange the standardized scores of each expert's corresponding score in ascending order to form a corresponding dataset; calculate the first quantile and second quantile corresponding to the dataset using a preset quantile order and preset first and second position parameters; and calculate the difference between the first quantile and the second quantile as the quantile distance.

[0047] The anomaly screening module is connected to the scoring calculation module. The anomaly screening module is used to set an outlier determination boundary based on the percentile and to determine the scores corresponding to the standardized scores that exceed the outlier determination boundary as outlier scores.

[0048] This invention discloses a method and system for screening abnormal expert scores. It avoids horizontal comparisons between expert scores and the need to acquire and analyze historical scores from previous bidding activities. Instead, it calculates and analyzes the scores given by each expert individually to screen for potentially abnormal scores. The screening process involves minimal data and is easy to analyze. Furthermore, the specific screening method employs the principle of a statistical discrete model. By analyzing the degree of deviation in the same expert's scores for different bidders, it identifies outliers that deviate from the overall scoring trend, distinguishing between "reasonable differences" and "subjective bias." This eliminates the influence of different experts' scoring habits and personal preferences, shifting score supervision from "experience-based judgment" to "data-driven support." This provides regulatory authorities with quantitative evidence for the review process, compels experts to perform their duties rigorously, and ultimately enhances the persuasiveness and acceptance of bidding results. This provides a favorable guarantee for achieving fairness and impartiality in the bidding field and improving regulatory level and efficiency. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the steps of an expert anomaly scoring screening method according to an embodiment of the present invention.

[0051] Figure 2 This is a structural diagram of an expert anomaly scoring and screening system according to an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0053] An expert anomaly scoring screening method according to an embodiment of the present invention, such as Figure 1 As shown, it includes the following steps:

[0054] Step S10: Obtain expert scores for a single bidding event and construct a scoring matrix.

[0055] Step S20: Based on the scoring matrix, calculate the mean and standard deviation of the scores given by each expert to different participating companies.

[0056] Step S30: Calculate the standardized score for each rating using the mean and standard deviation.

[0057] Step S40: Sort the standardized scores of each expert's corresponding rating from smallest to largest and form the corresponding dataset.

[0058] Step S50: Calculate the first quantile and the second quantile corresponding to the data set using the preset quantile order and the preset first position parameter and second position parameter.

[0059] Step S60: Calculate the difference between the first quantile and the second quantile as the quantile distance.

[0060] Step S70: Set outlier determination boundaries based on quantiles, and determine the scores corresponding to standardized scores that exceed the outlier determination boundaries as outlier scores.

[0061] The expert anomaly scoring screening method of this invention does not perform horizontal comparative analysis of scores between experts, nor does it require obtaining and analyzing the historical scores of the experts in previous bidding activities. Instead, it only calculates and analyzes the scores given by each expert, thereby screening out scores that may be abnormal. The screening process involves a small amount of data and is easy to analyze. In addition, the specific screening method adopts the principle of statistical discrete model. By analyzing the degree of deviation of the same expert's scores for different bidders, it identifies outliers that deviate from the overall scoring trend, distinguishes between "reasonable differences" and "subjective bias," and eliminates the influence of different experts' scoring habits and personal preferences. This shifts the scoring supervision from "experience judgment" to "data support," providing regulatory authorities with quantitative basis for the verification and review process, and also forcing experts to perform their duties rigorously. Ultimately, it enhances the persuasiveness and acceptance of bidding results, and provides a favorable guarantee for achieving fairness and impartiality in the bidding field and improving the level and efficiency of supervision.

[0062] Specifically, based on the above embodiments, step S70, which sets out anomaly determination boundaries according to quantile distance, specifically includes:

[0063] The outlier detection boundary is set as follows:

[0064] ;

[0065] The upper limit value of the boundary is: The lower boundary value is: ; For standardized scores, It is the second quantile. The first quantile, The threshold coefficient is, and >0, The interval is the quantile. , The number of experts; , The number of participating companies.

[0066] The embodiments of the present invention do not limit the threshold coefficient. The specific value can be determined by those skilled in the art based on the specific circumstances during implementation. The optimal value is set by [the setter / factor]. The value is .

[0067] Specifically, in step S10, expert scores for a single bidding activity are obtained, and a scoring matrix is ​​constructed, including:

[0068] Obtain information on experts, participating companies, and the scoring information provided by experts for the bids submitted by the participating companies;

[0069] Constructing an expert information set based on expert information And constructing a set of enterprise information based on the information of participating enterprises. ;

[0070] A scoring matrix is ​​constructed based on scoring information, expert information sets, and enterprise information sets. ;

[0071] in, ,element For experts For enterprises The ratings given.

[0072] Specifically, in step S20, based on the scoring matrix, the mean and standard deviation of the scores given by each expert to different participating companies are calculated, including:

[0073] Computational experts The average score of all scores given to the participating companies is The average score represents the average level of the judges' scores.

[0074] Calculate the standard deviation, for The standard deviation represents the dispersion of the judges' scores.

[0075] Specifically, in step S30, the standardized score corresponding to each rating is calculated using the mean and standard deviation, including: Standardized scores make it possible to compare scores from different judges.

[0076] The embodiments of the present invention calculate and analyze all the scores given by each expert to obtain the mean, standard deviation and standardized score of each score for different experts.

[0077] Specifically, in step S40, the standardized scores for each expert's corresponding rating are sorted from smallest to largest to form a corresponding dataset, including:

[0078] Summary Expert Standardized score corresponding to the rating To form the initial dataset ;

[0079] Sort the standardized scores in the dataset and convert them into a data set. ;in, .

[0080] Specifically, in step S50, the first and second quantiles corresponding to the data set are calculated using a preset quantile order and preset first and second position parameters, including:

[0081] Calculate the first quantile of the data set as follows: Second quantile :

[0082] ;

[0083] In the formula, Position parameters; position parameters Include For the first position parameter, The second positional parameter is used to calculate the first quantile. Second quantile ; and The value depends on the quantile order. , preset position and The value is calculated from this; It means to take integer values, , as well as Belongs to data set ; Represents an integer;

[0084] Specifically, in step S60, the difference between the first quantile and the second quantile is calculated as the quantile interval, including: through... Calculate the percentile.

[0085] Most preferably, in the above embodiments, the preset quantile order is 4, and the first position parameter... Second positional parameter That is, the first quantile The first quartile and the second quartile It is the third quartile.

[0086] Specifically, the method in this embodiment of the invention further includes:

[0087] Step S80: Calculate the relative degree of the abnormal score exceeding the boundary, and generate a reminder message after matching the relative degree with the preset level.

[0088] The calculation of the relative degree to which the abnormal score exceeds the boundary includes:

[0089] Calculate the degree of abnormality for abnormally high and abnormally low values;

[0090] The method for calculating the degree of abnormality of abnormally high values ​​is as follows: ;

[0091] The method for calculating the degree of abnormality of abnormally low values ​​is as follows: ;

[0092] Preset levels include: when At that time, the level is "abnormal". When the level is "significantly abnormal", At that time, the level is "extremely abnormal". The preset level in this embodiment can also be adjusted appropriately according to the actual situation. The boundary values ​​of different levels listed in this embodiment are only examples. In conjunction with the above implementation methods, the present invention proposes a specific implementation scheme. It should be noted that this implementation scheme is only an example and does not limit the scope of protection of the present invention. Specifically:

[0093] Suppose that 30 companies participate in a bidding process, and 5 experts are assigned to evaluate the bids. This embodiment first obtains expert information, participating company information, and the experts' scoring information for the participating companies' bids. Then, it constructs an expert information set based on the expert information. ,Right now And constructing a set of enterprise information based on the information of participating enterprises. ,Right now .

[0094] Then, a scoring matrix is ​​constructed based on the scoring information, expert information set, and enterprise information set. ;

[0095] in, .

[0096] To the second expert Taking the abnormal scoring as an example for screening, the calculation expert The mean and standard deviation of the scores given to the 30 participating companies are as follows: ,as well as .

[0097] Calculate the standardized score for each rating: A total of 30 standardized scores were obtained, which are denoted as the initial dataset. .

[0098] Sort the standardized scores in the dataset and convert them into a data set. ;in, .

[0099] Set the quantile order to 4, the first quantile Second quantile The first and third quartiles were selected as follows:

[0100] First, calculate the first position parameter. , ,in The numerator 1 represents the first quartile; the second positional parameter , ,in The numerator 3 represents the third quartile; and The denominator 4 represents the order of the quantile as 4.

[0101] Calculate the first quartile = Then, the specific numerical value is calculated.

[0102] Calculate the third quartile = Then, the specific numerical value is calculated.

[0103] The interval can also be determined by... The specific value is calculated.

[0104] by As an example, the upper bound value is: The lower boundary value is: The upper and lower bounds of the boundary can also be calculated to obtain specific values. Then, the standardized scores that exceed the boundary are determined by the outlier determination boundary, and the scores corresponding to these standardized scores are determined to be outlier scores.

[0105] After identifying abnormal scores, it is necessary to determine the degree of abnormality of each score, which is measured in step S80. Abnormal scores exceeding the upper limit of the boundary are called abnormally high values, and those exceeding the lower limit are called abnormally low values. The degree of abnormality for abnormally high and abnormally low values ​​(i.e., the relative degree to which the abnormal score exceeds the boundary) is calculated, and then matched with a preset level to generate an alert message.

[0106] The following analysis and explanation are based on specific scoring values:

[0107] Taking the scoring of real project A as an example, the scoring details for this project are shown in Table 1:

[0108] Table 1

[0109]

[0110] The calculation and analysis conducted through the embodiments of this invention revealed that the scores given to bidder A1 by judges A and B were abnormally high. It is particularly noteworthy that judge B's score was neither the highest nor the lowest, and its deviation from the average score of all judges was not significant. Traditional methods of identifying highest and lowest scores, or methods based on a certain percentage above the average, would not have been sufficient to determine that the judge's score was abnormal. Therefore, the screening method of this invention is more convincing.

[0111] Table 2 below compares the ranking results calculated from the sum of the judges' scores:

[0112] Table 2

[0113]

[0114] Comparing the rankings based on the average scores of the five experts, the average scores after removing the highest and lowest scores, and the average scores after removing the highest score from judge A, A1 ranked first in all three cases. However, after removing the scores from the two experts who gave abnormally high scores and taking the average, the ranking changed, with bidder A2 ranking first. Considering the scores from all the experts, the rankings of bidder A2 among the judges were 4th, 2nd, 2nd, 3rd, and 4th, indicating that the five judges unanimously agreed that bidder A2's technical score was superior among the 34 bidders, and the ranking after removing abnormally high scores is more reasonable.

[0115] Taking the scoring of real project B as an example, in order to further verify the effectiveness of the screening method of the present invention, the scores of judges A and E for A1 were artificially increased based on the real scores. The project scores after processing are detailed in Table 3. According to the scores in this table, the expert with the highest score is not necessarily the expert with an abnormally high score.

[0116] Table 3

[0117]

[0118] In the table above, judge C gives the highest score and judge B gives the lowest score. A horizontal comparison shows that judges A and E both gave normal scores. However, the screening method of this invention calculates that the scores of judges A and E in the table are abnormal scores. Although these two judges' scores are neither the highest nor the lowest, their scores have actually deviated significantly from normal values.

[0119] like Figure 2 As shown, this is an expert anomaly scoring and screening system according to an embodiment of the present invention. The system includes an information acquisition module 101, a scoring calculation module 102, and an anomaly screening module 103, wherein:

[0120] The information acquisition module 101 is connected to the scoring calculation module 102. The information acquisition module 101 is used to acquire the expert scores of a single bidding activity.

[0121] The scoring calculation module 102 is connected to the information acquisition module 101 and the anomaly screening module 103. The scoring calculation module 102 is used to construct a scoring matrix based on expert scores; and, based on the scoring matrix, calculate the mean and standard deviation of the scores given by each expert to different participating companies; calculate the standardized score corresponding to each score through the mean and standard deviation; arrange the standardized scores of each expert's corresponding score in ascending order to form a corresponding data set; calculate the first quantile and the second quantile corresponding to the data set through a preset quantile order and preset first position parameters and second position parameters; and calculate the difference between the first quantile and the second quantile as the quantile interval.

[0122] The anomaly screening module 103 is connected to the scoring calculation module 102. The anomaly screening module 103 is used to set the outlier judgment boundary according to the percentile and to judge the scores corresponding to the standardized scores that exceed the outlier judgment boundary as outlier scores.

[0123] The expert anomaly scoring and screening system in this embodiment is implemented in the same way as the aforementioned expert anomaly scoring and screening method, so it will not be explained again here.

[0124] The main body implementing this invention can be a computer equipped with the corresponding computer program. After judging abnormal scores through the above calculation method, it generates screening results or abnormality levels and displays warnings. Managers can use these results to investigate and question relevant experts to further determine whether there is genuine unfairness in the scoring. Furthermore, an online scoring function is added to this system. After logging into the system, experts score the tender documents sequentially, submitting their scores within the system. The system then screens the expert's scores, sending warnings to the backend if abnormal scores are found. This process is unaffected by the scoring progress of other experts, offering high flexibility.

[0125] The present invention has been further described above with reference to specific embodiments. However, it should be understood that the specific description herein should not be construed as limiting the nature and scope of the present invention. Various modifications made to the above embodiments by those skilled in the art after reading this specification are all within the scope of protection of the present invention.

Claims

1. A method for screening expert anomaly scores, characterized in that, Includes the following steps: Obtain expert scores for a single bidding event and construct a scoring matrix; Based on the scoring matrix, the mean and standard deviation of the scores given by each expert to different participating companies are calculated. The standardized score corresponding to each score is calculated using the mean and the standard deviation. The standardized scores for each expert are sorted from smallest to largest to form a corresponding dataset; The first and second quantiles of the data set are calculated using a preset quantile order and preset first and second position parameters. The difference between the first quantile and the second quantile is calculated as the quantile distance; An outlier determination boundary is set based on the quantile, and scores corresponding to standardized scores that exceed the outlier determination boundary are determined as outlier scores.

2. The expert anomaly scoring screening method as described in claim 1, characterized in that, The outlier determination boundary is set based on the quantile distance, including: The outlier detection boundary is set as follows: ； The upper limit value of the boundary is: The lower boundary value is: ; The standardized score, This is the second quantile. This is the first quantile. The threshold coefficient is, and >0, The interval is the distance between the quantiles; , The number of experts; , The number of participating companies.

3. The expert anomaly scoring screening method as described in claim 2, characterized in that, Obtain expert scores for a single bidding event and construct a scoring matrix, including: Obtain information on experts, participating companies, and the scoring information provided by experts for the bids submitted by the participating companies; Construct an expert information set based on the aforementioned expert information. And, based on the information of the participating companies, construct a set of company information. ; A scoring matrix is ​​constructed based on the scoring information, the expert information set, and the enterprise information set. ; in, ,element For experts For enterprises The ratings given.

4. The expert anomaly scoring screening method as described in claim 3, characterized in that, Based on the scoring matrix, the mean and standard deviation of the scores given by each expert to different participating companies are calculated, including: Computational experts The average score of all scores given to the participating companies is ; Calculate the standard deviation, for ; The step of calculating the standardized score corresponding to each rating using the mean and the standard deviation includes: .

5. The expert anomaly scoring screening method as described in claim 4, characterized in that, The standardized scores for each expert's rating are sorted from smallest to largest to form a corresponding dataset, including: Summary Expert Standardized score corresponding to the rating To form the initial dataset ; The standardized scores in the dataset are sorted and transformed into the dataset. ;in, .

6. The expert anomaly scoring screening method as described in claim 5, characterized in that, The first and second quantiles of the data set are calculated using a preset quantile order and preset first and second position parameters, including: The first quantile corresponding to the data set is calculated in the following manner. Second quantile : ； In the formula, Position parameters; position parameters Include For the first position parameter, The second positional parameter is used to calculate the first quantile. Second quantile ; and The value is based on the quantile order. , preset position and The value is calculated; It means to take integer values, , as well as Belonging to the data set ; Represents an integer; The calculation of the difference between the first quantile and the second quantile as the quantile distance includes: through... Calculate the quantile.

7. The expert anomaly scoring screening method as described in claim 6, characterized in that, The preset quantile order is 4, and the first positional parameter The second position parameter That is, the first quantile The first quartile and the second quartile It is the third quartile.

8. The expert anomaly scoring screening method as described in claim 6, characterized in that, Also includes: Calculate the relative degree to which the abnormal score exceeds the boundary, and generate a reminder message after matching the relative degree with a preset level.

9. The expert anomaly scoring screening method as described in claim 8, characterized in that, The calculation of the relative degree to which the abnormal score exceeds the boundary includes: Calculate the degree of abnormality for abnormally high and abnormally low values; The method for calculating the degree of abnormality of the abnormally high value is as follows: ; The method for calculating the degree of abnormality of the abnormally low value is as follows: ; The preset levels include: when At that time, the level is "abnormal". At that time, the level was "significantly abnormal". At that time, the level was "extremely abnormal".

10. An expert anomaly scoring and screening system, characterized in that, The system includes an information acquisition module, a scoring calculation module, and an anomaly filtering module, wherein: The information acquisition module is connected to the scoring calculation module, and the information acquisition module is used to acquire the expert scores for a single bidding activity. The scoring calculation module is connected to the information acquisition module and the anomaly screening module. The scoring calculation module is used to construct a scoring matrix based on the expert scores; and, based on the scoring matrix, calculate the mean and standard deviation of the scores given by each expert to different participating companies; calculate the standardized score corresponding to each score using the mean and standard deviation; arrange the standardized scores of each expert's corresponding score in ascending order to form a corresponding dataset; calculate the first quantile and second quantile corresponding to the dataset using a preset quantile order and preset first and second position parameters; and calculate the difference between the first quantile and the second quantile as the quantile distance. The anomaly screening module is connected to the scoring calculation module. The anomaly screening module is used to set an anomaly judgment boundary based on the percentile and to judge the scores corresponding to the standardized scores that exceed the anomaly judgment boundary as anomaly scores.

Images