Method, device and equipment for evaluating score effectiveness, medium and program product

Abstract

The invention provides a score validity evaluation method and device, equipment, a medium and a program product. The method comprises the following steps: calculating a score stability parameter, a score deviation parameter, a score tendency parameter, an inter-scorer consistency parameter and an intra-scorer consistency parameter of each review person according to the score of each project by each review person; determining a score volatility index, a score tendency index and a score consistency index of the reviewer according to the parameters; and determining a score effectiveness index of the reviewer according to the determined three indexes. According to the scheme, the score validity index of the reviewer is determined through the score volatility index, the score tendency index and the score consistency index of the reviewer, so that the evaluation accuracy of the score validity is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular, to a method, device, equipment, medium, and program product for evaluating the validity of scores. Background Art

[0002] With the continuous development of technology and society, there are more and more scientific research projects, construction projects, work projects, etc. carried out by various companies and units. In order to determine the quality of project completion, it is usually necessary for reviewers to score the projects. The reviewers score based on their own experience, and there will inevitably be problems with poor score validity. Therefore, it is necessary to evaluate the score validity to eliminate scores with poor validity.

[0003] In the prior art, to determine the validity of scores, it is usually to calculate the consistency index of the reviewers' scores for the projects, and use the consistency index as the standard for evaluating validity. The scores with better consistency have better validity, and the scores with poorer consistency have poorer validity.

[0004] In summary, the existing method for evaluating score validity only uses the consistency of scores as the evaluation standard, resulting in low accuracy of validity evaluation. Summary of the Invention

[0005] The method, device, equipment, medium, and program product for evaluating score validity provided by the embodiments of the present application are used to solve the problem that in the prior art, only the consistency of scores is used as the evaluation standard, resulting in low accuracy of validity evaluation.

[0006] In a first aspect, an embodiment of the present application provides a method for evaluating score validity, including:

[0007] According to the scores of each reviewer for each project obtained, calculate the score stability parameter, score deviation parameter, score tendency parameter, inter-rater consistency parameter, and intra-rater consistency parameter of each reviewer;

[0008] For each reviewer, determine the score volatility index of the reviewer according to the score stability parameter and score deviation parameter of the reviewer;

[0009] For each reviewer, determine the score tendency index of the reviewer according to the score tendency parameter of the reviewer;

[0010] According to the inter-rater consistency parameter and intra-rater consistency parameter of each reviewer, calculate the score consistency index of each reviewer;

[0011] For each of the reviewers, determine the scoring effectiveness index of the reviewer according to the scoring volatility index, scoring tendency index, and scoring consistency index of the reviewer.

[0012] In a possible implementation, calculating the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter, and intra-rater consistency parameter of each reviewer according to the scores given by each reviewer to each project includes:

[0013] Determine the average score, median score, and standard deviation of the scores of each reviewer, as well as the average score and standard deviation of the scores of each project, based on the scores given by each reviewer to each project;

[0014] Calculate the scoring stability parameter of each reviewer according to the scores given by each reviewer to each project, the average score of each reviewer, and a first preset formula;

[0015] Determine the scoring deviation parameter of each reviewer according to the scores given by each reviewer to each project and the standard deviation of the scores of each project;

[0016] Calculate the scoring tendency parameter of each reviewer according to the average score, median score, and standard deviation of each reviewer, and a second preset formula;

[0017] Calculate the inter-rater consistency parameter of each reviewer according to the average score of each reviewer, the average score of each project, and a third preset formula;

[0018] Calculate the intra-rater consistency parameter of each reviewer according to the scores given by each reviewer to each project, the average score of each reviewer, the average score of each project, and a fourth preset formula.

[0019] In a possible implementation, the first preset formula is: ;

[0020] The second preset formula is: ;

[0021] The third preset formula is: ;

[0022] The fourth preset formula is: ;

[0023] Wherein, represents the scoring stability parameter of the i-th reviewer, represents the scoring tendency parameter of the i-th reviewer, represents the inter-rater consistency parameter of the i-th reviewer, represents the intra-rater consistency parameter of the i-th reviewer, represents the score of the i-th reviewer for the j-th project, represents the average score of the i-th reviewer, and n represents the number of projects, represents the median score of the i-th reviewer, represents the standard deviation of the scores of the i-th reviewer, represents the average score of the j-th project.

[0024] In a possible implementation manner, determining the scoring deviation parameter of each reviewer according to the scores of each reviewer for each project and the standard deviation of the scores of each project includes:

[0025] Determining the confidence interval of each project according to the scores of each reviewer for each project and the standard deviation of the scores of each project;

[0026] For each reviewer, determining the scoring deviation parameter of the reviewer according to the scores of the reviewer for each project and the confidence interval of each project.

[0027] In a possible implementation manner, calculating the scoring consistency index of each reviewer according to the inter-rater consistency parameter and intra-rater consistency parameter of each reviewer includes:

[0028] Calculating the scoring consistency index of each reviewer according to the inter-rater consistency parameter and intra-rater consistency parameter of each reviewer and a fifth preset formula;

[0029] The fifth preset formula is: ;

[0030] Wherein, represents the scoring consistency index of the i-th reviewer, represents the inter-rater consistency parameter of the i-th reviewer, represents the intra-rater consistency parameter of the i-th reviewer.

[0031] In a possible implementation manner, determining the scoring effectiveness index of the reviewer according to the scoring volatility index, scoring tendency index and scoring consistency index of the reviewer includes:

[0032] If the scoring volatility index is less than or equal to a first preset threshold and the absolute value of the scoring tendency index is equal to a second preset threshold, then use the scoring consistency index as the scoring validity index;

[0033] If the scoring volatility index is greater than the first preset threshold, or the absolute value of the scoring tendency index is not equal to the second preset threshold, then calculate the scoring validity index based on the scoring volatility index, the scoring tendency index, and the scoring consistency index.

[0034] In a second aspect, an embodiment of the present application provides an evaluation device for scoring validity, including:

[0035] A processing module, configured to:

[0036] According to the scores of each reviewer for each project obtained, calculate the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter, and intra-rater consistency parameter of each reviewer;

[0037] For each reviewer, determine the scoring volatility index of the reviewer according to the scoring stability parameter and scoring deviation parameter of the reviewer;

[0038] For each reviewer, determine the scoring tendency index of the reviewer according to the scoring tendency parameter of the reviewer;

[0039] Calculate the scoring consistency index of each reviewer according to the inter-rater consistency parameter and intra-rater consistency parameter of each reviewer;

[0040] An evaluation module, configured to determine the scoring validity index of each reviewer according to the scoring volatility index, scoring tendency index, and scoring consistency index of the reviewer.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0042] A processor, a memory, and a communication interface;

[0043] The memory is used to store executable instructions of the processor;

[0044] Wherein, the processor is configured to execute the evaluation method for scoring validity according to any one of the first aspects by executing the executable instructions.

[0045] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the evaluation method for scoring validity according to any one of the first aspects is implemented.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which is used to implement the evaluation method for scoring effectiveness according to any one of the first aspects when executed by a processor.

[0047] The evaluation method, device, equipment, medium and program product for scoring effectiveness provided by the embodiments of the present application calculate the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter and intra-rater consistency parameter of each reviewer according to the scores of each reviewer for each item, and then determine the scoring volatility index, scoring tendency index and scoring consistency index of the reviewer according to these parameters; furthermore, the scoring effectiveness index of the reviewer is determined according to the three determined indexes. This solution determines the scoring effectiveness index of the reviewer through the scoring volatility index, scoring tendency index and scoring consistency index of the reviewer, effectively improving the evaluation accuracy of the scoring effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing the embodiments in line with the present application, and are used together with the specification to explain the principles of the present application.

[0049] Figure 1 It is a schematic flowchart of the first embodiment of the evaluation method for scoring effectiveness provided by the present application;

[0050] Figure 2 It is a schematic flowchart of the second embodiment of the evaluation method for scoring effectiveness provided by the present application;

[0051] Figure 3 It is a schematic structural diagram of the embodiment of the evaluation device for scoring effectiveness provided by the present application;

[0052] Figure 4 It is a schematic structural diagram of an electronic device provided by the present application.

[0053] Through the above accompanying drawings, the clear embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and text descriptions are not intended to limit the scope of the concept of the present application in any way, but to explain the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0055] The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0056] With the continuous development of technology and society, companies and units are carrying out more and more projects, etc. In order to determine the quality of project completion, it is usually necessary for reviewers to score the projects. Reviewers score based on their own experience, and there will inevitably be problems with poor scoring effectiveness. Therefore, it is necessary to evaluate the scoring effectiveness to eliminate scores with poor effectiveness.

[0057] In the prior art, to determine the effectiveness of scoring, it is usually to calculate the consistency index of the reviewers' scores for the projects and use the consistency index as the standard for evaluating effectiveness. The scores with better consistency have better scoring effectiveness, and the scores with poorer consistency have poorer scoring effectiveness. Since only the consistency of the scores is considered, it will lead to the problem of low evaluation accuracy.

[0058] In view of the problems existing in the prior art, the inventors found in the process of researching the evaluation method of scoring effectiveness that in addition to the scoring consistency, the scoring tendency and scoring volatility of reviewers will also affect the scoring effectiveness. Therefore, after determining the scoring volatility index, scoring tendency index and scoring consistency index of reviewers based on the scores of each reviewer for each project, the scoring effectiveness index of reviewers can be determined according to the three indexes, improving the evaluation accuracy. Based on the above inventive concept, the evaluation scheme for scoring effectiveness in the present application is designed.

[0059] The execution subject of the evaluation method for scoring validity in this application can be a computer, or a server, a terminal device, etc. This application does not limit it. Hereinafter, a computer will be taken as an example for illustration.

[0060] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0061] Hereinafter, an application scenario of the evaluation method for scoring validity provided by this application will be illustrated by examples.

[0062] Exemplarily, in this application scenario, there are multiple scientific research projects that need to be scored. After multiple reviewers score each scientific research project respectively, the staff input the scores into the computer.

[0063] After the computer obtains the scores of each reviewer for each project, based on these scores, it calculates the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter, and intra-rater consistency parameter of each reviewer;

[0064] Furthermore, for each reviewer, according to the scoring stability parameter and scoring deviation parameter of the reviewer, the scoring volatility index of the reviewer is determined; according to the scoring tendency parameter of the reviewer, the scoring tendency index of the reviewer is determined; according to the inter-rater consistency parameter and intra-rater consistency parameter of the reviewer, the scoring consistency index of the reviewer is calculated.

[0065] Finally, for the reviewer, according to the scoring volatility index, scoring tendency index, and scoring consistency index of the reviewer, the scoring validity index of the reviewer is determined.

[0066] The computer can display the scoring validity index of each reviewer. The staff removes the scores of the reviewers with lower scoring validity indexes, and the remaining scores have higher validity. The remaining scores are used to determine the quality of each scientific research project.

[0067] It should be noted that the above scenario is only an example of an application scenario provided by the embodiments of this application. The embodiments of this application do not limit the actual forms of various devices included in this scenario. In the specific application of the solution, it can be set according to actual needs.

[0068] Next, the technical solutions of the present application will be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0069] Figure 1 FIG. 4 is a schematic flowchart of the first embodiment of the evaluation method for the validity of scoring provided by the present application. In the embodiment of the present application, after the computer calculates the scoring volatility index, scoring tendency index, and scoring consistency index of each reviewer based on the scores of each reviewer for each item, the situation of evaluating the validity of the scores based on the three indexes is described. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware. As Figure 1 shown, the evaluation method for the validity of scoring specifically includes the following steps:

[0070] S101: Calculate the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter, and intra-rater consistency parameter of each reviewer according to the scores of each reviewer for each item obtained.

[0071] In this step, in order to determine the validity of the scores of the reviewers for the items, after the computer obtains the scores of each reviewer for each item, it calculates the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter, and intra-rater consistency parameter of each reviewer according to these scores.

[0072] The scores of each reviewer for each item can form a scoring matrix , and the element in the scoring matrix represents the score of the i-th reviewer for the j-th item, m represents the number of reviewers, and n represents the number of items.

[0073] The scoring stability parameter is used to characterize the stability of the reviewers' scores. The larger the scoring stability parameter, the worse the stability of the reviewers' scores.

[0074] The scoring deviation parameter is used to characterize the deviation degree of the reviewers' scores. The larger the scoring deviation parameter, the smaller the deviation degree of the reviewers' scores.

[0075] The scoring tendency parameter is used to characterize the tendency degree of the reviewers' scores. When the scoring tendency parameter is positive, it means that the reviewer tends to give high scores; when the scoring tendency parameter is negative, it means that the reviewer tends to give low scores; the larger the scoring tendency parameter, the greater the tendency of the reviewer to give high scores; the smaller the scoring tendency parameter, the greater the tendency of the reviewer to give low scores; the closer the scoring tendency parameter is to 0, the smaller the tendency of the reviewer to give high scores and low scores.

[0076] The inter-rater consistency parameter of this reviewer is used to characterize the consistency between the ratings of this reviewer and those of other reviewers.

[0077] The intra-rater consistency parameter of this reviewer is used to characterize the consistency of the ratings of this reviewer.

[0078] S102: For each reviewer, determine the rating volatility index of this reviewer according to the rating stability parameter and the rating deviation parameter of this reviewer.

[0079] In this step, after the computer determines the rating stability parameter, the rating deviation parameter, the rating tendency parameter, the inter-rater consistency parameter, and the intra-rater consistency parameter of each reviewer, for each reviewer, the rating volatility index of this reviewer can be determined according to the rating stability parameter and the rating deviation parameter of this reviewer.

[0080] The rating volatility index is used to characterize the degree of volatility of the reviewer's ratings. The larger the rating volatility index, the greater the degree of volatility of the reviewer's ratings.

[0081] Specifically, if the rating stability parameter is greater than the preset stability threshold and the rating deviation parameter is greater than the preset deviation threshold, then determine that the rating volatility index of this reviewer is the first preset volatility value.

[0082] If the rating stability parameter is less than or equal to the preset stability threshold and the rating deviation parameter is greater than the preset deviation threshold, then determine that the rating volatility index of this reviewer is the second preset volatility value.

[0083] If the rating stability parameter is greater than the preset stability threshold and the rating deviation parameter is less than or equal to the preset deviation threshold, then determine that the rating volatility index of this reviewer is the third preset volatility value.

[0084] If the rating stability parameter is less than or equal to the preset stability threshold and the rating deviation parameter is less than or equal to the preset deviation threshold, then determine that the rating volatility index of this reviewer is the fourth preset volatility value.

[0085] It should be noted that the first preset volatility value, the second preset volatility value, the third preset volatility value, and the fourth preset volatility value are in a decreasing relationship. The first preset volatility value can be 1, 0.97, 0.95, etc., the second preset volatility value can be 0.85, 0.8, 0.75, etc., the third preset volatility value can be 0.55, 0.5, 0.45, etc., and the fourth preset volatility value can be 0.1, 0.05, 0, etc. The embodiments of the present application do not limit the first preset volatility value, the second preset volatility value, the third preset volatility value, and the fourth preset volatility value, and they can be determined according to the actual situation.

[0086] It should be noted that the preset stability threshold can be 15%, 20%, 25%, etc., and the preset deviation threshold can be , , , etc. Here, n represents the number of items. The embodiments of the present application do not limit the preset stability threshold and the preset deviation threshold, which can be determined according to the actual situation.

[0087] S103: For each reviewer, determine the scoring tendency index of this reviewer according to the scoring tendency parameter of this reviewer.

[0088] In this step, after the computer determines the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter, and intra-rater consistency parameter of each reviewer, for each reviewer, the scoring tendency index of this reviewer can be determined according to the scoring tendency parameter of this reviewer.

[0089] The scoring tendency index is used to characterize the tendency degree of a reviewer's scoring. The larger the absolute value of the scoring tendency index, the greater the tendency degree of the reviewer to give high scores or low scores.

[0090] Specifically, if the scoring tendency index is greater than or equal to the first preset tendency threshold, determine the scoring tendency index of this reviewer as the first preset tendency value.

[0091] If the scoring tendency index is less than the first preset tendency threshold and greater than the second preset tendency threshold, determine the scoring tendency index of this reviewer as the second preset tendency value.

[0092] If the scoring tendency index is less than or equal to the second preset tendency threshold, determine the scoring tendency index of this reviewer as the third preset tendency value.

[0093] It should be noted that the first preset tendency threshold is greater than the second preset tendency threshold. The first preset tendency threshold can be 0.8, 1, 1.2, etc.; the second preset tendency threshold can be -0.8, -1, -1.2, etc.; the embodiments of the present application do not limit the first preset tendency threshold and the second preset tendency threshold, which can be determined according to the actual situation.

[0094] It should be noted that the first preset tendency value, the second preset tendency value, and the third preset tendency value are in a decreasing relationship. The first preset tendency value can be 0.8, 1, 1.2, etc.; the first preset tendency value can be 0.2, 0, -0.2, etc.; the third preset tendency value can be -0.8, -1, -1.2, etc.; the embodiments of the present application do not limit the first preset tendency value, the second preset tendency value, and the third preset tendency value, which can be determined according to the actual situation.

[0095] S104: Calculate the scoring consistency index for each reviewer based on the inter-rater consistency parameter and the intra-rater consistency parameter of each reviewer.

[0096] In this step, after the computer determines the scoring stability parameter, the scoring deviation parameter, the scoring tendency parameter, the inter-rater consistency parameter, and the intra-rater consistency parameter of each reviewer, the scoring consistency index of each reviewer can be calculated based on the inter-rater consistency parameter and the intra-rater consistency parameter of each reviewer.

[0097] The scoring consistency index is used to characterize the consistency degree of the reviewers' scores, and also represents the rationality degree and objectivity degree of the scores. The larger the scoring consistency index, the higher the consistency degree, the higher the rationality degree, and the higher the objectivity degree of the reviewers' scores.

[0098] Specifically, calculate the scoring consistency index for each reviewer according to the inter-rater consistency parameter and the intra-rater consistency parameter of each reviewer, and the fifth preset formula.

[0099] The fifth preset formula is: ;

[0100] Among them, represents the scoring consistency index of the i-th reviewer, represents the inter-rater consistency parameter of the i-th reviewer, represents the intra-rater consistency parameter of the i-th reviewer. Using the inter-rater consistency parameter and the intra-rater consistency parameter to calculate the scoring consistency index can improve the accuracy of the scoring consistency index.

[0101] It should be noted that for the execution order of step S102, step S103, and step S104, it can be: first execute step S102, then execute step S103, and finally step S104; it can also be: first execute step S103, then execute step S102, and finally step S104; it can also be: first execute step S104, then execute step S103, and finally step S102; it can also be: step S102, step S103, and step S104 are executed simultaneously. The embodiments of the present application do not limit the execution order of step S102, step S103, and step S104, and can be determined according to the actual situation.

[0102] S105: For each reviewer, determine the scoring effectiveness index of the reviewer according to the scoring volatility index, the scoring tendency index, and the scoring consistency index of the reviewer.

[0103] In this step, after the computer determines the scoring volatility index, scoring tendency index, and scoring consistency index for each reviewer, for each reviewer, based on the scoring volatility index, scoring tendency index, and scoring consistency index of this reviewer, the scoring effectiveness index of this reviewer can be determined.

[0104] The scoring effectiveness index is used to characterize the effectiveness of a reviewer's scoring. The larger the scoring effectiveness index, the higher the effectiveness of the reviewer's scoring.

[0105] Specifically, if the scoring volatility index is less than or equal to the first preset threshold, and the absolute value of the scoring tendency index is equal to the second preset threshold, it indicates that the fluctuation degree of the reviewer's scoring is small, and the tendency to give high scores and low scores is small. Then, the scoring consistency index is used as the scoring effectiveness index.

[0106] If the scoring volatility index is greater than the first preset threshold, or the absolute value of the scoring tendency index is not equal to the second preset threshold, it indicates that the fluctuation degree of the reviewer's scoring is large, or the tendency to give high scores or low scores is large. Then, the scoring effectiveness index will be calculated based on the scoring volatility index, scoring tendency index, and scoring consistency index.

[0107] The preset effectiveness calculation formula can be used to calculate the scoring effectiveness index. The preset effectiveness calculation formula is: . Wherein, represents the scoring effectiveness index of the i-th reviewer, represents the scoring consistency index of the i-th reviewer, represents the scoring volatility index of the i-th reviewer, represents the scoring tendency index of the i-th reviewer, represents the first preset adjustment coefficient, represents the second preset adjustment coefficient.

[0108] It should be noted that the first preset adjustment coefficient can be 0.1, 0.2, 0.5, etc., and the second preset adjustment coefficient can be 0.1, 0.2, 0.5, etc. The embodiments of the present application do not limit the first preset adjustment coefficient and the second preset adjustment coefficient, and they can be determined according to the actual situation.

[0109] It should be noted that the second preset threshold is equal to the second preset tendency value. The absolute value of the scoring tendency index being equal to the second preset threshold can be replaced with: the scoring tendency index is less than the first preset tendency threshold and greater than the second preset tendency threshold.

[0110] The absolute value of the scoring tendency index not being equal to the second preset threshold can be replaced with: the scoring tendency index is greater than or equal to the first preset tendency threshold, or less than or equal to the second preset tendency threshold.

[0111] It should be noted that the first preset threshold can be 0.45, 0.5, 0.55, etc. The embodiments of the present application do not limit the first preset threshold, which can be determined according to the actual situation.

[0112] For the evaluation method of scoring effectiveness provided in this embodiment, by calculating the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter, and intra-rater consistency parameter for each reviewer based on the scores of each reviewer for each item, and then determining the scoring volatility index, scoring tendency index, and scoring consistency index of the reviewer according to these parameters; furthermore, determining the scoring effectiveness index of the reviewer according to the three determined indexes. Compared with the prior art that only evaluates the scoring effectiveness based on the consistency of scores, this solution determines the scoring effectiveness index of the reviewer through the scoring volatility index, scoring tendency index, and scoring consistency index of the reviewer, effectively improving the evaluation accuracy of the scoring effectiveness.

[0113] Figure 2 FIG. 9 is a schematic flowchart of Embodiment 2 of the evaluation method for scoring effectiveness provided by the present application. On the basis of the above embodiment, the embodiments of the present application illustrate the situation where the computer calculates the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter, and intra-rater consistency parameter for each reviewer according to the scores of each reviewer for each item. As Figure 2 shown, the evaluation method for scoring effectiveness specifically includes the following steps:

[0114] S201: Determine the average score, median score, and standard deviation of scores of each reviewer, as well as the average score and standard deviation of scores of each item, according to the scores of each reviewer for each item.

[0115] In this step, after the computer obtains the scores of each reviewer for each item, it determines the average score, median score, and standard deviation of scores of each reviewer, as well as the average score and standard deviation of scores of each item, according to the scores of each reviewer for each item.

[0116] The average score of the i-th reviewer The standard deviation of the scores of the i-th reviewer The average score of the j-th item The standard deviation of the scores of the j-th item .

[0117] Among them, n represents the number of items, represents the score of the i-th reviewer for the j-th item, and m represents the number of reviewers.

[0118] S202: Calculate the scoring stability parameter for each reviewer based on the scores given by each reviewer to each project, the average score of each reviewer, and the first preset formula.

[0119] In this step, after the computer obtains the average score of each reviewer, it calculates the scoring stability parameter for each reviewer in combination with the scores given by each reviewer to each project and the first preset formula.

[0120] The first preset formula is: ; where represents the scoring stability parameter of the i-th reviewer, n represents the number of projects, represents the score given by the i-th reviewer to the j-th project, represents the average score of the i-th reviewer.

[0121] S203: Determine the scoring deviation parameter for each reviewer based on the scores given by each reviewer to each project and the standard deviation of the scores of each project.

[0122] In this step, after the computer obtains the standard deviation of the scores of each project, it can determine the scoring deviation parameter for each reviewer in combination with the scores given by each reviewer to each project.

[0123] Specifically, based on the scores given by each reviewer to each project and the standard deviation of the scores of each project, determine the confidence interval for each project.

[0124] The confidence interval for the j-th project is: , where m represents the number of reviewers, represents the score given by the i-th reviewer to the j-th project, represents the standard deviation of the scores of the j-th project.

[0125] For each reviewer, based on the scores given by the reviewer to each project and the confidence interval for each project, determine the scoring deviation parameter for the reviewer.

[0126] For each project, if the score given by the reviewer to the project does not fall within the confidence interval of the project, take this score as the deviation score of the reviewer, and then take the number of the reviewer's deviation scores as the scoring deviation parameter of the reviewer.

[0127] S204: Calculate the scoring tendency parameter for each reviewer based on the average score, median score, standard deviation of the scores of each reviewer, and the second preset formula.

[0128] In this step, after the computer determines the average score, median score, and standard deviation of scores for each reviewer, it can calculate the scoring tendency parameter for each reviewer in combination with the second preset formula.

[0129] The second preset formula is: , where represents the scoring tendency parameter of the i-th reviewer, represents the average score of the i-th reviewer, represents the median score of the i-th reviewer, represents the standard deviation of scores of the i-th reviewer.

[0130] S205: Calculate the inter-rater consistency parameter for each reviewer based on the average score of each reviewer, the average score of each project, and the third preset formula.

[0131] In this step, after the computer determines the average score of each reviewer and the average score of each project, it calculates the inter-rater consistency parameter for each reviewer in combination with the third preset formula.

[0132] The third preset formula is: , where represents the inter-rater consistency parameter of the i-th reviewer, n represents the number of projects, represents the average score of the j-th project, represents the average score of the i-th reviewer.

[0133] S206: Calculate the intra-rater consistency parameter for each reviewer based on the scores of each reviewer for each project, the average score of each reviewer, the average score of each project, and the fourth preset formula.

[0134] In this step, after the computer determines the scores of each reviewer for each project, the average score of each reviewer, and the average score of each project, it can calculate the intra-rater consistency parameter for each reviewer in combination with the fourth preset formula.

[0135] The fourth preset formula is: , where represents the intra-rater consistency parameter of the i-th reviewer, n represents the number of projects, represents the score of the i-th reviewer for the j-th project, represents the average score of the i-th reviewer, represents the average score of the j-th project.

[0136] It should be noted that the execution order of steps S202 - S206 can be: step S202, step S203, step S204, step S205, step S206; it can also be: step S203, step S202, step S204, step S206, step S205; it can also be: step S206, step S205, step S203, step S204, step S202, etc. The embodiments of the present application do not limit the execution order of steps S202 - S206, which can be determined according to the actual situation.

[0137] The evaluation method for scoring validity provided in this embodiment determines the average score, median score, and standard deviation of each reviewer's scores, as well as the average score and standard deviation of each item's scores based on the scores of each reviewer for each item. Then, based on these data, the score stability parameter, score deviation parameter, score tendency parameter, inter-rater agreement parameter, and intra-rater agreement parameter of each reviewer are determined, improving the accuracy of parameter calculation.

[0138] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment of the present application.

[0139] Figure 3 It is a schematic structural diagram of an embodiment of an evaluation device for scoring validity provided by the present application. As Figure 3 shown, the evaluation device 30 for scoring validity includes:

[0140] A processing module 31, configured to:

[0141] Calculate the score stability parameter, score deviation parameter, score tendency parameter, inter-rater agreement parameter, and intra-rater agreement parameter of each reviewer according to the scores of each reviewer for each item obtained;

[0142] For each reviewer, determine the score volatility index of the reviewer according to the score stability parameter and score deviation parameter of the reviewer;

[0143] For each reviewer, determine the score tendency index of the reviewer according to the score tendency parameter of the reviewer;

[0144] Calculate the score agreement index of each reviewer according to the inter-rater agreement parameter and intra-rater agreement parameter of each reviewer;

[0145] An evaluation module 32, configured to determine a scoring effectiveness index for each of the reviewers according to the scoring volatility index, scoring tendency index, and scoring consistency index of the reviewer.

[0146] Further, the processing module 31 is specifically configured to:

[0147] Determine the average score, median score, and standard deviation of scores of each reviewer according to the scores of each reviewer for each item, as well as the average score and standard deviation of scores of each item;

[0148] Calculate a scoring stability parameter for each reviewer according to the scores of each reviewer for each item, the average score of each reviewer, and a first preset formula;

[0149] Determine a scoring deviation parameter for each reviewer according to the scores of each reviewer for each item and the standard deviation of scores of each item;

[0150] Calculate a scoring tendency parameter for each reviewer according to the average score, median score, and standard deviation of scores of each reviewer, and a second preset formula;

[0151] Calculate an inter-rater consistency parameter for each reviewer according to the average score of each reviewer, the average score of each item, and a third preset formula;

[0152] Calculate an intra-rater consistency parameter for each reviewer according to the scores of each reviewer for each item, the average score of each reviewer, the average score of each item, and a fourth preset formula.

[0153] Further, the first preset formula is: ;

[0154] The second preset formula is: ;

[0155] The third preset formula is: ;

[0156] The fourth preset formula is: ;

[0157] Wherein, represents the scoring stability parameter of the i-th reviewer, represents the scoring tendency parameter of the i-th reviewer, represents the inter-rater consistency parameter of the i-th reviewer, represents the within-rater consistency parameter of the i-th reviewer, represents the score of the i-th reviewer for the j-th item, represents the average score of the i-th reviewer, and n represents the number of items, represents the median score of the i-th reviewer, represents the standard deviation of the scores of the i-th reviewer, represents the average score of the j-th item.

[0158] Furthermore, the processing module 31 is specifically further configured to:

[0159] Determine the confidence interval of each item according to the scores of each reviewer for each item and the standard deviation of the scores of each item;

[0160] For each reviewer, determine the score deviation parameter of the reviewer according to the scores of the reviewer for each item and the confidence interval of each item.

[0161] Furthermore, the processing module 31 is specifically further configured to:

[0162] Calculate the score consistency index of each reviewer according to the inter-rater consistency parameter and the within-rater consistency parameter of each reviewer, and a fifth preset formula;

[0163] The fifth preset formula is: ;

[0164] Wherein, represents the score consistency index of the i-th reviewer, represents the inter-rater consistency parameter of the i-th reviewer, represents the within-rater consistency parameter of the i-th reviewer.

[0165] Furthermore, the evaluation module 32 is specifically configured to:

[0166] If the score volatility index is less than or equal to a first preset threshold, and the absolute value of the score tendency index is equal to a second preset threshold, then use the score consistency index as the score validity index;

[0167] If the score volatility index is greater than the first preset threshold, or the absolute value of the score tendency index is not equal to the second preset threshold, then calculate the score validity index according to the score volatility index, the score tendency index, and the score consistency index.

[0168] The evaluation device for scoring effectiveness provided in this embodiment is used to implement the technical solutions in any of the foregoing method embodiments. Its implementation principle and technical effects are similar and will not be elaborated here.

[0169] Figure 4 It is a schematic structural diagram of an electronic device provided by this application. As Figure 4 shown, the electronic device 40 includes:

[0170] a processor 41, a memory 42, and a communication interface 43;

[0171] The memory 42 is used to store the executable instructions of the processor 41;

[0172] Wherein, the processor 41 is configured to execute the technical solutions in any of the foregoing method embodiments by executing the executable instructions.

[0173] Optionally, the memory 42 can be either independent or integrated with the processor 41.

[0174] Optionally, when the memory 42 is a device independent of the processor 41, the electronic device 40 may further include:

[0175] a bus 44. The memory 42 and the communication interface 43 are connected to the processor 41 through the bus 44 to complete mutual communication. The communication interface 43 is used to communicate with other devices.

[0176] Optionally, the communication interface 43 can be specifically implemented by a transceiver. The communication interface is used to implement communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include a random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0177] The bus 44 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0178] The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0179] The electronic device is used to execute the technical solutions in any of the foregoing method embodiments. The implementation principles and technical effects are similar and will not be elaborated here.

[0180] The embodiment of the present application also provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the technical solutions provided in any of the foregoing method embodiments.

[0181] The embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, it is used to implement the technical solutions provided in any of the foregoing method embodiments.

[0182] Those of ordinary skill in the art can understand that all or part of the steps of implementing the foregoing method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the foregoing method embodiments; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disk and other media that can store program codes.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for evaluating the effectiveness of scoring, characterized in that: include: Calculate the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter and intra-rater consistency parameter of each reviewer according to the obtained scores of each reviewer on each item; For each of the reviewers, determining a score volatility index of the reviewer according to the score stability parameter and score deviation parameter of the reviewer; For each of the reviewers, determining a scoring tendency index of the reviewer according to the reviewer's scoring tendency parameter; Calculating the scoring consistency index of each reviewer according to the inter-rater consistency parameter and the intra-rater consistency parameter of each reviewer; For each of the reviewers, the reviewer's scoring validity index is determined based on the reviewer's scoring volatility index, scoring tendency index, and scoring consistency index.

2. The method according to claim 1, characterized in that The step of calculating the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter and intra-rater consistency parameter of each reviewer based on the obtained scores of each reviewer on each project includes: According to the scores of each reviewer on each project, determine the average score, median score and standard deviation of the scores of each reviewer, as well as the average score and standard deviation of the scores of each project; Calculate the score stability parameter of each reviewer according to the score of each reviewer on each project, the average score of each reviewer and the first preset formula; Determine the scoring deviation parameter of each reviewer according to the scoring of each item by each reviewer and the scoring standard deviation of each item; Calculate the scoring tendency parameter of each reviewer according to the scoring average, scoring median, scoring standard deviation of each reviewer and a second preset formula; Calculating the inter-rater consistency parameter of each reviewer according to the average score of each reviewer, the average score of each item, and a third preset formula; The intra-rater consistency parameter of each reviewer is calculated based on the score of each reviewer on each item, the average score of each reviewer, the average score of each item and a fourth preset formula.

3. The method according to claim 2, characterized in that The first preset formula is: ; The second preset formula is: ; The third preset formula is: ; The fourth preset formula is: ; in, represents the scoring stability parameter of the i-th reviewer, represents the scoring tendency parameter of the i-th reviewer, represents the inter-rater agreement parameter of the ith reviewer, represents the intra-rater agreement parameter of the ith reviewer, represents the score of the i-th reviewer on the j-th project, represents the average score of the i-th reviewer, n represents the number of projects, represents the median score of the i-th reviewer, represents the standard deviation of the score of the i-th reviewer, represents the average score of the j-th item.

4. The method according to claim 2, characterized in that: Determining the scoring deviation parameter of each reviewer according to the scoring of each item by each reviewer and the scoring standard deviation of each item includes: Determine the confidence interval of each item according to the score of each reviewer on each item and the standard deviation of the score of each item; For each of the reviewers, the reviewer's score deviation parameter is determined based on the reviewer's score for each item and the confidence interval of each item.

5. The method according to claim 1, characterized in that The step of calculating the scoring consistency index of each reviewer according to the inter-rater consistency parameter and the intra-rater consistency parameter of each reviewer comprises: Calculate the scoring consistency index of each reviewer according to the inter-rater consistency parameter and intra-rater consistency parameter of each reviewer and the fifth preset formula; The fifth preset formula is: ; in, represents the scoring consistency index of the i-th reviewer, represents the inter-rater agreement parameter of the ith reviewer, represents the intra-rater agreement parameter of the ith reviewer.

6. The method according to claim 1, characterized in that Determining the reviewer's scoring validity index based on the reviewer's scoring volatility index, scoring tendency index, and scoring consistency index includes: If the score volatility index is less than or equal to the first preset threshold, and the absolute value of the score tendency index is equal to the second preset threshold, the score consistency index is used as the score validity index; If the score volatility index is greater than the first preset threshold, or the absolute value of the score tendency index is not equal to the second preset threshold, the score validity index will be calculated based on the score volatility index, the score tendency index and the score consistency index.

7. A scoring effectiveness evaluation device, characterized in that: include: Processing modules for: Calculate the scoring stability parameter, scoring deviation parameter, scoring tendency parameter, inter-rater consistency parameter and intra-rater consistency parameter of each reviewer according to the obtained scores of each reviewer on each item; For each of the reviewers, determining a score volatility index of the reviewer according to the score stability parameter and score deviation parameter of the reviewer; For each of the reviewers, determining a scoring tendency index of the reviewer according to the reviewer's scoring tendency parameter; Calculating the scoring consistency index of each reviewer according to the inter-rater consistency parameter and the intra-rater consistency parameter of each reviewer; The evaluation module is used to determine the scoring validity index of each reviewer according to the reviewer's scoring volatility index, scoring tendency index and scoring consistency index.

8. An electronic device, characterized in that: include: Processor, memory, communication interface; The memory is used to store executable instructions of the processor; Wherein, the processor is configured to execute the scoring effectiveness evaluation method described in any one of claims 1 to 6 by executing the executable instructions.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for evaluating the effectiveness of scoring according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, is used to implement the scoring effectiveness evaluation method described in any one of claims 1 to 6.

Images