Score scale adjustment method and device, storage medium and electronic equipment

Abstract

This disclosure provides a method, apparatus, storage medium, and electronic device for adjusting a scoring scale. The method includes: acquiring a target scoring scale; constructing multiple sets of sample data; inputting the same set of sample data into at least two large language models to obtain at least two sets of multidimensional evaluation results; the multidimensional evaluation results include evaluation information corresponding to at least two evaluation dimensions; based on the at least two sets of multidimensional evaluation results corresponding to the multiple sets of sample data, determining the cross-model difference metric of the at least two large language models, and a set of highly divergent sample objects; the set of highly divergent sample objects includes sample objects whose evaluation information difference in at least one evaluation dimension is greater than a predetermined range; adjusting the target scoring scale based on the cross-model difference metric and the set of highly divergent sample objects; and repeating the step of constructing the sample data.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to a method, apparatus, storage medium, and electronic device for adjusting a scoring scale. Background Technology

[0002] Using Large Language Models (LLMs) as the scoring subject (i.e., the "LLM-as-Rater" or "LLM-as-Judge" paradigm) to conduct multi-dimensional evaluations of text, images, multimedia content, and structured objects (such as professions and products) has become a cutting-edge research hotspot in this field. Existing techniques typically rely on manually constructed "rubric" standards to guide the LLM output of structured scoring results; these standards generally encompass operational definitions of each evaluation dimension, grading criteria, and anchor samples of positive and negative examples used for calibration. Summary of the Invention

[0003] In view of this, embodiments of the present disclosure provide a method, apparatus, storage medium, and electronic device for adjusting a scoring scale.

[0004] In a first aspect, embodiments of this disclosure provide a method for adjusting a scoring scale, comprising the following steps: Obtain the target rating scale; Construct multiple sets of sample data, each set of sample data including text prompts corresponding to the sample object and the target rating scale; The same sample data is input into at least two large language models to obtain at least two sets of multidimensional evaluation results; the multidimensional evaluation results include evaluation information corresponding to at least two evaluation dimensions. Based on the at least two sets of multidimensional evaluation results corresponding to multiple sets of the sample data, determine the cross-model difference measure of the at least two large language models, and the set of highly divergent sample objects; the set of highly divergent sample objects includes sample objects whose difference in evaluation information on at least one evaluation dimension is greater than a set range. The target scoring scale is adjusted based on the cross-model difference metric and the set of highly divergent sample objects. The step of constructing sample data is repeated until the adjustment stop condition of the target scoring scale is met.

[0005] Furthermore, the at least two large language models are mutually independent large language models with different pre-trained data and / or reinforcement learning training lineages based on human feedback.

[0006] Furthermore, the cross-model difference measure includes a correlation indicator for measuring the frequency of scoring trends and / or a numerical deviation indicator for measuring the degree of deviation of scoring values.

[0007] Furthermore, the correlation indicators include Pearson correlation coefficient and / or Spearman rank correlation coefficient; the numerical deviation indicators include mean absolute difference and / or cross-model deviation.

[0008] Furthermore, the target scoring scale includes a level description corresponding to each evaluation dimension; adjusting the target scoring scale based on the cross-model difference metric and the set of highly divergent sample objects includes: If the correlation index corresponding to at least one evaluation dimension is lower than a preset first threshold and the numerical deviation index is greater than a preset second threshold, then the level description of the at least one evaluation dimension is adjusted according to the direction of the numerical deviation index.

[0009] Furthermore, adjusting the target scoring scale based on the cross-model difference metric and the set of highly divergent sample objects further includes: If the correlation index corresponding to at least one evaluation dimension is greater than or equal to the preset first threshold, and the sample objects in the high divergence sample object set show commonalities in the preset meta-attributes, then at least a portion of the objects in the high divergence sample object set are extracted to generate new negative example anchors or positive example anchors, which are then added to the target scoring scale.

[0010] Furthermore, at least a portion of the objects are extracted from the set of highly divergent sample objects to generate new negative or positive example anchors, which are then added to the target scoring scale, including: The sample objects in the set of highly divergent sample objects are summarized using an auxiliary large language model to generate candidate anchor text; Receive manual confirmation or revision instructions for the candidate anchor text, and add the confirmed or revised candidate anchor text to the target scoring scale.

[0011] Furthermore, the adjustment stopping conditions include: The average relevance index of the at least two large language models in each evaluation dimension is not lower than a preset third threshold; or The numerical deviation index for each evaluation dimension is lower than a preset fourth threshold.

[0012] Furthermore, the sample objects include at least one of occupations, products, and educational works; the text prompts include the category code, title, and / or long text description of the sample objects.

[0013] Secondly, embodiments of this disclosure provide a scoring scale adjustment device, including: The scale acquisition module is used to acquire the target scoring scale; The data construction module is used to construct multiple sets of sample data, each set of sample data including text prompts corresponding to the sample object and the target scoring scale; The evaluation acquisition module is used to input the same sample data into at least two large language models respectively to obtain at least two sets of multidimensional evaluation results; the multidimensional evaluation results include evaluation information corresponding to at least two evaluation dimensions; The difference determination module is used to determine the cross-model difference metric of the at least two large language models and the set of high-disagreement sample objects based on the at least two sets of multidimensional evaluation results corresponding to multiple sets of the sample data; the set of high-disagreement sample objects includes sample objects whose evaluation information difference is greater than a set range in at least one evaluation dimension. The scale adjustment module is used to adjust the target scoring scale based on the cross-model difference metric and the set of highly divergent sample objects. The iterative control module is used to jump to the scale acquisition module and repeat the process until the adjustment stop condition of the target scoring scale is met.

[0014] Thirdly, embodiments of this disclosure provide an electronic device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in any of the above aspects.

[0015] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing a computer program, wherein the program, when executed by a processor, implements the steps of the method described in any of the preceding aspects.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments of this disclosure with reference to the accompanying drawings, in which: Figure 1 This is a flowchart illustrating the scoring scale adjustment method proposed in the embodiments of this disclosure; Figure 2 This is a schematic flowchart of an example of the scoring scale adjustment method proposed in this embodiment of the present disclosure; Figure 3 This is a structural block diagram of the scoring scale adjustment device proposed in the embodiments of this disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device suitable for implementing the scoring scale adjustment method according to an embodiment of the present disclosure. Detailed Implementation

[0018] The present disclosure is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present disclosure below, certain specific details are described in detail. However, those skilled in the art will fully understand the present disclosure without these details. To avoid obscuring the substance of the present disclosure, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0019] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0020] It should be noted that the specific numerical values ​​listed in this disclosure (including the examples in the various optional implementations below and Embodiments 1 and 2) include the consistency coefficient ρ_d of each evaluation dimension and the cross-model bias. d The mean absolute difference (MAE_d), preset thresholds, and the number of occupations / samples / objects are all exemplary values ​​used to illustrate the technical solution and its technical effects, and are not actual test results on a specific dataset. People skilled in the art may obtain different values ​​when using different large language models, sample sets, or scoring scale configurations, which does not affect the implementation and effect of the technical solution disclosed.

[0021] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0022] In the description of this disclosure, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0023] The inventors of this disclosure have discovered the following shortcomings in the solutions described in the background art: (1) Manually written rating scales (Rubric) have inherent limitations: Ambiguity exists: Due to the ambiguity of natural language, manually defined operational standards are often ambiguous and difficult to cover all edge cases.

[0024] Sample bias: The anchoring of positive and negative samples often relies on the experience of experts, which can easily lead to sample imbalance and cause the scoring criteria to fail in certain scenarios.

[0025] Because manual review is costly and has low coverage, these defects are difficult to detect in the early stages through traditional "expert review".

[0026] (2) The evaluation and verification level is prone to the illusion of consistency: False consistency: High consistency obtained by repeatedly scoring the same LLM under the same scoring scale only proves that the output of the model under that rule is deterministic, not that the scoring is accurate.

[0027] Systematic bias masking: If the fuzzy definition of the scoring scale happens to match the pre-training prior of LLM, single-model testing will perfectly mask the systematic bias of the scoring scale, causing errors to be repeatedly confirmed.

[0028] (3) Missed / false detection of unsupervised signals: Domain coverage gap: When the scoring scale has domain bias in its description of a certain dimension (for example, "remote work ability" only focuses on technical positions and ignores service-oriented remote positions), the existing model cannot detect this gap.

[0029] High cost of manual verification: The current system lacks an automated anomaly detection mechanism and must rely on manual verification against actual business scenarios, which is almost unacceptable from an engineering perspective.

[0030] (4) Lack of methodology at the iterative optimization level: Empiricism-driven: Current adjustments to the scoring scale rely heavily on the intuition of the cue word engineer (such as "adding a few counterexamples based on intuition" or "tightening the wording of a definition").

[0031] Lack of data closure: There is a lack of an automated iterative mechanism based on data feedback, making it impossible to deduce in reverse the direction of the scoring scale based on anomalies in the scoring distribution.

[0032] To address the aforementioned issues, this disclosure aims to provide an iterative closed-loop scheme that utilizes multi-model ensemble scoring and cross-model discrepancy analysis to automatically identify and correct deficiencies in the scoring scale (Rubric). This scheme scores the same batch of structured objects using at least two independent large language models, calculates the cross-model pairwise (object × dimension) score difference distribution, and thus automatically identifies systematic biases or loopholes in the scoring scale. Furthermore, the structured objects that cause these discrepancies are used as negative or strong positive examples for automatic or semi-automatic correction back into the scoring scale. This technical solution solves the core technical challenges of traditional single-model evaluation, such as the inability to identify scoring scale deficiencies, the lack of unsupervised signals for missed and false judgments, and the lack of scientific methodologies in iterative optimization.

[0033] This disclosure proposes a method for adjusting the scoring scale; see [link to relevant documentation]. Figure 1 As shown, the method includes the following steps: Step S101: Obtain the target scoring scale.

[0034] In some embodiments, the rating scale may include operational definitions and grading criteria for multiple assessment dimensions, and may also include positive and / or negative anchor samples for calibration. At the start of the adjustment, the target rating scale may, for example, be an initial rating scale R(0) of the Holland Occupational Interest Model (RIASEC) containing 30 dimensions.

[0035] Step S102: Construct multiple sets of sample data. Each set of sample data includes text prompts (containing NOC codes, titles, and long text descriptions) corresponding to the sample objects (such as 491 National Occupational Classifications of Canada (NOC) occupations) and target scoring scales.

[0036] Step S103: Input the same sample data into at least two large language models (such as Qwen-3.5 and DeepSeek-R1) to obtain at least two sets of multidimensional evaluation results. The multidimensional evaluation results include evaluation information corresponding to 30 evaluation dimensions (such as scores of 0-3).

[0037] Step S104: Based on at least two sets of multidimensional evaluation results corresponding to multiple sample data, determine the cross-model difference measure of at least two large language models (e.g., each set of multidimensional evaluation results is a rating matrix, calculate the difference between the two rating matrices), and a set of highly divergent sample objects, which includes sample objects whose evaluation information difference in at least one evaluation dimension is greater than a set range (e.g., the score difference is ≥ 2 levels). Step S105: Based on the cross-model difference measure and the set of highly divergent sample objects, adjust the target scoring scale, for example, modify the operational definition or add anchors for the highly divergent dimensions. Step S106: Repeat the step of constructing sample data until the adjustment stop condition of the target scoring scale is met, and output the final scoring scale. and rating dataset.

[0038] In some alternative implementations, the at least two large language models are mutually independent large language models with different pre-trained data and / or reinforcement learning training lineages based on human feedback.

[0039] This embodiment defines the independence condition for the large language models. The at least two large language models are mutually independent models with different pre-training data compositions and / or different reinforcement learning (RLHF) training lineages based on human feedback. For example, M_a = Qwen-3.5-122B-A10B (Tongyi Qianwen series) and M_b = DeepSeek-R1-671B (DeepSeek series model) are selected. These two belong to different model vendor families, have different pre-training corpus compositions, and have completely different RLHF training lineages. This leads to different judgment tendencies in their underlying prior knowledge when faced with fuzzy definitions in the same scoring scale, thus effectively exposing scale defects that cannot be detected by single-model testing.

[0040] In some alternative implementations, the cross-model difference measure includes a correlation indicator to measure the frequency of rating trends and / or a numerical deviation indicator to measure the degree of deviation of rating values.

[0041] For example, when assessing the "remote work suitability" dimension, if the two models show a consistent overall trend in their scores for 500 occupations (e.g., technical positions all score highly, while manual labor positions all score low), then the correlation index is high. If model A generally gives a score one point higher than model B, then the numerical deviation index is large. By combining these two types of indicators, it is possible to comprehensively determine whether the discrepancies between the models are merely due to "inconsistent scoring standards" (large numerical deviation but high correlation) or due to "fundamental cognitive conflict" (low correlation), thus providing diagnostic basis from different dimensions for subsequent attribution of scale defects.

[0042] In some alternative implementations, the correlation metrics include the Pearson correlation coefficient and / or the Spearman rank correlation coefficient; the numerical bias metrics include the mean absolute difference and / or cross-model bias.

[0043] For dimension d, calculate and The Pearson correlation coefficient or Spearman rank correlation coefficient ρ_d between models is used to measure ordination consistency; cross-model bias is calculated. To measure systematic shifts; the mean absolute difference (MAE_d) is calculated to measure the magnitude of absolute deviation. For example, it can be observed that ρ_{A3} for the A3 (performing arts) dimension is only 0.32, indicating that the rating trends of the two models in this dimension are significantly different.

[0044] In some optional implementations, the target scoring scale includes a level description corresponding to each evaluation dimension; if the relevance index corresponding to at least one evaluation dimension is lower than a preset first threshold (e.g., Spearman ρ_d < 0.6), and the numerical deviation index is greater than a preset second threshold (e.g., |bias|), the target scoring scale is further supported by the target scoring scale.d If |>0.3), then the level description of at least one assessment dimension is adjusted according to the direction of the numerical bias index. For example, in the L3 "Self-Management" dimension of the CAP-25 competency assessment, ρ_{L3} = 0.36, |bias| = 0.5. Analysis reveals that the bias direction is positive (the Claude model's score is systematically higher than the Llama model's), with Claude biasing towards giving high scores to "Self-Employment" and Llama biasing towards giving high scores to "Management Positions". To address this systematic bias, the level description of "Self-Management" is precisely revised to "Individual's Arrangement and Decision-Making Power over Their Own Workflow" to eliminate the bias in understanding between the two models.

[0045] In some optional implementations, if the correlation index corresponding to at least one evaluation dimension is greater than or equal to the preset first threshold (e.g., ρ_d ≥ 0.6, indicating that the overall trend is the same), and the sample objects in the set of highly divergent sample objects show commonalities in preset meta-attributes (e.g., object type, industry, field), then at least a portion of the objects in the set of highly divergent sample objects are extracted to generate new negative example anchors or positive example anchors, which are then added to the target scoring scale.

[0046] In this implementation, when the overall scoring trends among models are consistent (correlation meets the standard), but significant discrepancies arise for certain specific sample objects, the commonalities in the meta-attributes of these highly divergent sample objects can be analyzed. If common features are extracted, at least a portion of these sample objects can be extracted as new "negative example anchors" or "positive example anchors" to supplement the target scoring scale. For example, in the "remote work adaptability" dimension, the overall ρ_d = 0.65, but the 45 objects in the highly divergent set Δ_d share a commonality in the meta-attribute "industry"—all belong to the "customer service / telemarketing" category. This is identified as a fuzzy area missed by the scoring scale, therefore representative samples are extracted from these 45 objects as new positive example anchors.

[0047] In some alternative implementations, if the correlation index corresponding to at least one evaluation dimension is greater than or equal to the preset first threshold, and the sample objects in the set of highly divergent sample objects have no commonality in the preset meta-attribute, then the difference is determined to be scoring noise, and the target scoring scale is not revised.

[0048] In some optional implementations, at least a portion of the objects in the set of highly divergent sample objects are extracted as new anchors, including: summarizing the sample objects in the set of highly divergent sample objects using an auxiliary large language model to generate candidate anchor texts; receiving manual confirmation or revision instructions for the candidate anchor texts, and adding the confirmed or revised candidate anchor texts to the target scoring scale.

[0049] This implementation doesn't directly insert the original high-disagreement sample pairs into the scoring scale. Instead, it uses an auxiliary large language model to summarize and analyze these high-disagreement sample objects, automatically generating structured "candidate anchor text." This candidate anchor text can then be pushed to relevant personnel for confirmation or revision instructions. Only after confirmation by relevant personnel will the text be officially added to the target scoring scale. This ensures that the automatically generated scoring scale is both efficient and professionally accurate. For example, the auxiliary large language model automatically extracts a typical representative from the high-disagreement object set Δ_d, "NOC 64410: Retail Store Clerk (Online Customer Support)," and generates a candidate positive example anchor text: "Online customer service retail store clerks have high remote adaptability." After manual review and revision, this is added to the scoring scale as a formal positive example anchor, while the operational definition is revised to "including but not limited to technical support and online customer service," achieving incremental updates through human-machine collaboration.

[0050] In some optional implementations, the adjustment stopping condition includes: the average relevance index of the at least two large language models in each evaluation dimension is not lower than a preset third threshold; or the numerical deviation index of each evaluation dimension is lower than a preset fourth threshold.

[0051] In this implementation, after each adjustment, it can be checked whether either of the following conditions is met: First, the average relevance index of all evaluation dimensions is not lower than the preset third threshold, indicating that a high degree of consensus has been reached among the models; second, the numerical deviation index of each evaluation dimension is lower than the preset fourth threshold (e.g., 0.5), indicating that the scoring scale has been fully aligned. As long as either condition is met, the adjustment is stopped and the final version of the scoring scale is output, avoiding falling into an invalid infinite loop.

[0052] For example, the preset third threshold is set to 0.75 (i.e., the average ρ). ≥ 0.75), or a preset fourth threshold of 0.2 (i.e., |bias| for all dimensions). d |<0.2). After three rounds of automated iterations, the system's cross-model Spearman ρ in the "remote work adaptability" dimension jumped from 0.45 to 0.82, and the average ρ across 30 dimensions... Reaching 0.71 across all dimensions | bias d | All values ​​are less than 0.2, which meets the adjustment stopping condition. The system determines that it has converged and outputs the final version R(3).

[0053] In some implementations, the sample object includes at least one of occupations, products, documents, and educational works; the text prompt includes the category code, title, and / or long text description of the sample object.

[0054] For example, when the sample object is an occupation, the text prompt includes the NOC (National Occupational Classification) code, occupation title, and a long text description of the job content; when the sample object is an educational work, the text prompt may include the educational work feature code, work title, and content description.

[0055] The above-described technical solutions proposed in this disclosure are described below through exemplary embodiments.

[0056] See Figure 2 As shown, this technical solution includes the following six core steps: S1 — Constructing the initial scoring scale: Provide an initial rating scale R(0) containing a description of at least one evaluation dimension (denoted as D). Each evaluation dimension includes at least: an operational definition, rating level conditions (e.g., ordinal levels 0–3), and may also include at least one positive anchor and / or at least one negative anchor.

[0057] For example, taking a structured object as a profession, in a job evaluation scenario, assuming one dimension of the scoring scale is remote work adaptability, the corresponding operability definition in R(0) is: whether the position supports completing core business completely outside the physical office. The scoring level is set from 0 to 3. The positive example provided manually is anchored as "software engineer (can deliver code entirely online)", and the negative example is anchored as "assembly line worker (must be on-site to operate machines)".

[0058] S2 — Multi-model independent scoring: Select at least two independent large language models (such as M_a and M_b), and each model scores multiple (let's say K) structured objects one by one across D dimensions based on a scoring scale R(n) (where n represents the number of iterations) under the same cue word template, generating at least two D × K scoring matrices. and The independent large language models can be understood as large language models with different training lineages composed of pre-trained data and / or RLHF (Reinforcement Learning Based on Human Feedback).

[0059] For example, let's select Llama-3 as M_a and Qwen-2.5 as M_b. Input a batch of 500 occupational text prompts and the initial rating scale constructed in the previous step. These text prompts may include NOC (National Occupational Classification) codes, titles, and other corresponding lengthy text descriptions. M_a and M_b run independently, ultimately producing two rating matrices with 500 rows and 30 columns, where 30 represents the dimension of the initial rating scale. Due to the different underlying prior knowledge, they will exhibit different judgment tendencies when facing fuzzy boundaries.

[0060] S3 — Cross-model difference analysis: For each evaluation dimension, calculate the scores between the outputs of at least two large language models: Overall variance statistics: such as calculating the scores of all structured objects on dimension d. and Overall difference statistics between models, such as Pearson correlation coefficient r_d, mean absolute difference MAE_d, cross-model bias, etc. Pairwise differences: Select a set of highly divergent objects Δ_d whose differences are greater than the difference threshold τ (e.g., a difference of ≥ 2 rating levels), where k represents the kth sample object.

[0061] For example, taking the remote work adaptability dimension, the overall Spearman rank correlation ρ_d was found to be only 0.45, and there exists a highly divergent set Δ_d containing 45 objects. Among these 45 objects, Llama-3 generally gave a score of 2, while Qwen-2.5 gave a score of 0. These highly divergent objects are the key clues for subsequent investigation.

[0062] S4 — Automatic Defect Attribution on the Rating Scale: Three types of attribution diagnoses are performed for the differences in each assessment dimension d: (1) If the correlation ρ_d is lower than the correlation threshold (e.g., 0.6) and the bias |bias d If the value is greater than the deviation threshold (e.g., 0.3), then the definition of dimension d is marked as having a systematic bias (e.g., the rating anchor is too strict for one type of object and too lenient for another). (2) If the correlation ρ_d is greater than or equal to the correlation threshold but the set of highly divergent objects Δ_d contains a common subset (such as all objects of a certain type), then the structured objects in the common subset are determined to belong to the "fuzzy zone" missed by the scoring scale. (3) If the correlation ρ_d is greater than or equal to the correlation threshold but the structured objects in the high divergence set Δ_d have no commonality, it is judged as scoring noise and no revision of the scoring scale is required.

[0063] For example, feature clustering of the 45 highly divergent objects revealed that they all belonged to the "customer service / telemarketing" category. This triggered the second scenario mentioned above, which is that it is not noise generated by the model, but rather that the definition of R(0) only focuses on technical output and overlooks the ambiguous area that service-oriented positions can also be completed entirely remotely.

[0064] S5 — Incremental update of rating scale: Make targeted modifications based on the attribution results from the previous step: Regarding attribution (1), then based on bias d Adjust the description of the direction level (e.g., change "≥ 0.5 hours" to "≥ 0.5 hours and is a core defining activity"). For attribution (2), representative objects are extracted from the set of highly divergent objects Δ_d as new negative or positive examples and added to R(n) to generate R(n+1). This process can be summarized by an auxiliary LLM and confirmed by manual sampling.

[0065] For example, the auxiliary LLM automatically extracted the typical structured object representative "NOC 64410: Retail Clerk (Online Customer Support)" from the high-divergence object set Δ_d and added it as a new positive example anchor to R(1). At the same time, the operational definition was revised to: "Does this position support the completion of core business completely outside the physical office, including but not limited to technical support and online customer service?"

[0066] S6 — Iterative convergence: Repeat S2 through S5 using the updated R(n+1) until the cross-model consistency rate (average correlation ρ across dimensions) is achieved. ) greater than or equal to a preset threshold (e.g., ≥ 0.75), or the bias across all dimensions. d |Converges to a minimum value (e.g., <0.2), finally outputting a high-quality version. And its corresponding final score dataset.

[0067] For example, after three rounds of automated iterations, the system's cross-model consistency in the remote work adaptability dimension jumped from the initial 0.45 to 0.82, and at least two large language models were completely aligned in their scores for customer service positions. The system determined that it had converged, stopped iterations, and locked the final R(3) as the final scoring scale.

[0068] The above technical solution can achieve the following technical effects: (1) Defects in the scoring scale are automatically visible, significantly reducing the cost of manual review: This disclosure transforms scoring discrepancies between different models into external supervision signals through cross-model difference analysis, which can automatically calculate the set of highly discrepancies Δ_d, making previously hidden problems such as fuzzy definitions and omissions explicit and accurately located. This automated attribution mechanism effectively replaces the traditional manual item-by-item review process, saving more than 90% of the cost of manual investigation and error correction.

[0069] (2) Significantly improved scoring quality and scientific convergence of the scale: Based on incremental updates and iterative closed loop, this scheme realizes the dynamic self-evolution of the scoring scale. As can be seen in the exemplary application scenario, in the multi-dimensional structured scoring task, by using highly divergent objects as anchors to replenish the scale and continuously iterating, cross-model consistency can be greatly improved. For example, taking occupations as an example of structured objects, 491 occupations were selected as samples, and 5 large language models were used to perform 3 rounds of RRI iteration on the 30-dimensional RIASEC scoring task. The Spearman rank correlation coefficient ρ_d increased from 0.42 in R(0) to 0.71 in R(3), an increase of +0.29. That is, after 3 rounds of iteration, the final performance was significantly better than the evaluation level of the initial single model.

[0070] (3) Provides complete interpretability and end-to-end traceability: The differences between the scoring scale updated in each iteration and the one before the update are verifiable. This solution can clearly record the specific dimension d that triggers the revision, the set of highly divergent objects Δ_d that causes disagreement, and the specific revision content (such as newly added positive / negative example anchors or adjustments to operational definitions). This feature ensures the transparency of the AI ​​evaluation process and meets enterprise-level AI audit and compliance requirements.

[0071] The technical solution of this disclosure is further described below through specific embodiments. It should be noted that the consistency coefficient, deviation value, and sample size appearing in Embodiments 1 and 2 below are all exemplary values ​​mentioned above, and are only used to illustrate the technical effects of the iterative optimization process.

[0072] Example 1: Iterative optimization based on the 30-dimensional score of the Holland Occupational Interest Model (RIASEC) This embodiment illustrates a method for improving the scoring consistency of large language models in occupational psychological assessments through multi-model cross-validation and dynamic anchoring correction. The specific implementation steps are as follows: First, an initial version of the assessment scale was constructed as the baseline input. This scale includes 30 subdivisions of the Holland Occupational Interest Model, with each assessment dimension equipped with 4 operational definitions, 5 positive examples, and 5 negative examples as anchoring criteria. Simultaneously, a test sample set containing 491 occupations from the National Occupational Classification of Canada (NOC 2021) was introduced.

[0073] Secondly, two independent large language models from different vendor families and trained with different human feedback reinforcement learning (RLHF) methods, such as M_a = Qwen-3.5-122B-A10B (Tongyi Qianwen series) and M_b = DeepSeek-R1-671B (Deep Seek series model), were selected to score 491 occupations on 30 subdivisions of RIASEC, generating two independent sets of scoring results, which are expressed by a 30 × 491 scoring matrix.

[0074] Next, the consistency coefficient ρ_d between the two sets of rating results on each dimension is calculated. d. During the investigation, it was found that the consistency coefficient ρ_{A3} = 0.32 for the sub-dimension A3 (Performing Arts), which is below the threshold. Further data comparison showed that 18 professions with a rating difference δ[A3, :] > τ (the preset deviation threshold) on this dimension A3 were added to the high-disagreement object set Δ_{A3} corresponding to the A3 dimension.

[0075] Subsequently, attribution analysis was conducted on these highly divergent object sets Δ_{A3}. The analysis revealed that the original operational definition of the A3 dimension in the original rating scale, "activities involving performances for an audience," was too broad, leading to serious divergence in the model's evaluation of "teacher / trainer" positions—with M_a scoring 2 and M_b scoring 0.

[0076] To address this issue, the operational definition and anchoring of the scoring scale were revised in a targeted manner. In the counterexample anchoring library for the "Performing Arts" dimension, an exclusion rule was explicitly added: "The act of a teacher lecturing to students from a podium is not included in the category of performing arts."

[0077] After completing the first round of updates to the scoring scale, the dual-model scoring process was retried using the updated evaluation scale R(1). At this point, the consistency coefficient of the A3 dimension successfully increased to ρ_{A3} = 0.58. Based on this, the second and third rounds of closed-loop iteration of "scoring-detection-revision" were continued, and the average consistency coefficient of the 30 dimensions increased to 0.66 and then to 0.71, respectively, achieving significant convergence and objectification of the scoring standard.

[0078] Example 2: Iterative optimization of 25-dimensional scoring based on the CAP-25 professional competence assessment scale. First, using an initial version of a vocational competency assessment scale comprising 25 vocational competency dimensions as the input benchmark, and K=900 occupations from the Canadian National Occupational Classification (NOC) as the sample set to be assessed, two heterogeneous large language models, such as M_a = Llama-3.1-70B-AWQ and M_b = Claude Sonnet 4.5 (tool_use mode), were employed as independent scoring engines to evenly distribute the occupational samples in the aforementioned sample set.

[0079] During the scoring and consistency testing process, the observation results showed that the consistency coefficient ρ_{L3} = 0.36 for the "self-management" competency dimension, which is below the threshold. Further attribution analysis revealed an underlying logical conflict: M_b = Claude Sonnet 4.5 tends to rate "self-employment (freelancer)" with high scores, while M_a = Llama-3.1-70B-AWQ tends to rate the traditional "management position" with high scores, indicating a discrepancy in their understanding of the connotation of "self-management".

[0080] To address this core issue, targeted interventions were made to the scoring criteria. On the one hand, the operational definition of "autonomous management" was precisely revised to "an individual's right to arrange and make decisions regarding their own work processes." On the other hand, "assembly line workers strictly constrained by detailed procedures" were explicitly added as a typical representative of low autonomy in the negative example anchoring, thereby eliminating cognitive ambiguity between models.

[0081] Finally, the dual-model scoring was re-performed using a revised scoring scale R(1), and the consistency coefficient ρ_{L3} for the "self-management" dimension significantly increased from 0.36 to 0.56. This embodiment fully demonstrates the universal value of this dynamic optimization mechanism in solving the model semantic alignment problem and improving the reliability of occupational psychological assessments.

[0082] In summary, this disclosure provides a method for using at least two independent large language models to score the same batch of target objects under the same scoring scale, calculating the cross-model pairwise (object × dimension) score difference distribution, automatically identifying dimensions with systematic biases or loopholes in the scoring scale, and automatically or semi-automatically supplementing the scoring scale by anchoring the source objects of the differences as negative examples or strong positive examples to form an iterative closed loop. This method addresses the problems of scoring scale defects not being identified by a single model, unsupervised signals causing missed or misjudged cases, and a lack of methodological approach in iteration.

[0083] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein.

[0084] Figure 3 This is a structural block diagram of the scoring scale adjustment device proposed in an embodiment of this disclosure. This device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 3 As shown, the scoring scale adjustment device includes: The scale acquisition module is used to acquire the target scoring scale; The data construction module is used to construct multiple sets of sample data, each set of sample data including text prompts corresponding to the sample object and the target scoring scale; The evaluation acquisition module is used to input the same sample data into at least two large language models respectively to obtain at least two sets of multidimensional evaluation results; the multidimensional evaluation results include evaluation information corresponding to at least two evaluation dimensions; The difference determination module is used to determine the cross-model difference metric of the at least two large language models and the set of high-disagreement sample objects based on the at least two sets of multidimensional evaluation results corresponding to multiple sets of the sample data; the set of high-disagreement sample objects includes sample objects whose evaluation information difference is greater than a set range in at least one evaluation dimension. The scale adjustment module is used to adjust the target scoring scale based on the cross-model difference metric and the set of highly divergent sample objects. The iterative control module is used to jump to the scale acquisition module and repeat the process until the adjustment stop condition of the target scoring scale is met.

[0085] The scoring scale adjustment device in this embodiment corresponds to the scoring scale adjustment method described above. For details, please refer to the description of the scoring scale scheduling method above.

[0086] Figure 4 This is a schematic diagram of the structure of an electronic device suitable for implementing the scoring scale adjustment method according to an embodiment of the present disclosure.

[0087] like Figure 4 As shown, the electronic device 400 includes a processing unit 401, which can be implemented as a CPU, GPU, FPGA, NPU, or other processing unit. The processing unit 401 can execute various processes according to any of the methods described above in this disclosure, based on a program stored in the read-only memory (ROM) 402 or a program loaded from the storage portion 408 into the random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the electronic device 400. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0088] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.

[0089] In particular, according to embodiments of this disclosure, any of the methods described above in the embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing any of the methods in the embodiments of this disclosure. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411.

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0091] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The described units or modules can also be located in a processor, and the names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0092] In another aspect, this disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs that are used by one or more processors to perform the methods described in this disclosure.

[0093] Another embodiment of this disclosure relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0094] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by specifying relevant hardware through a program. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0095] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for adjusting a scoring scale, characterized in that, Includes the following steps: Obtain the target rating scale; Construct multiple sets of sample data, each set of sample data including text prompts corresponding to the sample object and the target rating scale; The same set of sample data is input into at least two large language models to obtain at least two sets of multidimensional evaluation results. The multidimensional evaluation results include evaluation information corresponding to at least two evaluation dimensions; Based on the at least two sets of multidimensional evaluation results corresponding to multiple sets of the sample data, determine the cross-model difference measure of the at least two large language models, and the set of highly divergent sample objects; the set of highly divergent sample objects includes sample objects whose difference in evaluation information on at least one evaluation dimension is greater than a set range. The target scoring scale is adjusted based on the cross-model difference metric and the set of highly divergent sample objects. The step of constructing sample data is repeated until the adjustment stop condition of the target scoring scale is met.

2. The method according to claim 1, characterized in that, The at least two large language models are independent large language models with different training lineages, consisting of pre-trained data and / or reinforcement learning training based on human feedback.

3. The method according to any one of claims 1-2, characterized in that, The cross-model difference measure includes a correlation indicator to measure the frequency of scoring trends and / or a numerical deviation indicator to measure the degree of deviation of scoring values.

4. The method according to claim 3, characterized in that, The correlation metrics include Pearson correlation coefficient and / or Spearman rank correlation coefficient; the numerical bias metrics include mean absolute difference and / or cross-model bias.

5. The method according to claim 3, characterized in that, The target scoring scale includes a level description corresponding to each evaluation dimension; adjusting the target scoring scale based on the cross-model difference metric and the set of highly divergent sample objects includes: If the correlation index corresponding to at least one evaluation dimension is lower than a preset first threshold and the numerical deviation index is greater than a preset second threshold, then the level description of the at least one evaluation dimension is adjusted according to the direction of the numerical deviation index.

6. The method according to claim 3, characterized in that, The step of adjusting the target scoring scale based on the cross-model difference metric and the set of highly divergent sample objects further includes: If the correlation index corresponding to at least one evaluation dimension is greater than or equal to the preset first threshold, and the sample objects in the high divergence sample object set show commonalities in the preset meta-attributes, then at least a portion of the objects in the high divergence sample object set are extracted to generate new negative example anchors or positive example anchors, which are then added to the target scoring scale.

7. The method according to claim 6, characterized in that, At least a portion of the objects in the set of highly divergent sample objects are extracted to generate new negative or positive example anchors, which are then added to the target scoring scale, including: The sample objects in the set of highly divergent sample objects are summarized using an auxiliary large language model to generate candidate anchor text; Receive manual confirmation or revision instructions for the candidate anchor text, and add the confirmed or revised candidate anchor text to the target scoring scale.

8. The method according to claim 3, characterized in that, The adjustment stopping conditions include: The average relevance index of the at least two large language models in each evaluation dimension is not lower than a preset third threshold; or The numerical deviation index for each evaluation dimension is lower than a preset fourth threshold.

9. The method according to any one of claims 1-2 and 4-8, characterized in that, The sample objects include at least one of occupations, products, and educational works; the text prompts include the category code, title, and / or long text description of the sample objects.

10. A scoring scale adjustment device, characterized in that, include: The scale acquisition module is used to acquire the target scoring scale; The data construction module is used to construct multiple sets of sample data, each set of sample data including text prompts corresponding to the sample object and the target scoring scale; The evaluation acquisition module is used to input the same sample data into at least two large language models respectively to obtain at least two sets of multidimensional evaluation results; The multidimensional evaluation results include evaluation information corresponding to at least two evaluation dimensions; The difference determination module is used to determine the cross-model difference metric of the at least two large language models and the set of high-disagreement sample objects based on the at least two sets of multidimensional evaluation results corresponding to multiple sets of the sample data; the set of high-disagreement sample objects includes sample objects whose evaluation information difference is greater than a set range in at least one evaluation dimension. The scale adjustment module is used to adjust the target scoring scale based on the cross-model difference metric and the set of highly divergent sample objects. The iterative control module is used to jump to the scale acquisition module and repeat the process until the adjustment stop condition of the target scoring scale is met.

11. A computer device, comprising a memory and a processor, characterized in that, When the processor executes the computer program in the memory, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images