Multiple large language models integrate scoring methods, devices, equipment, and storage media.
By dividing the system by vendor family and optimizing the integration of large language models across families, the problems of resource waste and commercial API integration in existing technologies have been solved, achieving efficient and stable multi-model integration.
Patent Information
- Application Number
- CN202610967116.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-30
- Publication Date
- 2026-11-17
AI Technical Summary
Existing methods for integrating multiple language models lack theoretical guidance, leading to resource waste and limited improvement in integration quality. Commercial closed-source APIs are difficult to integrate, and inconsistent structured outputs affect system stability.
By dividing the models by manufacturer family, a cross-family strategy is adopted to successively add candidate models, and trust-weighted scoring and custom tools are used to ensure structured output. The monotonic scaling law of model quality is empirically verified, and the ensemble of integrated models is optimized.
It maximizes integration quality within a limited cost, avoids resource waste, makes full use of commercial API capabilities, and improves the stability and availability of the integration system.
Smart Images

Figure CN123114931A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, specifically to a scoring method, apparatus, storage medium, and electronic device for integrating multiple large language models. Background Technology
[0002] Multi-Language Model (LLM) ensemble methods, including but not limited to majority voting, weighted averaging, and trust mapping weighting, rely on the collaboration of multiple large language model ensemble members to achieve a better output than a single LLM. Existing multi-LLM ensemble methods (such as self-consistency, constitutional AI, and LLM-as-Judge ensemble) largely rely on the naive assumption that "the more models, the better," believing that continuously increasing the number of models will continuously improve the ensemble quality. Summary of the Invention
[0003] In view of this, embodiments of the present disclosure provide a method, apparatus, storage medium, and electronic device for integrated scoring of multiple large language models.
[0004] In a first aspect, embodiments of this disclosure provide an integrated scoring method for multiple large language models, comprising the following steps: Multiple large language models are divided into different vendor families, and the classification of vendor families is based on at least one of the following: the composition of the model's pre-training data and / or the lineage of human feedback reinforcement learning training. Obtain an initial ensemble model combination and use the initial ensemble model combination as the current ensemble model combination; Select a candidate model that belongs to a different vendor family from any of the integration members in the current integration model combination, and add it to the current integration model combination; The sample data is integrated and scored using the ensemble members in the current ensemble model combination to obtain the ensemble score result; wherein, the sample data includes text prompts corresponding to the sample objects; Based on the integrated scoring results and the pre-set standard scoring results, determine whether the model quality of the current integrated model combination has increased after adding the candidate model; If the quality of the current ensemble model combination does not increase, then reduce the ensemble weight of the candidate model or remove the candidate model from the current ensemble model combination; Return to the step of selecting a candidate model that belongs to a different vendor family from any of the integration members in the current integration model combination, and repeat the process until the set stopping condition is met.
[0005] It should be noted that the "model quality" mentioned in this disclosure refers to the degree of consistency between the ensemble scoring results obtained after the ensemble model combination scores the sample data and the standard scoring results. It is a higher-level concept for measuring the scoring ability of the ensemble model combination. Model quality can be measured in various ways, including but not limited to: the rank correlation coefficient, Pearson correlation coefficient, accuracy, F1 score, etc., between the ensemble scoring results and the standard scoring results. In a specific embodiment of this disclosure, marginal gain is used as one method to measure model quality. The marginal gain is the difference between the rank correlation coefficient corresponding to the current ensemble model combination and the rank correlation coefficient corresponding to the model combination before adding the candidate model. When the marginal gain is non-negative, it indicates that the model quality has increased; when the marginal gain is negative, it indicates that the model quality has not increased. Those skilled in the art will understand that marginal gain is only an exemplary implementation method for measuring whether the model quality has increased, and other measurement methods that can reflect changes in model quality are all within the scope of protection of this disclosure.
[0006] Further, the step of using the ensemble members in the current ensemble model combination to perform ensemble scoring on the sample data to obtain the ensemble scoring result includes: The sample data are weighted by trust integration using the integration members in the current integration model combination. The integration weights of the integration members are predetermined based on the rank correlation coefficients of the integration score results and the standard score results in each dimension.
[0007] Further, based on the ensemble scoring results and the pre-set standard scoring results, determining whether the model quality of the current ensemble model combination has increased after adding the candidate model includes: Obtain the first rank correlation coefficient between the ensemble score result and the standard score result of the current ensemble model combination, and the second rank correlation coefficient between the ensemble score result and the standard score result of the model combination before adding the candidate model; The model quality of the current ensemble model combination is determined based on the first rank correlation coefficient and the second rank correlation coefficient.
[0008] Furthermore, the method also includes: Define a custom tool; the parameter mode of the custom tool is multidimensional structured data; The step of using the ensemble members in the current ensemble model combination to perform ensemble scoring on the sample data to obtain the ensemble scoring result includes: Under the premise of forcing the output of the integrated member to call the custom tool, the integrated member is used to score the sample data respectively; Based on the scores of each of the integrated members, the integrated score result is obtained; the structure of the integrated score result is the same as that of the multidimensional structured data.
[0009] Furthermore, the execution stop condition includes at least one of the following: the increase in model quality is less than a preset threshold; the number of families corresponding to the integrated member reaches a preset number of families; the total cost of calling the integrated member exceeds the budget.
[0010] Furthermore, the method also includes: Once the execution stop condition is met, output at least one of the following: final ensemble score result, model quality increase curve, and weights of ensemble members.
[0011] Further, the step of selecting a candidate model that belongs to a different vendor family from any of the integration members in the current integration model combination and adding it to the current integration model combination includes: Candidate models belonging to different vendor families from any of the integrated members are selected according to one or more of the following rules: priority is given to candidate models with low technical relevance to the family to which the integrated member belongs; priority is given to candidate models with low invocation costs; priority is given to candidate models with high historical weights.
[0012] Furthermore, the sample objects include at least one of occupations, products, and educational works.
[0013] Secondly, embodiments of this disclosure provide an integrated scoring device for multiple large language models, including: The partitioning module is configured to partition multiple large language models into different vendor families, the partitioning criteria of which include at least one of the following: the composition of the model’s pre-training data and / or the lineage of human feedback reinforcement learning training. The acquisition module is configured to acquire an initial integrated model combination and use the initial integrated model combination as the current integrated model combination. The selection module is configured to select a candidate model that belongs to a different vendor family from any integration member in the current integration model combination and add it to the current integration model combination; The scoring module is configured to use the ensemble members in the current ensemble model combination to perform ensemble scoring on the sample data and obtain an ensemble scoring result; wherein the sample data includes text prompts corresponding to the sample objects; The determination module is configured to determine, based on the integrated scoring results and the pre-set standard scoring results, whether the model quality of the current integrated model combination increases after adding the candidate model; The elimination module is configured to reduce the ensemble weight of the candidate model or eliminate the candidate model from the current ensemble if the model quality of the current ensemble model combination does not increase. The return module is configured to return to the selected module and repeat execution until the set stop condition is met.
[0014] Thirdly, embodiments of this disclosure provide an electronic device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in any of the above aspects.
[0015] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing a computer program, wherein the program, when executed by a processor, implements the steps of the method described in any of the preceding aspects.
[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0017] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a flowchart illustrating the integrated scoring method for multiple large language models proposed in the embodiments of this disclosure; Figure 2 This is a schematic diagram of the core process of the integrated scoring method for multiple large language models proposed in the embodiments of this disclosure; Figure 3 This is a structural block diagram of the integrated scoring device for multiple large language models proposed in the embodiments of this disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device suitable for implementing the scoring scale adjustment method according to an embodiment of the present disclosure. Detailed Implementation
[0018] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.
[0019] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.
[0020] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".
[0021] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0022] The inventors of this disclosure have discovered the following shortcomings in the prior art described in the background section: (1) Lack of methodology for model selection: Existing integration methods mostly select integration members based on availability or cost, without proposing a methodology principle of "joining in order of vendor family". This results in a lack of theoretical guidance for the construction of integration panels, making it difficult to maximize integration quality under limited cost.
[0023] (2) Low contribution of multiple models within the family and no guidance: In practice, the improvement of adding multiple models from the same vendor (different scale versions of the same vendor) is much smaller than that of adding cross-vendor models. However, there is no theoretical or empirical guidance in the existing literature. In engineering, there is often a waste of resources in the case of "adding a new model from the same family but no significant improvement in integration quality".
[0024] (3) Commercial closed-source APIs are excluded from integration: Open source LLM integration research is the mainstream, but commercial closed-source APIs (such as OpenAI GPT-4, Anthropic Claude, etc.) are difficult to be directly integrated into the standard ensemble pipeline due to the limitation of "no logits output and no fine-tuning interface"; existing ensemble methods cannot make full use of the powerful capabilities of commercial APIs, resulting in the upper limit of the integration panel's capabilities.
[0025] (4) Monotonicity has not been verified: Existing literature lacks systematic empirical verification of "monotonically improved integration quality after family-by-family addition", making it impossible to determine whether the expansion of the integrated panel still generates positive benefits, and also unable to provide data support for "when to stop adding new models".
[0026] (5) Structured JSON output is difficult to guarantee: When commercial APIs are added to the integration, the standard dialogue mode is difficult to output scores that strictly conform to the K-dimensional JSON schema, resulting in a high parsing failure rate and seriously affecting the stability and availability of the integrated system.
[0027] To address the aforementioned problems, embodiments of this disclosure provide a method, apparatus, storage medium, and electronic device for ensemble scoring of multiple large language models. This method divides multiple large language models according to vendor families, and sequentially adds new candidate models to the ensemble model combination in a cross-family manner. After each addition, it determines whether the model quality has increased based on the rank correlation coefficient between the ensemble scoring result and the standard scoring result, thereby achieving monotonic scaling expansion of the ensemble model combination. The technical solution of this disclosure is described in detail below with reference to the accompanying drawings.
[0028] Figure 1 This is a flowchart illustrating the integrated scoring method for multiple large language models proposed in this disclosure. Figure 1 As shown, this integrated scoring method for multiple large language models includes the following steps: Step S101: Divide multiple large language models into different vendor families, the classification of vendor families based on at least one of the following: the composition of the model's pre-training data and / or the lineage of human feedback reinforcement learning training.
[0029] In this step, multiple candidate large language models are categorized according to vendor families. The classification of vendor families is based on the composition of the model's pre-training data and / or the training lineage of Human Feedback Reinforcement Learning (RLHF). Models of different sizes or versions from the same vendor belong to the same vendor family; for example, Llama-3.1-70B and Llama-3.1-405B belong to the Meta family. Models from different vendors (such as Qwen and Llama) belong to different families. Vendor families may include, but are not limited to: the Meta family, the Qwen family, the DeepSeek family, the Anthropic / Claude family, the OpenAI / GPT family, the Google / Gemini family, and the Mistral family.
[0030] It should be noted that the classification of vendor families is not limited to the composition of pre-training data and the RLHF training lineage mentioned above. It can also include other factors that can reflect the technical relevance between models, such as model architecture characteristics, training corpus sources, alignment strategies, and whether they belong to the same vendor. Those skilled in the art can choose one or more criteria for family classification according to the actual application scenario, as long as models with high technical relevance can be grouped into the same vendor family and models with high technical independence (i.e., low technical relevance) can be grouped into different vendor families.
[0031] Step S102: Obtain the initial ensemble model combination and use the initial ensemble model combination as the current ensemble model combination.
[0032] In this step, the initial ensemble model combination can include ensemble members from a single vendor family (i.e., models categorized into different vendor families and used to score input sample data), or it can include ensemble members from several pre-defined vendor families. In one exemplary implementation, the initial ensemble model combination includes only one or more ensemble members from a single vendor family, such as the Llama-3.1-70B model from the Meta family. This initial ensemble model combination is used as the current ensemble model combination, and subsequent steps will progressively expand upon this current ensemble model combination.
[0033] In another exemplary embodiment, the initial ensemble model combination may also be an empty set, i.e., it does not contain any ensemble members, and subsequent steps begin with adding ensemble members from the first family. This disclosure does not strictly limit the specific composition of the initial ensemble model combination, and those skilled in the art can flexibly set it according to actual needs.
[0034] Step S103: Select a candidate model that belongs to a different vendor family than any of the integration members in the current integrated model combination, and add it to the current integrated model combination.
[0035] In this step, a candidate model is selected from vendor families that have not yet been added to the current integrated model portfolio. This candidate model belongs to a different vendor family than any integrated member in the current portfolio; that is, each added candidate model comes from a new vendor family, not one already existing in the current integrated model portfolio. This cross-family addition strategy is one of the core innovations of this disclosure, aiming to maximize the technological diversity among models in the integrated model portfolio and avoid redundant contributions between models of the same family.
[0036] In one alternative implementation, candidate models can be selected according to one or more of the following rules in combination: (1) prioritizing candidate models with low technical relevance to the family to which the current integration member belongs, wherein the technical relevance can be measured by quantitative indicators such as differences in pre-training data and architectural differences; (2) prioritizing candidate models with low invocation costs to maximize integration quality within a limited budget; and (3) prioritizing candidate models with high historical weights, i.e., models that have performed well in past integration tasks. The above selection rules can be used individually or in a weighted combination, and this disclosure does not limit this. As mentioned above, the level of technical relevance can be determined by at least one of the following factors: differences in the composition of pre-training data, differences in RLHF training lineage, differences in model architectural features, differences in the source of training corpus, differences in alignment strategies, and whether they belong to the same vendor, etc., which can reflect the technical relevance between models.
[0037] Step S104: Use the ensemble members in the current ensemble model combination to perform ensemble scoring on the sample data to obtain the ensemble scoring result; wherein, the sample data includes text prompts corresponding to the sample objects; the text prompts include the classification code, title and / or long text description of the sample objects.
[0038] In this step, the sample data includes text prompts corresponding to the sample objects. The sample objects can include at least one of occupations, products, and educational works. For example, in an occupational competency assessment scenario, the sample object is the occupation to be assessed, and the text prompt is a prompt describing the competencies of that occupation; in a product assessment scenario, the sample object is the product to be assessed, and the text prompt is a prompt describing the characteristics of that product; in an educational work assessment scenario, the sample object is the educational work to be assessed (such as a paper, assignment, etc.), and the text prompt is a prompt describing the content of that work. For example, when the sample object is an occupation, the text prompt includes the NOC (National Occupational Classification) code, the occupation title, and a long text description of the job content; when the sample object is an educational work, the text prompt may include the educational work's feature code, the work's title, and a content description.
[0039] Each ensemble member (i.e., each large language model ensemble member) in the current ensemble model combination scores the sample data individually, resulting in an individual score for each ensemble member. Then, the individual score results of each ensemble member are aggregated to obtain the ensemble score.
[0040] In one optional implementation, the aggregation method is a trust-weighted ensemble, where the ensemble weight of each member is normalized based on the rank correlation coefficients of the member's individual rating and the standard rating in each dimension after a non-negative weight transformation. Specifically, for the i-th member, its trust weight in the k-th dimension of the rating can be expressed as: w_{i,k} = max(0, ρ_{i,k}) / Σ_j max(0, ρ_{j,k}), where ρ_{i,k} is the Spearman rank correlation coefficient between the i-th member and the standard rating in the k-th dimension. This trust-weighted method can automatically reduce the influence of low-quality members and improve the reliability of the integrated rating results.
[0041] In one optional implementation, the integrated model combination includes at least one commercial closed-source large language model API. The commercial closed-source large language model API is integrated into the scoring panel via a tool calling mode. Specifically, a custom tool is defined, whose parameter mode is multi-dimensional structured data (e.g., a K-dimensional target JSON structure). When the commercial closed-source large language model API is called, the model is forced to output the score through this custom tool; that is, the model can only return the score result in the form of a tool call, and cannot output it in free text form. In this way, the structure of the integrated scoring result is the same as the multi-dimensional structured data, ensuring the strictness of the structured output.
[0042] For example, a tool with a custom name like "submit_ratings" or a semantically equivalent identifier can have an input pattern containing K target dimensions, each field being an ordinal or floating-point number. When calling a commercial closed-source large language model API, the tool selection parameter (e.g., tool_choice="submit_ratings") forces the model to output scores only through this tool. Taking occupations as an example, the K target dimensions can correspond to predefined assessment dimension systems such as CAP-25 (a 25-dimensional occupational ability assessment panel) or RIASEC-30 (a 30-dimensional occupational interest assessment). This tool call pattern effectively solves the problem of commercial closed-source APIs struggling to output strictly JSON-compliant scores and experiencing high parsing failure rates in standard dialogue modes.
[0043] Step S105: Based on the integrated scoring results and the pre-set standard scoring results, determine whether the model quality of the current integrated model combination has increased after adding the candidate model.
[0044] The standard score (i.e., the pre-set standard score, also known as the "golden signal" or "ground truth") can be a reference score given by human experts or authoritative evaluation systems for the sample data. By comparing the degree of consistency between the ensemble score and the standard score, the model quality of the current ensemble model combination can be measured.
[0045] In one optional implementation, the specific method for determining whether the model quality has increased includes: obtaining the first rank correlation coefficient between the ensemble score result and the standard score result of the current ensemble model combination (i.e., the model combination after adding candidate models), and the second rank correlation coefficient between the ensemble score result and the standard score result of the model combination before adding candidate models; and determining whether the model quality of the current ensemble model combination has increased based on the first rank correlation coefficient and the second rank correlation coefficient.
[0046] In a specific embodiment of this disclosure, marginal gain is described as one method for measuring whether the model quality has improved. The marginal gain is the difference between the first-rank correlation coefficient of the current ensemble model combination and the second-rank correlation coefficient of the model combination before adding the candidate model, i.e., Δρ = ρ_after. ρ_before. When the marginal gain Δρ ≥ 0, it indicates that the consistency between the integrated scoring result and the standard scoring result does not decrease after adding the candidate model, that is, the model quality increases (or at least does not decrease); when the marginal gain Δρ < 0, it indicates that the consistency between the integrated scoring result and the standard scoring result decreases after adding the candidate model, that is, the model quality does not increase. The embodiments of this disclosure empirically verify the monotonic scaling law of non-negativity of marginal gain, that is, when candidate models are added sequentially in a cross-family order, the marginal gain after each addition is statistically non-negative.
[0047] It should be noted that marginal gain is only one exemplary implementation for measuring whether model quality has increased. In other embodiments of this disclosure, model quality can also be measured by other methods, such as, but not limited to, changes in the Pearson correlation coefficient, changes in accuracy, changes in the F1 score, and changes in the KL divergence. Any technical solution that judges whether model quality has increased by comparing the change in consistency index between the integrated scoring results and the standard scoring results before and after adding the candidate model is within the scope of protection of this disclosure. This disclosure describes marginal gain (i.e., the difference in Spearman rank correlation coefficients) as a preferred implementation in detail because the Spearman rank correlation coefficient has good adaptability to the ordinal properties of scoring, and is simple to calculate and robust.
[0048] Furthermore, the first and second rank correlation coefficients can be obtained using Spearman's rank correlation coefficient, Kendall's rank correlation coefficient, etc. In a preferred embodiment, Spearman's rank correlation coefficient is used, and its calculation formula is: ρ = 1 − 6Σd 2 / (n(n 2 -1), where d is the rank difference between each pair of observations and n is the sample size. The Spearman rank correlation coefficient ranges from [ [1,1], the closer the value is to 1, the stronger the positive correlation between the integrated scoring result and the standard scoring result.
[0049] Step S106: If the model quality of the current ensemble model combination does not increase, then reduce the ensemble weight of the candidate model or remove the candidate model from the current ensemble model combination.
[0050] When step S105 determines that the model quality has not increased (e.g., the marginal gain is negative), it indicates that the newly added candidate model contributes negatively to the ensemble model combination. In this case, the candidate model needs to be processed. There are two processing methods: one is to reduce the weight of the candidate model in the ensemble, so as to reduce its impact on the ensemble scoring result; the other is to directly remove the candidate model from the current ensemble model combination, so that the ensemble model combination is restored to the state before the candidate model was added.
[0051] In one alternative implementation, when the marginal gain is negative but its absolute value is small (e.g., |Δρ| < ε, where ε is a preset small threshold), the ensemble weight of the candidate model can be reduced instead of being directly eliminated, in order to preserve its potential contributions in other dimensions. When the marginal gain is negative and its absolute value is large, the candidate model can be directly eliminated. This disclosure does not strictly limit the specific processing strategy, and those skilled in the art can flexibly choose according to the actual application scenario.
[0052] Step S107: Return to the step of selecting a candidate model that belongs to a different vendor family from any of the integration members in the current integration model combination, and repeat the process until the set stopping condition is met.
[0053] Specifically, in this embodiment of the disclosure, steps S103 to S106 constitute an iterative loop. In each iteration, a new cross-family candidate model is selected and added to the current ensemble model combination, the improvement in model quality is evaluated, and the ensemble model combination is adjusted based on the evaluation results. Then, the process returns to step S103 to continue the next iteration until the set stopping condition is met.
[0054] In one optional implementation, the stopping condition includes at least one of the following: (1) the improvement in model quality is less than a preset threshold, that is, the marginal gain Δρ of the most recent iteration is less than the preset threshold δ, indicating that the benefit of continuing to add new models is no longer significant; (2) the number of families corresponding to the integrated members reaches a preset number of families, that is, the current integrated model combination has covered a preset number of vendor families, achieving the goal of family diversity; (3) the total cost of calling the integrated members exceeds the budget, that is, the sum of the calling costs of all integrated members in the current integrated model combination exceeds the preset budget limit. The above stopping conditions can be used individually or in combination (for example, stopping when any one of the conditions is met, or stopping only when multiple conditions are met simultaneously).
[0055] In an optional implementation, after the stopping condition is met, the method of this disclosure further includes outputting at least one of the following: (1) the final ensemble score result, i.e. the ensemble score output of the current ensemble model combination on the sample data; (2) the model quality increase curve, i.e. the curve recording the change of model quality (e.g., marginal gain or cumulative rank correlation coefficient) after each iteration, used to visually demonstrate the quality evolution process of the ensemble model combination; (3) the weights of the ensemble members, i.e. the trust weight allocation of each ensemble member in the current ensemble model combination, used for direct reuse in subsequent scoring tasks.
[0056] For ease of understanding, the main technical solutions of the embodiments of this disclosure are summarized into five steps S1 to S5 below, see below. Figure 2 As shown: S1 - Vendor Family Classification: Candidate large language models are classified according to a predefined "vendor family". The vendor family is composed of the model's pre-training data and defined by the human feedback reinforcement learning training lineage. Models of different sizes or versions within the same family are considered to be in the same family, while models across families are considered to be in different families.
[0057] S2 – Cross-family sequential addition: Based on the initial integrated model combination, a candidate model is selected from the families that have not been added to obtain the expanded current integrated model combination; the selection rules can be based on strategies such as family diversity, cost, and availability, but the selection must satisfy cross-family rather than intra-family.
[0058] S3 – Model Quality Assessment and Monotonicity Validation: After each addition, calculate the rank correlation coefficient between the ensemble score and the standard score of the current ensemble model combination, and calculate the marginal gain (i.e., the difference between the current rank correlation coefficient and the previous rank correlation coefficient). Empirically verify the monotonic scaling law that the marginal gain is non-negative. If it is found that the addition of a certain family results in a negative marginal gain (i.e., the model quality does not increase), then adjust the candidate model to a low weight or remove it.
[0059] S4 – Integration of Commercial Closed-Source API Tools: For the integration members of commercial closed-source large language model APIs in the integrated model portfolio, a custom tool with a parameter mode defined as multi-dimensional structured data is used to integrate the model into the scoring panel by forcing the model to output scores through this tool, ensuring the strictness of the structured output.
[0060] S5 – Iteration and Stopping: Repeat S2 to S4 until the stopping condition is met (such as marginal gain being less than the threshold, the number of families reaching the preset value, cost exceeding the budget, etc.), and output the final ensemble score, model quality increase curve, and ensemble member weights.
[0061] Through the above technical solutions, the embodiments of this disclosure achieve the following technical effects: (1) The integration methodology principle of "adding according to the order of the manufacturer family" is proposed, which overcomes the lack of methodology in the model selection in the existing technology, and makes the construction of the integrated model combination have clear theoretical guidance, which can maximize the integration quality under limited cost.
[0062] (2) By adopting the cross-family joining strategy, redundant contributions from the same family model are avoided. The monotonic scaling law of non-negative marginal gain is empirically verified, providing data support for "when to stop adding new models" and avoiding resource waste.
[0063] (3) By integrating commercial closed-source large language model APIs into the scoring panel through the tool call mode, the problem of commercial APIs being difficult to integrate due to "no logits output and no fine-tuning interface" is solved. This fully utilizes the powerful capabilities of commercial APIs and increases the upper limit of the ability to combine integrated models.
[0064] (4) By using the multidimensional structured data parameter mode of the custom tool, the model is forced to output scores in a structured manner, which solves the problem of high JSON output parsing failure rate in the standard dialogue mode and ensures the stability and availability of the integrated system.
[0065] The implementation process of the technical solution disclosed herein is illustrated below through exemplary embodiments.
[0066] Example 1: Occupational Competency Assessment Scenario The following uses CAP-25 (25-dimensional professional competence assessment panel) as an example to illustrate the specific application of the embodiments of this disclosure.
[0067] In this embodiment, the sample consists of approximately 500 Canadian NOC 2021 occupations, and the text prompts are descriptions of the requirements for each occupation across 25 competency dimensions. The standard scoring results are reference scores (0-5 scale) derived from authoritative occupational competency databases (such as OaSIS in Canada, assessed by occupational analysts). Candidate large language models come from four vendor families: Claude-Sonnet (commercial closed-source API, integrated via tool_use mode) from the Anthropic / Claude family, Qwen3.5-122B from the Qwen family, Llama-3.1-70B from the Meta family, and DeepSeek-V2-Lite from the DeepSeek family.
[0068] Step 1: Divide the above candidate large language models into four vendor families: Anthropic / Claude family, Qwen family, Meta family, and DeepSeek family. Different sizes or versions of models within the same vendor family are counted as the same family.
[0069] Step 2: Obtain the initial ensemble model combination, for example, Claude-Sonnet (ensembled via tool_use mode) containing only the Anthropic / Claude family. Use this initial ensemble model combination as the current ensemble model combination.
[0070] Step 3: Select a candidate model that belongs to a different vendor family than any of the integrated members in the current integrated model combination. For example, following the rule of "prioritizing families with low technical relevance", select Qwen3.5-122B from the Qwen family as a candidate model and add it to the current integrated model combination.
[0071] Step 4: Use Claude-Sonnet and Qwen3.5-122B from the current ensemble model combination to score the sample data for the aforementioned occupations. For the commercial closed-source API (i.e., Claude-Sonnet), define a custom tool named "submit_ratings" with parameters in a 25-dimensional target dimension JSON structure (each dimension being a value from 0 to 5). The `tool_choice` parameter forces the model to output scores only through this tool. Aggregate the scores from each ensemble member using a trust-weighted ensemble method to obtain the ensemble score result.
[0072] Step 5: Calculate the Spearman rank correlation coefficient (first rank correlation coefficient) between the ensemble score and the standard score of the current ensemble model combination, and the Spearman rank correlation coefficient (second rank correlation coefficient) between the ensemble score and the standard score of the model combination before adding Qwen3.5-122B (Claude-Sonnet only). Calculate the marginal gain Δρ = ρ_after ρ_before. If Δρ ≥ 0, the model quality increases, and Qwen3.5-122B is retained; if Δρ < 0, the model quality does not increase, and the ensemble weights of Qwen3.5-122B are reduced or removed.
[0073] Step 6: Return to Step 3 and select the next cross-family candidate model (such as Llama-3.1-70B from the Meta family, followed by DeepSeek-V2-Lite from the DeepSeek family) to add, repeating the above process. Continue until the stopping conditions are met, such as all four families have been added, or the marginal gain is less than the preset threshold δ=0.005, or the total cost of the call exceeds the budget.
[0074] Step 7: Output the final ensemble score, the model quality improvement curve (i.e., the Spearman rank correlation coefficient change curve after each addition), and the trust weight of each ensemble member.
[0075] In the experiments of this embodiment, the scores were added sequentially across families in the order of Claude→Qwen→Llama→DeepSeek. The Spearman rank correlation coefficients between the integrated scoring results and the standard scoring results were as follows: family 1 0.721, family 2 0.748, family 3 0.753, and family 4 0.790. The marginal gain Δρ after each addition was non-negative (+0.027, +0.005, and +0.037 respectively), verifying the monotonic scaling law. Finally, the 0.790 of the four-family integration was significantly better than the 0.721 of the single-family integration. Among them, the perceptual dimensions (such as S2 auditory discrimination) showed the greatest improvement after adding the commercial closed-source API (marginal gain +0.202). At the same time, through the tool_use mode, the scoring parsing success rate of the commercial closed-source API (Claude-Sonnet) reached 100%, with no JSON parsing failures.
[0076] Example 2: Educational Works Evaluation Scenario The following uses the evaluation of educational works (such as student papers) as an example to illustrate another application scenario of the embodiments of this disclosure.
[0077] In this embodiment, the sample objects are student-submitted papers, and the text prompts are descriptions of the paper's evaluation dimensions (such as originality, logic, writing quality, citation accuracy, etc.). The standard scoring results are the teacher's scores for each paper on each evaluation dimension. Candidate large language models include open-source models (Meta family, Qwen family, DeepSeek family) and commercial closed-source APIs (Claude family, GPT family).
[0078] Step 1: Divide the candidate large language models into 5 vendor families.
[0079] Step 2: Obtain the initial ensemble model combination, such as Llama-3.1-70B which includes the Meta family.
[0080] Steps 3 to 7: Following a similar process to Example 1, candidate models are added across families one by one. After each addition, the marginal gain is calculated to verify whether the model quality has increased, and the ensemble model combination is adjusted based on the results.
[0081] In this embodiment, the parameter mode of the custom tool is defined as 4-dimensional structured data (originality, logic, writing quality, and citation standardization), with each dimension being a floating-point value from 1 to 10. Commercial closed-source APIs output scores through this custom tool, ensuring the rigor of the structured output. This embodiment is an exemplary application scenario: following the same monotonic addition process across families, the expected consistency between the evaluation results of cross-family integration and teacher scores is higher than that of a single model and multi-model integration within the same family (the values in this paragraph are illustrative examples and not based on actual measurements on a specific dataset).
[0082] Figure 3 This is a structural block diagram of the integrated scoring device for multiple large language models proposed in the embodiments of this disclosure. For example... Figure 3 As shown, the integrated scoring device for multiple large language models includes: The partitioning module 301 is configured to partition multiple large language models into different vendor families, wherein the partitioning criteria include at least one of the following: the composition of the model’s pre-training data and / or the lineage of human feedback reinforcement learning training.
[0083] The acquisition module 302 is configured to acquire an initial integrated model combination and use the initial integrated model combination as the current integrated model combination.
[0084] Selection module 303 is configured to select a candidate model that belongs to a different vendor family than any integration member in the current integration model combination and add it to the current integration model combination.
[0085] The scoring module 304 is configured to use the ensemble members in the current ensemble model combination to perform ensemble scoring on the sample data and obtain an ensemble scoring result; wherein the sample data includes text prompts corresponding to the sample objects.
[0086] The determination module 305 is configured to determine, based on the integrated scoring results and the pre-set standard scoring results, whether the model quality of the current integrated model combination increases after adding the candidate model.
[0087] The elimination module 306 is configured to reduce the integration weight of the candidate model or eliminate the candidate model from the current integrated model combination if the model quality of the current integrated model combination does not increase.
[0088] Return module 307 is configured to return to the selected module and repeat execution until the set stop condition is met.
[0089] This multi-language model ensemble scoring device corresponds to the multi-language model ensemble scoring method mentioned above. For details, please refer to the description of the multi-language model ensemble scoring device mentioned above, which will not be repeated here.
[0090] It should be noted that the modules described in the embodiments of this disclosure can be implemented by software, hardware, or a combination of both. In one implementation, the modules can be located in the processor, and the names of the modules do not necessarily limit the module itself under certain circumstances.
[0091] Figure 4 This is a schematic diagram of the structure of an electronic device suitable for implementing the scoring scale adjustment method according to an embodiment of the present disclosure.
[0092] like Figure 4 As shown, the electronic device 400 includes a processing unit 401, which can be implemented as a CPU, GPU, FPGA, NPU, or other processing unit. The processing unit 401 can execute various processes according to any of the methods described above in this disclosure, based on a program stored in the read-only memory (ROM) 402 or a program loaded from the storage portion 408 into the random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the electronic device 400. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0093] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.
[0094] In particular, according to embodiments of this disclosure, any of the methods described above in the embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing any of the methods in the embodiments of this disclosure. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411.
[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0096] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The described units or modules can also be located in a processor, and the names of these units or modules do not necessarily constitute a limitation on the unit or module itself.
[0097] In another aspect, this disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs that are used by one or more processors to perform the methods described in this disclosure.
[0098] Another embodiment of this disclosure relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.
[0099] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by specifying relevant hardware through a program. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0100] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A scoring method integrating multiple large language models, characterized in that, Includes the following steps: Multiple large language models are divided into different vendor families, and the classification of vendor families is based on at least one of the following: the composition of the model's pre-training data and / or the lineage of human feedback reinforcement learning training. Obtain an initial ensemble model combination and use the initial ensemble model combination as the current ensemble model combination; Select a candidate model that belongs to a different vendor family from any of the integration members in the current integration model combination, and add it to the current integration model combination; The sample data is integrated and scored using the ensemble members in the current ensemble model combination to obtain the ensemble score result; wherein, the sample data includes text prompts corresponding to the sample objects; Based on the integrated scoring results and the pre-set standard scoring results, determine whether the model quality of the current integrated model combination has increased after adding the candidate model; If the quality of the current ensemble model combination does not increase, then reduce the ensemble weight of the candidate model or remove the candidate model from the current ensemble model combination; Return to the step of selecting a candidate model that belongs to a different vendor family from any of the integration members in the current integration model combination, and repeat the process until the set stopping condition is met.
2. The method according to claim 1, characterized in that, The step of using the ensemble members in the current ensemble model combination to perform ensemble scoring on the sample data to obtain the ensemble scoring result includes: The sample data are weighted by trust integration using the integration members in the current integration model combination. The integration weights of the integration members are predetermined based on the rank correlation coefficients of the integration score results and the standard score results in each dimension.
3. The method according to claim 1, characterized in that, Based on the ensemble scoring results and the pre-set standard scoring results, determine whether the model quality of the current ensemble model combination has increased after adding the candidate model, including: Obtain the first rank correlation coefficient between the ensemble score result and the standard score result of the current ensemble model combination, and the second rank correlation coefficient between the ensemble score result and the standard score result of the model combination before adding the candidate model; The model quality of the current ensemble model combination is determined based on the first rank correlation coefficient and the second rank correlation coefficient.
4. The method according to claim 1, characterized in that, The method further includes: Define a custom tool; the parameter mode of the custom tool is multidimensional structured data; The sample data is ensembled and scored using the ensemble members in the current ensemble model combination to obtain the ensemble score result, including: Under the premise of forcing the output of the integrated member to call the custom tool, the integrated member is used to score the sample data respectively; Based on the scores of each of the integrated members, the integrated score result is obtained; the structure of the integrated score result is the same as that of the multidimensional structured data.
5. The method according to claim 1, characterized in that, The execution stop conditions include at least one of the following: the increase in model quality is less than a preset threshold; the number of families corresponding to the integrated members reaches a preset number of families; the total cost of calling the integrated members exceeds the budget.
6. The method according to claim 1, characterized in that, The method further includes: Once the execution stop condition is met, output at least one of the following: final ensemble score result, model quality increase curve, and weights of ensemble members.
7. The method according to claim 1, characterized in that, The step of selecting a candidate model that belongs to a different vendor family from any of the integration members in the current integration model combination and adding it to the current integration model combination includes: Candidate models belonging to different vendor families from any of the integrated members are selected according to one or more of the following rules: priority is given to candidate models with low technical relevance to the family to which the integrated member belongs; priority is given to candidate models with low invocation costs; priority is given to candidate models with high historical weights.
8. The method according to any one of claims 1-2 and 4-7, characterized in that, The sample objects include at least one of occupations, products, and educational works; the text prompts include the category code, title, and / or long text description of the sample objects.
9. A scoring device integrating multiple large language models, characterized in that, include: The partitioning module is configured to partition multiple large language models into different vendor families, the partitioning criteria of which include at least one of the following: the composition of the model’s pre-training data and / or the lineage of human feedback reinforcement learning training. The acquisition module is configured to acquire an initial integrated model combination and use the initial integrated model combination as the current integrated model combination. The selection module is configured to select a candidate model that belongs to a different vendor family from any integration member in the current integration model combination and add it to the current integration model combination; The scoring module is configured to use the ensemble members in the current ensemble model combination to perform ensemble scoring on the sample data and obtain an ensemble scoring result; wherein the sample data includes text prompts corresponding to the sample objects; The determination module is configured to determine, based on the integrated scoring results and the pre-set standard scoring results, whether the model quality of the current integrated model combination increases after adding the candidate model; The elimination module is configured to reduce the ensemble weight of the candidate model or eliminate the candidate model from the current ensemble if the model quality of the current ensemble model combination does not increase. The return module is configured to return to the selected module and repeat execution until the set stop condition is met.
10. A computer device, comprising a memory and a processor, characterized in that, When the processor executes the computer program in the memory, it implements the steps of the method according to any one of claims 1 to 8.
11. A computer-readable storage medium storing a computer program, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.
