A context learning example adaptive ranking method based on performance supervision

Abstract

The application provides a context learning example adaptive ranking method based on performance supervision, comprising the following steps: for each query sample in a training query set, retrieving a fixed example set from an example library, defining a ranking space through arbitrary ranking, and sampling a subset of candidate rankings in the ranking space; for each sampled ranking, calling a target LLM to obtain a prediction result, and calculating the prediction result with a real answer to obtain a task index score; training a ranking predictor to learn a mapping relationship of "query-ranking-real performance" based on query embedding vectors and ordered context embedding vectors, and obtaining a trained ranking predictor; in an online inference stage, using the trained ranking predictor to perform adaptive ranking on the example set, and selecting an optimal ranking. The application solves the problems of too large data volume in ranking space with factorial explosion, unstable correspondence between heuristic / proxy indicators and real performance, and lack of learnable ranking selector.

Description

Technical Field

[0001] This invention relates to the field of context learning technology for large language models, and specifically to a performance-supervised adaptive sorting method for context learning examples. Background Technology

[0002] In In-Context Learning (ICL) of large language models, the common practice is to retrieve k examples (demonstrations) from a retrieval database for a test query sample x, concatenate them with the query to form prompt words, and input them into the target large model to obtain the predicted output. To improve performance, research and engineering often optimize in the following ways:

[0003] Retrieve example set: Use BM25 or vector similarity to retrieve the top-k; Construct prompt template: Organize the input text and labels of the examples according to a fixed template; Example ordering: Sort the retrieved examples according to a certain rule (such as heuristics such as similarity from high to low, entropy, length, category coverage, etc.) and then concatenate them into prompt words.

[0004] Among these factors, example ranking has a significant impact on ICL performance: when the example set is fixed, different orders lead to significant performance differences, so more and more works are trying to rank examples using surrogate scores (similarity / uncertainty, etc.). However, most of these rankings are still heuristic rules and lack supervision and adaptive mechanisms that directly address real-world performance.

[0005] The existing technology has the following main shortcomings:

[0006] (1) The sorting space is factorially explosive, and the optimal order cannot be directly selected based on actual performance.

[0007] Where k! is the number of permutations of k examples. Since k! grows extremely rapidly, it would be prohibitively costly to actually infer and calculate metrics for each ranking using a large target model, making it difficult to implement "finding the optimal ranking based on real performance" in existing solutions.

[0008] (2) Heuristic / surrogate metrics do not correspond consistently to real performance and lack adaptability to target models and specific inputs. Many methods use surrogate metrics such as similarity, example difficulty, or diversity for ranking, but these metrics are not always strongly correlated with the final task metrics (Accuracy / BLEU / ROUGE / MSE, etc.), and the optimal ranking varies greatly under different target LLMs, different datasets, and different query inputs. Therefore, the same set of heuristic rules is difficult to generalize stably.

[0009] (3) The lack of a “learnable sorting selector” makes it difficult to systematically reuse and extend sorting in engineering. Existing sorting often relies on manual rules and cannot learn from a large number of historical samples which sorting will bring higher metrics, nor can it form transferable and pluggable components (e.g., it can be used when replacing the retrieval tool, changing the target LLM, or changing the task). Summary of the Invention

[0010] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide a performance-supervised context learning example adaptive sorting method, which solves the technical problems of excessively large sorting space with factorial explosion and data volume, unstable correspondence between heuristic / surrogate indicators and real performance, and lack of learnable sorting selectors in the prior art.

[0011] To achieve the above-mentioned technical objectives, in a first aspect, the present invention provides a performance-supervised context learning example adaptive ranking method, comprising the following steps:

[0012] For each query sample in the training query set, retrieve the results from the example library. This provides a fixed set of examples.

[0013] The fixed set of examples is arbitrarily sorted to define a sorting space, which has k! candidate permutations. For each query sample, only a subset of candidate permutations is sampled from the sorting space, and the subset of candidate permutations includes M sampled permutations.

[0014] For each sample sort, construct the corresponding ICL prompt word and call the target LLM to obtain the prediction result. Calculate the task indicator score by comparing the prediction result with the actual answer.

[0015] To reduce the computational complexity of training and subsequent inference, query samples and examples are encoded once to generate query embedding vectors, single example embedding vectors, and ordered context embedding vectors.

[0016] Train the ranking predictor by learning the mapping relationship between "query-ranking-real performance" based on the query embedding vector and the ordered context embedding vector, and obtain the trained ranking predictor.

[0017] During the online inference phase, the trained ranking predictor is used to adaptively rank the example set and select the optimal ranking.

[0018] Compared with the prior art, the beneficial effects of the present invention include:

[0019] The core technical problem that this patent aims to solve is: given a query and the fixed set of retrieved examples Under the premise of not enumerating In this case, adaptive selection of example ranking that maximizes the actual downstream performance metrics And ensure that the number of inference calls to the target LLM remains at one (or the same order of magnitude as standard ICL).

[0020] To this end, this patent proposes an example ranking learning framework based on performance-based supervision: in the offline stage, a target LLM is used to evaluate a small number of candidate rankings to obtain real performance scores as supervision, and a lightweight "ranking predictor" is trained; in the online stage, candidate rankings are quickly scored in the embedding space and the optimal ranking is selected, and the target LLM is called only once to output the result, thus balancing performance and cost.

[0021] According to some embodiments of the present invention, after selecting the optimal sort, the method further includes the following steps:

[0022] The optimal sort is then concatenated with the test query samples according to the ICL suggestion term template to form the final suggestion terms.

[0023] According to some embodiments of the present invention, arbitrarily sorting the fixed set of examples defines a sorting space, the sorting space having k! candidate permutations, including the following steps:

[0024] For a fixed set of examples Arbitrary sorting can be represented as applying a permutation operator to it. The space of all possible sortings is defined as:

[0025]

[0026] According to some embodiments of the present invention, for each query sample, only a subset of candidate sorts is sampled in the sorting space, the subset of candidate sorts comprising M sampled sorts, including:

[0027] A sampling strategy is adopted: for each query, only a subset of candidate rankings is sampled.

[0028]

[0029] That is, "each query needs to be evaluated" The cost of "sorting" is reduced to "evaluation per query sample". "Sorting".

[0030] According to some embodiments of the present invention, for each sample sorting, a corresponding ICL prompt word is constructed and the target LLM is called to obtain the prediction result. The prediction result is then compared with the actual answer to obtain the task indicator score, including the following steps:

[0031] Sort each sample Construct the corresponding ICL prompt words and call the target LLM to obtain the prediction. Then compare with the real answer Calculate task indicator scores:

[0032]

[0033] in The task evaluation metrics include: MSE for regression and BLEU for translation, and scores. This refers to "the true performance of the ranking on the query," which is fundamentally different from heuristic surrogate scores.

[0034] According to some embodiments of the present invention, a one-time encoding process is performed on query samples and examples to generate query embedding vectors, single example embedding vectors, and ordered context embedding vectors, including the following steps:

[0035] The query sample is encoded only once:

[0036]

[0037] Each example is coded only once, with the example input and tags formatted according to the template. splicing:

[0038]

[0039] For a certain sort The corresponding example embeddings are concatenated in order to obtain the sorted "ordered context embedding vector":

[0040]

[0041] The difference between different sorting methods lies only in the concatenation order, thus replacing repetitive encoding with vector concatenation, significantly reducing complexity.

[0042] According to some embodiments of the present invention, a training-completed ranking predictor is obtained by learning the mapping relationship between "query-ranking-real performance" based on the query embedding vector and the ordered context embedding vector, including the following steps:

[0043] Define the ranking predictor The input is "query embedding + sort embedding", and the output is the prediction performance score of the sort.

[0044]

[0045] The training goal is to make Fitting the true performance .

[0046] According to some embodiments of the present invention, in the online inference phase, the trained ranking predictor is used to adaptively rank the example set and select the optimal ranking, including the following steps:

[0047] For test query retrieval, construct the embedding representation of the candidate ranking, and perform the same operations as in the offline training phase for the test query to be predicted:

[0048] K examples are retrieved from the example library to form a fixed example set; the test query is encoded once to obtain the embedding vector of the test query; the k retrieved examples are concatenated and encoded once according to the offline training template to obtain the embedding vector of a single example; a small number of candidate sorts are generated for this example set, and the example embedding vectors are concatenated according to the sort order to obtain the ordered context embedding vector of each candidate sort.

[0049] Predictors score and select the optimal ranking:

[0050] The "test query embedding vector and the ordered context embedding vector of each candidate sort" are input into the offline trained sorting predictor, which outputs a prediction performance score for each candidate sort. The optimal sort is selected according to the type of task evaluation index: if the index is "the larger the better", the sort with the highest prediction score is selected; if the index is "the smaller the better", the sort with the lowest prediction score is selected.

[0051] Secondly, the present invention provides a performance-supervised context learning example adaptive ranking system, which applies the performance-supervised context learning example adaptive ranking method as described in any one of the first aspects, including the following steps:

[0052] The example set building module retrieves the example set from the example library for each query sample in the training query set. This provides a fixed set of examples.

[0053] The subset sampling module arbitrarily sorts the fixed set of examples to define a sorting space, which has k! candidate permutations. For each query sample, only one subset candidate permutation is sampled in the sorting space, and the subset candidate permutation includes M sampled permutations.

[0054] The indicator score calculation module sorts each sample, constructs corresponding ICL prompt words, calls the target LLM to obtain the prediction result, and calculates the task indicator score by comparing the prediction result with the actual answer.

[0055] The encoding module performs a one-time encoding process on the query samples and examples to reduce the computational complexity of training and subsequent inference, generating query embedding vectors, single example embedding vectors, and ordered context embedding vectors.

[0056] Predictor training module, training the ranking predictor: learns the mapping relationship between "query-ranking-real performance" based on the query embedding vector and the ordered context embedding vector, and obtains the trained ranking predictor;

[0057] The adaptive sorting module, during the online inference phase, uses the trained sorting predictor to adaptively sort the example set and select the optimal sort.

[0058] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0059] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein the abstract drawings are to be completely consistent with one of the drawings in the specification:

[0060] Figure 1 A flowchart illustrating an example of an adaptive sorting method based on performance supervision and context learning, provided as an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0062] It should be noted that although functional modules are divided in the system diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0063] Reference Figure 1 , Figure 1 A flowchart illustrating an example of an adaptive sorting method based on performance supervision and context learning, provided as an embodiment of the present invention.

[0064] In one embodiment, the performance-supervised context learning example adaptive ranking method includes the steps of: for each query sample in the training query set, retrieving from the example library... A fixed set of examples is obtained. The fixed set of examples is then arbitrarily sorted, defining a sorting space with k! candidate permutations. For each query sample, only a subset of candidate permutations is sampled from the sorting space, consisting of M sampled permutations. For each sampled permutation, corresponding ICL (Information Classification) prompts are constructed, and the target LLM (Limited Context Management) is called to obtain the prediction result. The prediction result is then compared with the true answer to calculate the task performance score. To reduce the computational complexity of training and subsequent inference, query samples and examples are encoded once to generate query embedding vectors, single example embedding vectors, and ordered context embedding vectors. A sorting predictor is trained, learning the mapping relationship between "query-sorting-true performance" based on the query embedding vectors and ordered context embedding vectors, resulting in a trained sorting predictor. During the online inference phase, the trained sorting predictor is used to adaptively sort the example set, selecting the optimal sort.

[0065] This patented adaptive ranking method for context-learning examples, based on performance supervision, addresses core issues in existing context-learning example ranking systems, such as factorial space explosion, unstable correlation between heuristic surrogate metrics and real performance, and a lack of learnable ranking components. Through targeted design, it achieves several significant advantages. Instead of traversing the k!-order permutation space, this method samples only a small number of candidate rankings for each query sample, directly using real task metrics as supervision signals. This fundamentally solves the problem of excessively large ranking spaces and the inability to select the optimal ranking based on real performance, significantly reducing the number of inferences and computational costs for large target models, making performance-oriented ranking optimization feasible for engineering implementation.

[0066] Meanwhile, this patent's method abandons the traditional heuristic ranking approach based on indirect proxy metrics such as similarity, entropy, and length. Instead, it directly learns the mapping relationship between query embeddings, ordered context embeddings, and real performance by training a ranking predictor. This ensures that the ranking strategy is highly aligned with the task objective, enabling it to adaptively output the optimal ranking based on different queries, datasets, and target large models, effectively improving the stability and generalization ability of context learning. Furthermore, by encoding queries and examples once to generate reusable embedding vectors, the computational complexity of training and online inference is further reduced. The ranking logic is encapsulated into a learnable ranking predictor module, achieving decoupling from the retrieval unit, target large model, and specific tasks. This forms a pluggable, transferable, and continuously iteratively optimized standardized component, solving the problem of traditional manual ranking rules being difficult to reuse, extend, and upgrade. It possesses good adaptability and practical value in industrial-grade systems.

[0067] The following is combined Figure 1 The implementation principle is explained in detail. This patent consists of an offline training stage and an online inference stage, and is formally described and supported by equations (4) to (12).

[0068] (a) Offline training phase: Constructing a "ranking-real performance" supervised training method and training the ranking predictor:

[0069] Step 1: Data Preparation and Construction of Example Collection for Retrieval

[0070] For each query sample in the training query set Found in the example library Example, and reindexed as follows:

[0071]

[0072] In subsequent steps, this set remains "the example set is fixed, only the sorting is changed".

[0073] correspond Figure 1 Location: Search module (Top- (demonstrations).

[0074] Step 2: Define the sorting space and explain the fundamental reason for "k! explosion";

[0075] For a fixed set of examples Arbitrary sorting can be represented as applying a permutation operator to it. The space of all possible sorts is defined as:

[0076]

[0077] This step clarifies that sorting selection is not about "selecting examples," but rather from... Choosing one from a candidate permutation is not feasible through direct enumeration, which is one of the core bottlenecks of existing technologies.

[0078] correspond Figure 1 Location: Ordering space (k!) / Candidates.

[0079] Step 3: Candidate sorting and sampling (approximately replacing k! with M candidates);

[0080] To avoid exhaustive search, this patent employs a sampling strategy: for each query, only a subset of candidate rankings is sampled.

[0081]

[0082] That is, "each query needs to be evaluated" The cost of "one sort" is reduced to "per query evaluation". "Sorting".

[0083] correspond Figure 1Position: Candidate ordering sampling (M candidates).

[0084] Step 4: Performance-Based Supervision: Obtain supervision scores using the target LLM real-world evaluation;

[0085] Sort each sample Construct the corresponding ICL prompt words and call the target LLM to obtain the prediction. Then compare with the real answer Calculate task indicator scores:

[0086]

[0087] in This is a task evaluation indicator (such as MSE for regression and BLEU for translation). This score... This refers to "the true performance of the ranking on the query," which is fundamentally different from heuristic surrogate scores.

[0088] correspond Figure 1 Location: Target LLM evaluation → metric score .

[0089] Step 5: Embedding-first representation: Avoid repeatedly encoding the same example under different orderings;

[0090] To reduce training / inference overhead, this patent adopts an embedding-first approach:

[0091] • Encode the query only once:

[0092]

[0093] • Encode each example only once (compile the example input and labels according to the template). splicing):

[0094]

[0095] • For a certain sorting The corresponding example embeddings are concatenated in order to obtain the sorted "ordered context embeddings":

[0096]

[0097] This design ensures that the difference between different sortings is only reflected in the concatenation order, thus replacing repeated encoding with vector concatenation and significantly reducing complexity.

[0098] correspond Figure 1 Location: Encoder + Ordered embedding construction.

[0099] Step 6: Train the Ordering Predictor: Regress and predict the true performance of each order;

[0100] Define the ranking predictor The input is "query embedding + sort embedding", and the output is the prediction performance score of the sort.

[0101]

[0102] The training goal is to make Fitting the true performance (Described as standard regression loss or equivalent pairwise preference learning). This transforms "ranking selection" into a learnable performance prediction problem: learning "which permutations are more likely to yield higher metrics given a query and a set of examples."

[0103] correspond Figure 1 Position: Ordering predictor training (supervised by ).

[0104] (ii) Online inference phase: The optimal sort is quickly selected in the embedding space, and the target LLM is called only once.

[0105] Step 7: During testing, retrieve and construct the embedding representations of candidate rankings;

[0106] For test queries The search yielded top- Example collection And calculated consistently with the training phase:

[0107] • Query embedding ;

[0108] • Each example embeds (Same as (Formula 5));

[0109] Arbitrary candidate sorting ordered embedding Same as (Formula 6)).

[0110] Step 8: Use the predictor to score and select the optimal ranking;

[0111] For each in the candidate sort set Predict its effectiveness And select the optimal sort:

[0112]

[0113] in This indicates that if the indicator is "the larger the better", then take... If the error-related indicator is "the smaller the better", then take... .

[0114] correspond Figure 1 Location: Scoring candidates → select .

[0115] Step 9: Construct the final prompt words using the optimal sorting and call the target LLM output;

[0116] Finally, use the optimal sorting. Constructing ICL prompts, the final prediction is obtained by calling the target LLM only once:

[0117]

[0118] This allows for better downstream real-world performance while maintaining the target LLM inference call count ≈ standard ICL (one call).

[0119] correspond Figure 1 Location: Final prompt → Target LLM → output.

[0120] In the online inference phase, the trained ranking predictor adaptively ranks the example set and selects the optimal ranking. This includes the following steps: test query retrieval, constructing embedding representations of candidate rankings, and performing the same operations as in the offline training phase for the test query to be predicted: retrieving k examples from the example library to form a fixed example set; encoding the test query once to obtain its embedding vector; concatenating and encoding the k retrieved examples according to the offline training template to obtain the embedding vector of a single example; generating a small number of candidate rankings for this example set, and then concatenating the example embedding vectors according to the ranking order to obtain the ordered context embedding vector of each candidate ranking; predictor scoring and selecting the optimal ranking: inputting the "test query embedding vector and the ordered context embedding vectors of each candidate ranking" into the offline-trained ranking predictor, which outputs a prediction performance score for each candidate ranking; selecting the optimal ranking based on the type of task evaluation metric: if the metric is "the higher the better," selecting the ranking with the highest prediction score; if the metric is "the smaller the better," selecting the ranking with the lowest prediction score.

[0121] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described terminal embodiment, such that the processor performs the performance-supervised context learning example adaptive sorting method in the above-described embodiment.

[0124] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0125] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

[0126] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A performance-supervised context learning-based adaptive sorting method, characterized in that, Including the following steps: For each query sample in the training query set, retrieve the results from the example library. This provides a fixed set of examples. The fixed set of examples is arbitrarily sorted to define a sorting space, which has k! candidate permutations. For each query sample, only a subset of candidate permutations is sampled from the sorting space, and the subset of candidate permutations includes M sampled permutations. For each sample sort, construct the corresponding ICL prompt word and call the target LLM to obtain the prediction result. Calculate the task indicator score by comparing the prediction result with the actual answer. To reduce the computational complexity of training and subsequent inference, query samples and examples are encoded once to generate query embedding vectors, single example embedding vectors, and ordered context embedding vectors. Train the ranking predictor by learning the mapping relationship between "query-ranking-real performance" based on the query embedding vector and the ordered context embedding vector, and obtain the trained ranking predictor. During the online inference phase, the trained ranking predictor is used to adaptively rank the example set and select the optimal ranking.

2. The performance-supervised context learning example adaptive ranking method according to claim 1, characterized in that, After selecting the optimal sort, the following steps are also included: The optimal sort is then concatenated with the test query samples according to the ICL suggestion term template to form the final suggestion terms.

3. The performance-supervised context learning example adaptive ranking method according to claim 1, characterized in that, To arbitrarily sort the fixed set of examples, a sorting space is defined, wherein the sorting space has k! candidate permutations, including the following steps: For a fixed set of examples Arbitrary sorting can be represented as applying a permutation operator to it. The space of all possible sortings is defined as:

4. The performance-supervised context learning example adaptive ranking method according to claim 3, characterized in that, For each query sample, only a subset of candidate sorts is sampled in the sorting space, the subset of candidate sorts comprising M sampled sorts, including: A sampling strategy is adopted: for each query, only a subset of candidate rankings is sampled. That is, "each query needs to be evaluated" The cost of "sorting" is reduced to "evaluation per query sample". "Sorting".

5. The performance-supervised context learning example adaptive ranking method according to claim 4, characterized in that, For each sample sort, construct the corresponding ICL prompt word and call the target LLM to obtain the prediction result. Calculate the task indicator score by comparing the prediction result with the actual answer, including the following steps: Sort each sample Construct the corresponding ICL prompt words and call the target LLM to obtain the prediction. Then compare with the real answer Calculate task indicator scores: in The task evaluation metrics include: MSE for regression and BLEU for translation, and scores. This refers to "the true performance of the ranking on the query," which is fundamentally different from heuristic surrogate scores.

6. The performance-supervised context learning-based adaptive ranking method according to claim 5, characterized in that, The query samples and examples are encoded in one step to generate query embedding vectors, single example embedding vectors, and ordered context embedding vectors, including the following steps: The query sample is encoded only once: Each example is coded only once, with the example input and tags copied to the template. splicing: For a certain sort The corresponding example embeddings are concatenated in order to obtain the sorted "ordered context embedding vector": The difference between different sorting methods lies only in the concatenation order, thus replacing repetitive encoding with vector concatenation, significantly reducing complexity.

7. The performance-supervised context learning example adaptive ranking method according to claim 6, characterized in that, Based on the query embedding vector and the ordered context embedding vector, a mapping relationship of "query-ranking-real performance" is learned to obtain a trained ranking predictor, including the following steps: Define the ranking predictor The input is "query embedding + sort embedding", and the output is the prediction performance score of the sort. The training goal is to make Fitting the true performance .

8. The performance-supervised context learning example adaptive ranking method according to claim 7, characterized in that, During the online inference phase, the trained ranking predictor is used to adaptively rank the example set and select the optimal ranking, including the following steps: Test query retrieval and construct embedding representations for candidate rankings. For the test query to be predicted, perform the same operations as in the offline training phase: K examples are retrieved from the example library to form a fixed example set; the test query is encoded once to obtain the embedding vector of the test query; the k retrieved examples are concatenated and encoded once according to the offline training template to obtain the embedding vector of a single example; a small number of candidate sorts are generated for this example set, and the example embedding vectors are concatenated according to the sort order to obtain the ordered context embedding vector of each candidate sort. Predictors score and select the optimal ranking: The "test query embedding vector and the ordered context embedding vector of each candidate sort" are input into the offline trained sorting predictor, which outputs a prediction performance score for each candidate sort. The optimal sort is selected according to the type of task evaluation metric: if the metric is "the larger the better", the sort with the highest prediction score is selected; if the metric is "the smaller the better", the sort with the lowest prediction score is selected.

9. A performance-supervised context-learning example adaptive ranking system, employing the performance-supervised context-learning example adaptive ranking method as described in any one of claims 1 to 8, characterized in that, Including the following steps: The example set building module retrieves the example set from the example library for each query sample in the training query set. This provides a fixed set of examples. The subset sampling module arbitrarily sorts the fixed set of examples to define a sorting space, which has k! candidate permutations. For each query sample, only one subset candidate permutation is sampled in the sorting space, and the subset candidate permutation includes M sampled permutations. The indicator score calculation module sorts each sample, constructs corresponding ICL prompt words, calls the target LLM to obtain the prediction result, and calculates the task indicator score by comparing the prediction result with the actual answer. The encoding module performs a one-time encoding process on the query samples and examples to reduce the computational complexity of training and subsequent inference, generating query embedding vectors, single example embedding vectors, and ordered context embedding vectors. Predictor training module, training the ranking predictor: learns the mapping relationship of "query-ranking-real performance" based on the query embedding vector and the ordered context embedding vector, and obtains the trained ranking predictor; The adaptive sorting module, during the online inference phase, uses the trained sorting predictor to adaptively sort the example set and select the optimal sort.

Images