Data set construction and scoring method of reordering model

Abstract

The invention relates to the technical field of data retrieval and processing, and provides a data set construction and scoring method for a reordering model, which comprises the following steps of: selecting and preprocessing a data source, collecting query document pairs from a multi-scene data source, generating an embedded vector, and constructing a data set; large language model scoring: calling a large language model through the structured cue word to score the query document pair; carrying out multi-model scoring cross validation, and determining a final score by adopting independent scoring of a plurality of large language models and a dynamic consistency checking mechanism; performing in-file calibration on the reordering model, and optimizing a final score by using the reordering model on the basis of the score of the large language model; and data set evaluation and adjustment: analyzing score distribution and text length distribution, and adjusting data distribution through undersampling and oversampling to meet training requirements. Through multi-source data fusion, multi-model collaborative scoring and a dynamic calibration mechanism, the document correlation scoring accuracy is improved, and the model generalization ability is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of data retrieval and processing, and in particular relates to a data set construction and scoring method for a re-ranking model. Background Art

[0002] Retrieval-Augmented Generation (RAG) is a technology that combines information retrieval and text generation. It aims to improve the accuracy and reliability of generated results by dynamically incorporating external knowledge. RAG technology aims to address the problem that traditional generative models (such as GPT) rely on static training data and are prone to "hallucinations" (fabrication of facts). Before AIGC generates an answer, the system retrieves relevant information from external knowledge bases (such as documents and databases) and then generates an answer based on the search results. RAG uses a "retrieve first, generate later" mechanism to integrate external knowledge into the generation process in real time, significantly improving the model's performance in scenarios requiring factual and timely answers. It is currently a key technical direction in the field of AI.

[0003] Information retrieval (IR) is the art and science of efficiently and accurately retrieving required information from large data sets. It forms the core foundation of modern applications such as search engines, recommender systems, and question-answering systems. Since RAG technology requires retrieval, determining a retrieval solution is crucial. Current research focuses on three key areas: data recall, personalized retrieval techniques, and result ranking. In the data recall area, Vladimir Karpukhin et al. proposed a dense vector-based dual-encoder retrieval framework, which improves retrieval accuracy. In the personalized retrieval area, Qiang Zhang et al. proposed a cooperative privacy-preserving personalized search solution; Yujia Zhou et al. proposed a multi-stage query disambiguation model based on user intent. In the result ranking area, Nabil Hossain et al. improved the search and editing sequence-to-sequence method to enhance machine translation; and Gregor Geigle et al. combined dual encoders and cross-encoders to improve the efficiency of cross-modal retrieval models. All three data retrieval areas have improved retrieval capabilities in different ways. However, most of these methods rely on centralized user data analysis and cannot meet the security requirements of RAG systems in cross-domain scenarios. Furthermore, existing reranking technologies are limited to processing plaintext data and lack the ability to protect user privacy. Consequently, existing data retrieval methods struggle to guarantee data security and user privacy in cross-domain RAG systems. Against this backdrop, embedding-based reranking technology has emerged. By converting text into vectors and ranking them based on the similarity between these vectors, it offers a new approach to improving retrieval accuracy and efficiency.

[0004] However, in practical applications, current embedding-based reranking technology is limited by some inherent defects of existing information retrieval datasets, and its advantages have not been fully utilized. Summary of the Invention

[0005] In view of the above-mentioned shortcomings of the existing technology, the present invention proposes a dataset construction and scoring method for a re-ranking model to address the problems of insufficient evaluation of existing information retrieval datasets and difficulty in supporting privacy-preserving retrieval, and to improve the accuracy and reliability of re-ranking model training. The technical solution designed by the present invention includes the following steps: S1: Data source selection and preprocessing: collecting query-document pairs from multi-scenario data sources and generating embedding vectors to build a large-scale dataset; S2: Large language model scoring, using structured prompt words to call the large language model to score the query document pair; S3: Multi-model scoring cross-validation, using multiple large language models to independently score and a dynamic consistency check mechanism to determine the final score; S4: Re-ranking model calibration, using the re-ranking model to optimize the final score based on the large language model score; S5: Dataset evaluation and adjustment, analyzing the rating distribution and text length distribution, and adjusting the data distribution through undersampling and oversampling to meet training requirements.

[0006] Preferably, the S1 includes: S11: Collect query-document pairs from general search, professional field question answering, and academic retrieval scenarios as data sources; S12: Screen data sources based on four criteria: diversity, representativeness, high quality, and accessibility; S13: Delete duplicates from data sources, filter low-quality content or content containing sensitive information, remove entries with empty or very short content, and convert all data sources into a unified JSON format; S14: Select the translation mode to perform multilingual expansion on some data sources; S15: Balanced domain distribution through weighted sampling; S16: Use multiple embedding models to generate embedding vectors for data sources and build large-scale datasets.

[0007] Preferably, the translation mode in S14 includes: There are four modes: translate only the query, translate only the document, translate both the query and the document into the same language, and translate both the query and the document into different languages.

[0008] Preferably, the S2 includes: S21: Design a structured prompt that includes the definition of the rater's role, rating dimensions, grading standards, and format requirements; S22: Using structured prompt words, calling multiple large language models to perform multi-dimensional scoring on the query document pair, the multi-dimensional scoring including deterministic verification, requirement coverage, evidence completeness and structural adaptability.

[0009] Preferably, the S3 includes: S31: Select N large language models and score them up to N+1 times according to priority; S32: Uses dynamic batch processing and parallel request mechanisms for scoring, and monitors abnormal conditions of large language models in real time; S33: The final score is determined through preliminary score verification, abnormal score supplementation and consistency threshold judgment.

[0010] Preferably, the S33 includes: S331: Perform preliminary scoring using the first N / 2 scoring tasks and calculate the mean of the N / 2 scores; S332: Check whether all scores are within the range of ±0.2 of the mean. If two or more scores are out of range, go to S333; otherwise, go to S336. S333: Introduce new models to supplement the scoring. If the scores of N / 2 models are all within the range of the mean ± 0.2, go to S338; otherwise, go to S334. S334: If the number of currently scored models is all models, go to S335; otherwise go to S333; S335: Calculate the average of the scores of all models as the final score, and end; S336: If only one score is out of range, go to S337, otherwise go to S338; S337: Remove outliers and calculate the mean of the remaining N / 2-1 normal values ​​as the final score, and end; S338: Output the mean model score as the final score, and end.

[0011] Preferably, the S4 includes: S41: Call multiple re-ranking models to independently score the same query document pair; S42: normalize the original score of each re-ranking model and calculate the average to obtain the re-ranking model score; S43: Combine the large language model score and the reranking model score, and optimize the final score through the in-file calibration formula.

[0012] Preferably, the in-gear calibration formula in S43 is as follows:

[0013] Where, Scoring large language models, Score the reranking model, is the high value of the gear, The low value of the gear.

[0014] Preferably, the S5 includes: S51: Count the number and proportion of samples in different scoring ranges. If the dataset has reasonable sample coverage in each range and is consistent with the distribution in the actual scenario, go to S52; otherwise, go to S53. S52: Determine the distribution of query and document lengths to ensure that the dataset contains samples of various lengths, from short keywords to long, detailed texts, to fully train the model's processing capabilities under different text length conditions. If the distribution is insufficient, go to S53; otherwise, end. S53: For score intervals or length ranges with too many samples, under-sampling is performed, and some samples are randomly selected for deletion. For score intervals or length ranges with too few samples, over-sampling is performed, and the number of samples is increased by copying existing samples or generating new samples, and then go to S52.

[0015] Beneficial effects: 1. This application comprehensively ensures dataset quality and scoring reliability, from front-end data collection, screening, cleaning, multilingual amplification, domain balancing, and embedding vector generation to back-end scoring using carefully designed prompt words using multiple large language models, and final scoring determined through rigorous consistency testing. The multi-model scoring cross-validation mechanism uses multiple advanced large language models to independently score and perform consistency testing, effectively reducing bias in a single model and improving scoring accuracy and objectivity. 2. This application uses a reranking model to further optimize the scoring results of the large language model, and improves the internal scoring discrimination of the gear through normalization and recalibration within the gear. At the same time, the score distribution and text length distribution of the dataset are analyzed to ensure that the dataset has reasonable coverage in different score ranges and text lengths, comprehensively ensuring the quality and applicability of the dataset, and providing reliable data support for model training and practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a preferred embodiment of the present invention; Figure 2 This is a flow chart of data source selection and preprocessing according to a preferred embodiment of the present invention; Figure 3 1 is a flow chart of large language model scoring according to a preferred embodiment of the present invention; Figure 4 1 is a flow chart of a multi-model scoring cross-validation process according to a preferred embodiment of the present invention; Figure 5 1 is a flow chart of reordering model file calibration according to a preferred embodiment of the present invention; Figure 6 1 is a flow chart of data set evaluation and adjustment according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0017] The embodiments of the present invention are described in detail below. The following embodiments are implemented based on the technical solutions of the present invention, and provide detailed implementation methods and specific operating procedures. However, the protection scope of the present invention is not limited to the following embodiments.

[0018] The present invention designs a data set construction and scoring method for a re-ranking model, and the technical solution includes the following steps: Figure 1 As shown, specifically including: S1: Data source selection and preprocessing: collecting query-document pairs from multi-scenario data sources and generating embedding vectors to build a large-scale dataset; S2: Large language model scoring, which uses structured prompt words to call the large language model to score the query document pair; S3: Multi-model scoring cross-validation, using multiple large language models to independently score and a dynamic consistency check mechanism to determine the final score; S4: Re-ranking model calibration, using the re-ranking model to optimize the final score based on the large language model score; S5: Dataset evaluation and adjustment, analyzing the rating distribution and text length distribution, and adjusting the data distribution through undersampling and oversampling to meet training requirements.

[0019] Preferably, if Figure 2 As shown, S1 includes: S11: Collect query-document pairs from general search, professional field question answering, and academic retrieval scenarios as data sources; S12: Screen data sources based on four criteria: diversity, representativeness, high quality, and accessibility; S13: Delete duplicates from data sources, filter low-quality content or content containing sensitive information, remove entries with empty or very short content, and convert all data sources into a unified JSON format; S14: Select the translation mode to perform multilingual expansion on some data sources; S15: Balanced domain distribution through weighted sampling; S16: Use multiple embedding models to generate embedding vectors for data sources and build large-scale datasets.

[0020] Preferably, the translation mode in S14 includes: There are four modes: translate only the query, translate only the document, translate both the query and the document into the same language, and translate both the query and the document into different languages.

[0021] Specifically, data source selection and preprocessing lay the foundation for building a comprehensive and representative re-ranking dataset. At this stage, factors such as the diversity of data sources, data quality, and the convenience of subsequent processing need to be comprehensively considered; Among them, for data collection, it includes but is not limited to collecting query document pairs from general search, professional field question and answer, and academic retrieval scenarios as data sources, covering many data sources such as IndustryInstruction, NLI-zh, cMedQAv2, etc.; for data screening, the collected data are strictly screened to ensure that the data can reflect the real application scenarios, have high accuracy and consistency, and are easy to reproduce in research, so as to avoid introducing low-quality or unrepresentative data that affects the subsequent model training effect; for data cleaning, low-quality or sensitive information-containing content is filtered, and entries with empty or very short query or document content are removed, and all data are converted into a unified JSON format; for multilingual amplification, the data source that needs to be amplified in multiple languages ​​is selected, and multilingual amplification processing is performed on part of the data, using methods such as translating only queries, translating only documents, and translating both queries and documents. Four modes—translating into the same language and translating queries and documents into different languages—each account for a quarter of the augmented data. DeepL and Google Translate were used as translation engines, randomly selecting target languages ​​from the supported languages ​​for translation. This enriched the dataset's language coverage to meet the needs of cross-language information retrieval. For domain balancing, a weighted sampling strategy was employed, assigning weights based on the quality and representativeness of each data source while also adjusting for balanced domain coverage. This prevented the model from overfitting to data in a specific domain, improved its generalization, and enabled it to perform well in information retrieval across diverse domains. For embedding vector generation, multiple state-of-the-art embedding models, such as bge-m3, jina-embeddings-v3, and gte-multilingual-base, were selected to generate corresponding embedding vectors for each query and document. By converting text data into vectors, the model facilitated subsequent processing and analysis, ultimately constructing a dataset of one million query-document pairs.

[0022] Preferably, if Figure 3 As shown, S2 includes: S21: Design a structured prompt that includes the definition of the rater's role, rating dimensions, grading standards, and format requirements; S22: Using structured prompt words, multiple large language models are called to perform multi-dimensional scoring on query document pairs. The multiple dimensions include deterministic verification, requirement coverage, evidence completeness and structural adaptability.

[0023] Specifically, the design of prompt words includes designing a set of structured prompt words, clarifying the role of the scorer as a "professional information relevance assessment expert", and the task goal is to<query,doc> For scoring, the key dimensions of scoring are elaborated in detail, including deterministic verification, requirement coverage, evidence completeness, and structural adaptability, with clear definitions and quantitative standards provided for each dimension. The specific standards, score ranges, overall feature descriptions, examples, and handling principles for special cases for the six semantic tiers are clarified, and input and output format requirements are specified. This design guides large language models to accurately understand the scoring task and ensure the objectivity and accuracy of scoring. For model scoring, multiple large language models, such as gpt-4o-mini, deepseek-v3, and glm-4-flash, are used to score query-document pairs. Large language models have powerful language understanding and analysis capabilities, and can evaluate query-document pairs from multiple dimensions based on the requirements of the prompt word and assign corresponding scores.

[0024] Preferably, if Figure 4 As shown, S3 includes: S31: Select N large language models and score them up to N+1 times according to priority; S32: Uses dynamic batch processing and parallel request mechanisms for scoring, and monitors abnormal conditions of large language models in real time; S33: The final score is determined through preliminary score verification, abnormal score supplementation and consistency threshold judgment.

[0025] Preferably, S33 includes: S331: Perform preliminary scoring using the first N / 2 scoring tasks and calculate the mean of the N / 2 scores; S332: Check whether all scores are within the range of ±0.2 of the mean. If two or more scores are out of range, go to S333; otherwise, go to S336. S333: Introduce new models to supplement the scoring. If the scores of N / 2 models are all within the range of the mean ± 0.2, go to S338; otherwise, go to S334. S334: If the number of currently scored models is all models, go to S335; otherwise go to S333; S335: Calculate the average of the scores of all models as the final score, and end; S336: If only one score is out of range, go to S337, otherwise go to S338; S337: Remove outliers and calculate the mean of the remaining N / 2-1 normal values ​​as the final score, and end; S338: Output the mean model score as the final score, and end.

[0026] Specifically, model selection involves selecting N (N greater than or equal to 8) advanced large language models and scoring them up to N+1 times according to priority. Model selection includes, for example, gpt-4o-mini, deepseek-v3, glm-4-flash, gemini-1.5-flash, qwen2.5-72b, llama-3.3-70b, kimi_v1.6.8, and claude-3–haiku. Different models differ in architecture, training data, and language comprehension capabilities. Comprehensive scoring of multiple models can reduce the bias of a single model and improve scoring accuracy. Batch scoring involves combining a batch of query-document pairs (usually 5 to 30 pairs, dynamically selected based on model performance and query-document pair length) with prompt words to request each model. Efficiency is improved by calling the APIs of these models in parallel, and an independent request queue and response processing thread are maintained for each model. During the scoring process, progress and quality are monitored in real time. When a specific model behaves abnormally (such as continuously returning format parsing errors or missing scoring results), the scoring task of the model is paused and debugging and adjustments are performed to ensure the smooth progress of the scoring process. For preliminary scoring and verification, the first N / 2 scoring tasks (for example, gpt-4o-mini, deepseek-v3, glm-4-flash, and kimi_v1.6.8) are used for preliminary scoring and the average of the N / 2 scores is calculated.

[0027] Preferably, if Figure 5 As shown, S4 includes: S41: Call multiple re-ranking models to independently score the same query document pair; S42: normalize the original score of each re-ranking model and calculate the average to obtain the re-ranking model score; S43: Combine the large language model score and the reranking model score, and optimize the final score through the in-file calibration formula.

[0028] Preferably, the in-gear calibration formula in S43 is as follows:

[0029] Where, Scoring large language models, Score the reranking model, is the high value of the gear, The low value of the gear.

[0030] Specifically, for model scoring, we selected four SOTA-level reranking models, such as gte-multilingual-reranker-base, bge-reranker-v2-m3, jina-reranker-v2-base-multilingual, and lb-reranker-v1.0, to independently score the same query-document pair. These reranking models have performed well in information retrieval and ranking tasks and can evaluate query-document pairs from different perspectives. For normalization, we normalized the original scores of each reranking model and statistically analyzed the scores of the entire dataset. The minimum and maximum values ​​of the scores of each model are taken, and all scores are linearly normalized to the range of 0-1. After normalization, the average of the four model scores is taken to obtain the reranking model score (rerank_score). This step can eliminate the differences in the scoring scales of different models and make the scoring results more comparable; for intra-file calibration, the intra-file score of the large language model score (llm_score) is combined with the reranking model score (rerank_score). The intra-file recalibration method is adopted to keep the semantic file unchanged and make a finer distinction within each file, which improves the discrimination of the scores within the same file and makes the scoring more detailed and accurate.

[0031] Preferably, if Figure 6 As shown, S5 includes: S51: Count the number and proportion of samples in different scoring ranges. If the dataset has reasonable sample coverage in each range and is consistent with the distribution in the actual scenario, go to S52; otherwise, go to S53. S52: Determine the distribution of query and document lengths to ensure that the dataset contains samples of various lengths, from short keywords to long, detailed texts, to fully train the model's processing capabilities under different text length conditions. If the distribution is insufficient, go to S53; otherwise, end. S53: For score intervals or length ranges with too many samples, under-sampling is performed, and some samples are randomly selected for deletion. For score intervals or length ranges with too few samples, over-sampling is performed, and the number of samples is increased by copying existing samples or generating new samples, and then go to S52.

[0032] Specifically, for analyzing the score distribution, the number and proportion of samples in different score ranges are counted to ensure that the dataset has reasonable sample coverage at each level, which is consistent with the distribution in actual scenarios. For example, completely irrelevant samples (0-0.1) are relatively few, marginally correlated and partially correlated samples (0.25-0.75) are the largest, and highly correlated samples (0.9-1.0) are the rarest. Such a distribution can enable the model to better learn the characteristics of different degrees of correlation during training; for analyzing the text length distribution, the length distribution of queries and documents is checked to ensure that the dataset contains samples of various lengths from short keywords to long and detailed texts, which can fully train the model's processing level under different text length conditions. Most query lengths are concentrated in the range of 10-100 words, and document lengths are mainly distributed in the medium length range of 100-1000 words. At the same time, it contains enough short text and long text samples to meet the needs of different actual application scenarios.

[0033] The above describes in detail the preferred embodiments of the present invention. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible by those skilled in the art without inventive effort. Therefore, any technical solution that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A dataset construction and scoring method for a re-ranking model, characterized in that: include: S1: Data source selection and preprocessing: collecting query-document pairs from multi-scenario data sources and generating embedding vectors to build a large-scale dataset; S2: Large language model scoring, using structured prompt words to call the large language model to score the query document pair; S3: Multi-model scoring cross-validation, using multiple large language models to independently score and a dynamic consistency check mechanism to determine the final score; S4: Re-ranking model calibration, using the re-ranking model to optimize the final score based on the large language model score; S5: Dataset evaluation and adjustment, analyzing the rating distribution and text length distribution, and adjusting the data distribution through undersampling and oversampling to meet training requirements.

2. The method for constructing and scoring a data set for a re-ranking model according to claim 1, wherein: Said S1 comprises: S11: Collect query-document pairs from general search, professional field question answering, and academic retrieval scenarios as data sources; S12: Screen data sources based on four criteria: diversity, representativeness, high quality, and accessibility; S13: Delete duplicates from data sources, filter low-quality content or content containing sensitive information, remove entries with empty or very short content, and convert all data sources into a unified JSON format; S14: Select the translation mode to perform multilingual expansion on some data sources; S15: Balanced domain distribution through weighted sampling; S16: Use multiple embedding models to generate embedding vectors for data sources and build large-scale datasets.

3. The method for constructing and scoring a data set for a re-ranking model according to claim 2, wherein: The translation mode in S14 includes: There are four modes: translate only the query, translate only the document, translate both the query and the document into the same language, and translate both the query and the document into different languages.

4. The method for constructing and scoring a data set for a re-ranking model according to claim 1, wherein: The S2 includes: S21: Design a structured prompt that includes the definition of the rater's role, rating dimensions, grading standards, and format requirements; S22: Using structured prompt words, calling multiple large language models to perform multi-dimensional scoring on the query document pair, the multi-dimensional scoring including deterministic verification, requirement coverage, evidence completeness and structural adaptability.

5. The method for constructing and scoring a data set for a re-ranking model according to claim 1, wherein: The S3 includes: S31: Select N large language models and score them up to N+1 times according to priority; S32: Uses dynamic batch processing and parallel request mechanisms for scoring, and monitors abnormal conditions of large language models in real time; S33: The final score is determined through preliminary score verification, abnormal score supplementation and consistency threshold judgment.

6. The method for constructing and scoring a data set for a re-ranking model according to claim 5, wherein: The S33 includes: S331: Perform preliminary scoring using the first N / 2 scoring tasks and calculate the mean of the N / 2 scores; S332: Check whether all scores are within the range of ±0.2 of the mean. If two or more scores are out of range, go to S333; otherwise, go to S336. S333: Introduce new models to supplement the scoring. If the scores of N / 2 models are all within the range of the mean ± 0.2, go to S338; otherwise, go to S334. S334: If the number of currently scored models is all models, go to S335; otherwise go to S333; S335: Calculate the average of the scores of all models as the final score, and end; S336: If only one score is out of range, go to S337, otherwise go to S338; S337: Remove outliers and calculate the mean of the remaining N / 2-1 normal values ​​as the final score, and end; S338: Output the mean model score as the final score, and end.

7. The method for constructing and scoring a data set for a re-ranking model according to claim 1, wherein: The S4 includes: S41: Call multiple re-ranking models to independently score the same query document pair; S42: normalize the original score of each re-ranking model and calculate the average to obtain the re-ranking model score; S43: Combine the large language model score and the reranking model score, and optimize the final score through the in-file calibration formula.

8. The method for constructing and scoring a data set for a re-ranking model according to claim 7, wherein: The in-gear calibration formula in S43 is as follows: Where, Scoring large language models, Score the reranking model, is the high value of the gear, The low value of the gear.

9. The method for constructing and scoring a data set for a re-ranking model according to claim 1, wherein: The S5 includes: S51: Count the number and proportion of samples in different scoring ranges. If the dataset has reasonable sample coverage in each range and is consistent with the distribution in the actual scenario, go to S52; otherwise, go to S53. S52: Determine the distribution of query and document lengths to ensure that the dataset contains samples of various lengths, from short keywords to long, detailed texts, to fully train the model's processing capabilities under different text length conditions. If the distribution is insufficient, go to S53; otherwise, end. S53: For score intervals or length ranges with too many samples, under-sampling is performed, and some samples are randomly selected for deletion. For score intervals or length ranges with too few samples, over-sampling is performed, and the number of samples is increased by copying existing samples or generating new samples, and then go to S52.

Images