Evaluation method and system for large model retrieval result

Abstract

The invention relates to the technical field of artificial intelligence, in particular to an evaluation method and system for a large model retrieval result, and the method comprises the steps of dynamic parameter configuration, result matching and scoring, performance index calculation and optimal parameter recommendation. The method has the beneficial effects that manual scoring is replaced by an automatic evaluation means, so that the evaluation efficiency is remarkably improved, and the labor cost is greatly reduced; objectivity and consistency of evaluation results are ensured, errors caused by artificial factors are reduced, and reliability of the evaluation results is improved; through the automatic evaluation process, the user is helped to quickly find the optimal retrieval parameter configuration, and the performance of the retrieval system is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and specifically provides an evaluation method and system for the retrieval results of large models. Background Art

[0002] With the development of big data and artificial intelligence technologies, large models are increasingly widely used in information retrieval. However, how to effectively evaluate the quality of the retrieval results of these large models has become an urgent problem to be solved. Traditional evaluation methods usually rely on manual scoring, which is not only time-consuming and laborious, but also easily affected by subjective factors, resulting in inaccurate evaluation results.

[0003] Currently, there are mainly the following problems in the evaluation of the retrieval results of large models:

[0004] Low efficiency of manual scoring: For a large amount of data, it is very time-consuming to manually check and score the retrieval results item by item, and it is difficult to meet the requirements of large-scale data processing.

[0005] Inconsistent scoring criteria: Different evaluators may have differences in the judgment of retrieval results, resulting in increased subjectivity and uncertainty of scoring results.

[0006] Lack of automated tools: There is a lack of effective automated tools in the market to support the objective evaluation of the retrieval results of large models, making it difficult to standardize and scale the evaluation process.

[0007] Therefore, developing a method and device that can automatically evaluate the quality of the retrieval results of large models is of great significance for improving the efficiency and accuracy of information retrieval. Summary of the Invention

[0008] The purpose of the present invention is to provide an evaluation method and system for the retrieval results of large models to solve the problems raised in the above background art.

[0009] To achieve the above purpose, the present invention provides the following technical solution: An evaluation method for the retrieval results of large models, the method includes a step of dynamically configuring parameters, and the step of dynamically configuring parameters includes:

[0010] Provide a visual configuration interface, and the interface includes a scoring rule setting area, a threshold configuration area, and a test set import area;

[0011] In the scoring rule setting area, allow users to set the specific conditions for adding and subtracting points by means of a drop-down menu, a slider, or directly inputting values;

[0012] In the threshold configuration area, allow users to customize multiple score thresholds, each threshold representing a scoring segment, and support setting a set of different recall numbers;

[0013] In the test set import area, provide a file selector or a path input box, allowing users to specify the path where the test set Excel file is stored and automatically parsing the content of the Excel file;

[0014] After the configuration is completed, provide an instant preview function to show the impact of the currently set scoring rules, thresholds, and recall configurations on the scoring of retrieval results;

[0015] Implement a configuration saving mechanism that allows users to save multiple different parameter configuration schemes and provides a configuration history function.

[0016] Preferably, the method further includes a result matching and scoring step, and the result matching and scoring step includes:

[0017] Data parsing: Read the test set Excel file configured by the user from the specified location and parse the questions, standard answers, and source document information therein;

[0018] Matching algorithm: Use a string matching algorithm to compare the retrieval results obtained by the large model with the content of the standard answers in the test set and perform a strict matching check on the source documents;

[0019] Matching determination: Only when the content of the standard answer in the retrieval result is exactly the same and the source document completely matches the source document of the standard answer, it is regarded as a successful match;

[0020] Automatic scoring: Automatically score according to the scoring rules set by the user in the dynamic configuration parameter step, calculate the scores for the successfully matched retrieval results, and set a full score and a passing score line;

[0021] Result summary and feedback: Summarize all the successfully matched retrieval results and their corresponding scores, calculate the overall accuracy rate, and help users identify the optimal parameter combination.

[0022] Preferably, the method further includes a performance metric calculation step, and the performance metric calculation step includes:

[0023] Qualified result screening: From all the successfully matched and scored retrieval results, screen out the retrieval results whose scores reach the passing score line;

[0024] Result count: Automatically count the number of qualified results;

[0025] Accuracy rate calculation: Calculate the proportion of the number of qualified results to the total number of test sets to obtain the accuracy rate;

[0026] Parameter combination performance evaluation: For each pair of different score threshold and recall count combinations configured by the user, repeat the accuracy rate calculation process and record the accuracy rate data for each group;

[0027] Performance comparison: By comparing the accuracy rates under different parameter combinations, the performance differences of different configurations are shown.

[0028] Preferably, the method further includes an optimal parameter recommendation step, and the optimal parameter recommendation step includes:

[0029] Accuracy rate comparison: Compare the accuracy rates under all parameter combinations to find the parameter combination with the highest accuracy rate;

[0030] Optimal parameter recommendation: Recommend the parameter combination with the highest accuracy rate as the optimal parameter to the user, and at the same time conduct a comprehensive evaluation considering factors such as computing resource consumption and time efficiency;

[0031] Visualization display: Design a chart to clearly display the accuracy rate data under different parameter combinations, and implement a highlighting logic in the chart to automatically mark the parameter combination with the highest accuracy rate.

[0032] Preferably, the instant preview function in the dynamic configuration parameter step allows the user to fine-tune the configuration to ensure that the parameters are reasonable and meet expectations; and the configuration saving mechanism supports the user to name different parameter configuration schemes for distinction and reuse.

[0033] A system for an evaluation method of large model retrieval results includes a dynamic configuration parameter module, and the dynamic configuration parameter module includes:

[0034] A visualization configuration interface for providing a user interaction environment, and the interface is provided with a scoring rule setting area, a threshold configuration area, and a test set import area;

[0035] The scoring rule setting area is configured to allow the user to set the specific conditions for adding and subtracting scores by means of a drop-down menu, a slider, or directly entering a value;

[0036] The threshold configuration area is configured to allow the user to customize multiple score thresholds, each threshold representing a scoring segment, and support setting a set of different recall numbers;

[0037] The test set import area is configured with a file selector or a path input box for specifying the path to store the test set Excel file and automatically parsing the content of the Excel file;

[0038] The instant preview function is used to instantly show the impact of the currently set scoring rules, thresholds, and recall configurations on the retrieval result scoring after the configuration is completed;

[0039] The configuration saving mechanism is used to allow the user to save multiple different parameter configuration schemes and provide a configuration history function.

[0040] Preferably, the system further includes a result matching and scoring module, which includes:

[0041] A data parsing unit for reading the test set Excel file configured by the user from a specified location and parsing the questions, standard answers, and source document information therein;

[0042] A matching algorithm unit that uses a string matching algorithm to compare the retrieval results obtained by the large model with the standard answer content in the test set and performs a strict matching check on the source document;

[0043] A matching determination unit configured to consider a match successful only when the standard answer content in the retrieval result is exactly the same and the source document completely matches the source document of the standard answer;

[0044] An automatic scoring unit that performs automatic scoring according to the scoring rules set by the user in the dynamic configuration parameter module, calculates scores for the retrieval results with successful matches, and sets a full score and a passing score line;

[0045] A result summary and feedback unit for summarizing all retrieval results with successful matches and their corresponding scores, calculating the overall accuracy rate, and helping the user identify the optimal parameter combination.

[0046] Preferably, the system further includes a performance index calculation module, which includes:

[0047] A qualified result screening unit for screening out the retrieval results with scores reaching the passing score line from all the retrieval results with successful matches and having been scored;

[0048] A result count unit for automatically counting the number of qualified results;

[0049] An accuracy rate calculation unit for calculating the proportion of the number of qualified results to the total number of test sets to obtain the accuracy rate;

[0050] A parameter combination performance evaluation unit that repeats the accuracy rate calculation process for each pair of different score thresholds and recall count combinations configured by the user and records the accuracy rate data for each group;

[0051] A performance comparison unit for showing the performance differences of different configurations by comparing the accuracy rates under different parameter combinations.

[0052] Preferably, the system further includes an optimal parameter recommendation module, which includes:

[0053] An accuracy rate comparison unit for comparing the accuracy rates under all parameter combinations to find the parameter combination with the highest accuracy rate;

[0054] The optimal parameter recommendation unit is used to recommend the parameter combination with the highest accuracy as the optimal parameter to the user, and comprehensively evaluate by considering factors such as computing resource consumption and time efficiency;

[0055] The visualization display unit is used to design a chart to clearly display the accuracy data under different parameter combinations, and implement a highlighting logic in the chart to automatically mark the parameter combination with the highest accuracy.

[0056] Preferably, the system further includes:

[0057] The fine-tuning support unit allows the user to fine-tune the configuration in the instant preview function to ensure that the parameters are reasonable and meet the expectations;

[0058] The configuration naming and reuse unit supports the user to name different parameter configuration schemes in the configuration saving mechanism for distinction and reuse;

[0059] The hardware support structure includes a processor, a memory, an input / output interface, and a power supply module, which are used to support the software operation of the system and ensure the stability and compatibility of the system;

[0060] The security protection unit is built-in with a firewall and security software to protect the system from malicious attacks.

[0061] Compared with the prior art, the beneficial effects of the present invention are:

[0062] The evaluation method and system for the retrieval results of large models proposed by the present invention replace manual scoring with automated evaluation means, significantly improving the evaluation efficiency and greatly reducing the labor cost; ensuring the objectivity and consistency of the evaluation results, reducing the errors introduced by human factors, and improving the reliability of the evaluation results; helping users quickly find the optimal retrieval parameter configuration through an automated evaluation process, improving the performance of the retrieval system; improving the accuracy and reliability of information retrieval by accurately scoring the retrieval results; adopting a unified scoring rule and standard, reducing the interference of human factors in the scoring process, and making the evaluation results more fair; providing a standardized method and tool for the evaluation of large model retrieval results, which is conducive to the formation of a unified evaluation standard. Description of the Drawings

[0063] Figure 1 It is the flowchart of the method of the present invention;

[0064] Figure 2 It is the block diagram of the system of the present invention. Detailed Embodiments

[0065] In order to clearly and completely describe the objectives, technical solutions of the present invention, and make the advantages more clearly understood, the following further details the embodiments of the present invention with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all embodiments, and are only used to explain the embodiments of the present invention, not to limit the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0066] Example 1. Please refer to Figure 1 , the present invention provides a technical solution: an evaluation method for the retrieval results of large models, including the following steps:

[0067] 1. Dynamically configure parameters: Dynamically configuring parameters is one of the core components of the present invention, aiming to endow users with a high degree of flexibility and customization ability to adapt to different evaluation scenarios and requirements. Through this module, users can freely set evaluation indicators and parameters according to the specific requirements of actual applications, so as to achieve accurate evaluation of the retrieval results of large models.

[0068] (1) Visual configuration interface: Design a set of intuitive and easy-to-use graphical user interfaces (GUI) to provide a friendly operation environment. The interface should include, but not be limited to, the following sections:

[0069] (2) Scoring rule setting area: Users can set the specific conditions for adding and subtracting points by means of drop-down menus, sliders, or directly entering values. For example, the score deducted for each additional piece of irrelevant information can be defined, or scoring can be carried out according to the percentage of answer relevance.

[0070] (3) Threshold configuration area: In this area, users can customize multiple score thresholds, and each threshold represents a scoring segment. At the same time, it supports setting a set of different recall numbers to evaluate the performance at different recall levels.

[0071] (4) Test set import: Provide a file selector or path input box to allow users to specify the local or network path where the test set Excel file is stored. One-key import is supported, and the content of the Excel file is automatically parsed, including key information such as questions, standard answers, and source documents. The system should have the ability to detect errors. If it is found that the imported file format does not match or necessary fields are missing, an error prompt should be promptly feedback to guide users to correct.

[0072] (5) Instant preview and verification: After the configuration is completed, provide a "preview" function to immediately display how the currently set scoring rules, thresholds, and recall configurations will affect the scoring of retrieval results. Users can take this opportunity to fine-tune the configuration to ensure that the parameters are reasonable and meet expectations.

[0073] (6) Configuration Saving and History: Implement a configuration saving mechanism that allows users to save multiple different parameter configuration schemes and name them for differentiation. At the same time, provide a configuration history function for easy access and reuse of previous configurations.

[0074] (7) Dynamic Application and Feedback: After the user confirms the configuration, this module should be able to immediately apply these parameters to the evaluation process. The system automatically executes the evaluation according to the user settings and displays the evaluation results and the recommended optimal parameter combinations, including key indicators such as accuracy and recall rate, on the interface after completion.

[0075] 2. Result Matching and Scoring: This part details how to achieve an exact match between the retrieved results and the standard answers and source documents in the test set, as well as an automatic scoring mechanism based on this, ensuring that the evaluation process is both strict and flexible.

[0076] (1) Exact Matching Process

[0077] A. Data Parsing: First, the system reads the Excel file of the test set configured by the user from the specified location, parses the questions, standard answers, and source document information therein, and stores them in memory or a database to prepare for the subsequent matching process.

[0078] B. Matching Algorithm: For each retrieval result obtained from the large model, the system uses a string matching algorithm (such as the KMP algorithm or regular expressions) to compare it with the content of the standard answers in the test set. At the same time, a strict matching check is performed on the source document in the retrieval result and the source document corresponding to the standard answer to confirm whether the two come from the same source or have a very high degree of consistency.

[0079] C. Matching Judgment: Only when the content of the standard answer contained in the retrieval result is exactly the same and the source document matches the source document of the standard answer exactly, the retrieval result is considered a successful match and then enters the scoring session. Any deviation or mismatch will result in a score of 0 for this result, thus ensuring the rigor of the evaluation.

[0080] (2) Automatic Scoring

[0081] A. Application of Scoring Rule Configuration: The system automatically scores according to the scoring rules set by the user in the dynamic configuration parameter module. The user can set deduction conditions according to actual needs, such as setting a rule of "subtract 10 points for each additional result".

[0082] B. Dynamic scoring calculation: For the successfully matched retrieval results, the system automatically calculates their scores according to the rules configured by the user. The score calculation takes into account various factors, such as the completeness and relevance of the information, as well as the user-defined scoring criteria. The full score is 100 points, aiming to reflect the best retrieval effect; at the same time, a passing score line is set at 60 points, and the retrieval results with scores lower than this are considered to need further optimization.

[0083] C. Result summarization and feedback: All successfully matched retrieval results and their corresponding scores are summarized by the system, and the overall accuracy rate (i.e., the proportion of results with scores ≥ 60 points) is calculated according to the set recall limit. In addition, the system can also help users identify the optimal parameter combination based on the accuracy rates at various score thresholds and recall numbers, so as to guide the tuning work of the model or system.

[0084] 3. Performance metric calculation: This module is dedicated to automatically counting and analyzing the evaluation data of retrieval results. The specific implementation steps are as follows:

[0085] (1) Accuracy rate calculation

[0086] A. Screening of qualified results: From all the successfully matched and scored retrieval results, screen out the retrieval results with scores reaching 60 points and above, which are regarded as "qualified" results.

[0087] B. Counting the number of results: Automatically count the number of qualified results and record the number of retrieval results that meet the passing standard.

[0088] C. Accuracy rate formula: Calculate the proportion of the number of qualified results to the total number of the entire test set to obtain the accuracy rate. Accuracy rate = (Number of qualified results / Total number of the test set) * 100%.

[0089] (2) Performance evaluation of parameter combinations

[0090] A. Traversal of parameter combinations: For each pair of different score thresholds and recall number combinations configured by the user, the system automatically executes the evaluation process.

[0091] B. Accuracy rate calculation: For each group of parameter combinations, repeat the above accuracy rate calculation process and record the accuracy rate data of each group.

[0092] C. Performance comparison: By comparing the accuracy rates under different parameter combinations, the system can intuitively show the performance differences of different configurations.

[0093] 4. Optimal parameter recommendation:

[0094] (1) Identification of optimal parameters

[0095] A. Comparison of accuracy rates: Compare the accuracy rates under all parameter combinations and find out the parameter combination with the highest accuracy rate.

[0096] B. Optimal Parameter Recommendation: Usually, the parameter combination with the highest accuracy (score threshold and number of recall items) is recommended to users as the optimal parameters. However, other factors such as computing resource consumption and time efficiency are also considered for comprehensive evaluation.

[0097] (2) Visualization Display

[0098] A. Accuracy Display: Design a chart to clearly display the accuracy data under different parameter combinations, such as in the form of a bar chart, line chart, or heat map. Implement the interactive function of the chart, allowing users to view specific accuracy values or other relevant information by clicking or hovering.

[0099] B. Highlighting of Optimal Parameters: Implement the highlighting logic in the chart to automatically mark the parameter combination with the highest accuracy and make it stand out in the chart. Use different visual elements such as colors, shapes, or sizes to emphasize the optimal parameter combination, ensuring that users can easily identify the optimal solution at a glance.

[0100] 5. Structure of the Application Device: This module aims to design a hardware device integrating the above functions. The device includes key components such as a processor, memory, input / output interfaces, etc., supports software operation, and has good stability and compatibility. The specific design is as follows:

[0101] (1) Hardware Structure

[0102] A. Processor: Adopt a high-performance central processing unit (CPU) to ensure smooth software operation and fast processing speed.

[0103] B. Memory: Equip with sufficient random access memory (RAM) and non-volatile memory (such as SSD hard disk) to store test set data, scoring rule configuration, intermediate calculation results, and final evaluation reports, etc.

[0104] C. Input / Output Interfaces: Provide various types of input / output interfaces, such as USB, network interfaces (Ethernet / Wi-Fi), serial interfaces (RS232 / RS485), etc., to facilitate connecting external devices and networks.

[0105] D. Power Module: Equip with a stable power supply module to ensure the long-term stable operation of the device.

[0106] (2) Software Support

[0107] A. Operating System: Adopt a stable operating system (such as Linux or Windows Server) to ensure software compatibility and stability.

[0108] B. Application: Integrate the model retrieval result scoring system software to implement the above functions.

[0109] C. Security protection: Built-in firewall and security software to protect the device from malicious attacks.

[0110] Embodiment 2, based on Embodiment 1, proposes a system for evaluating the retrieval results of large models, including a dynamic configuration parameter module. The dynamic configuration parameter module includes: a visual configuration interface for providing a user interaction environment, which is provided with a scoring rule setting area, a threshold configuration area, and a test set import area; the scoring rule setting area is configured to allow users to set the specific conditions for adding or subtracting points by means of a drop-down menu, a slider, or directly entering a value; the threshold configuration area is configured to allow users to customize multiple score thresholds, each threshold representing a scoring segment, and support setting a set of different recall numbers; the test set import area is configured with a file selector or a path input box for specifying the path to store the test set Excel file and automatically parsing the content of the Excel file; an instant preview function for instantly displaying the impact of the currently set scoring rules, thresholds, and recall configurations on the retrieval result scoring after configuration; a configuration saving mechanism for allowing users to save multiple different parameter configuration schemes and providing a configuration history function.

[0111] The system also includes a result matching and scoring module. The result matching and scoring module includes: a data parsing unit for reading the test set Excel file configured by the user from a specified location and parsing the questions, standard answers, and source document information therein; a matching algorithm unit that uses a string matching algorithm to compare the retrieval results obtained by the large model with the standard answer content in the test set and performs a strict matching check on the source document; a matching determination unit configured to consider a match successful only when the standard answer content in the retrieval result is exactly the same and the source document completely matches the source document of the standard answer; an automatic scoring unit that automatically scores according to the scoring rules set by the user in the dynamic configuration parameter module, calculates the scores for the successfully matched retrieval results, and sets a full score and a passing score line; a result summary and feedback unit for summarizing all the successfully matched retrieval results and their corresponding scores, calculating the overall accuracy rate, and helping the user identify the optimal parameter combination.

[0112] The system further includes a performance metric calculation module, and the performance metric calculation module includes: a qualified result screening unit for screening out the retrieval results whose scores reach the passing score line from all the retrieval results that have successfully matched and been scored; a result number statistics unit for automatically counting the number of qualified results; an accuracy calculation unit for calculating the proportion of the number of qualified results to the total number of the entire test set to obtain the accuracy; a parameter combination performance evaluation unit for repeating the accuracy calculation process for each pair of different combinations of score thresholds and recall numbers configured by the user, and recording the accuracy data of each group; a performance comparison unit for demonstrating the performance differences of different configurations by comparing the accuracies under different parameter combinations.

[0113] The system further includes an optimal parameter recommendation module, and the optimal parameter recommendation module includes: an accuracy comparison unit for comparing the accuracies under all parameter combinations to find out the parameter combination with the highest accuracy; an optimal parameter recommendation unit for recommending the parameter combination with the highest accuracy to the user as the optimal parameter, and comprehensively evaluating considering factors such as computing resource consumption and time efficiency; a visualization display unit for designing a chart to clearly display the accuracy data under different parameter combinations, and implementing a highlighting logic in the chart to automatically mark out the parameter combination with the highest accuracy.

[0114] The system further includes: a fine-tuning support unit that allows the user to fine-tune the configuration in the instant preview function to ensure that the parameters are reasonable and meet the expectations; a configuration naming and reuse unit that supports the user in naming different parameter configuration schemes in the configuration saving mechanism for distinction and reuse; a hardware support structure including a processor, a memory, an input / output interface, and a power module for supporting the software operation of the system to ensure the stability and compatibility of the system; a security protection unit with a built-in firewall and security software for protecting the system from malicious attacks.

[0115] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An evaluation method for the retrieval results of large models, characterized in that: The method includes a step of dynamically configuring parameters, and the step of dynamically configuring parameters includes: Providing a visual configuration interface, which includes a scoring rule setting area, a threshold configuration area, and a test set import area; In the scoring rule setting area, allowing users to set specific conditions for adding or subtracting points by means of a drop-down menu, a slider, or directly entering a value; In the threshold configuration area, allowing users to customize multiple score thresholds, each threshold representing a scoring segment, and supporting the setting of a set of different recall numbers; In the test set import area, providing a file selector or a path input box, allowing users to specify the path where the test set Excel file is stored, and automatically parsing the content of the Excel file; After the configuration is completed, providing an instant preview function to show the impact of the currently set scoring rules, thresholds, and recall configurations on the scoring of retrieval results; Implementing a configuration saving mechanism, allowing users to save multiple different parameter configuration schemes, and providing a configuration history function.

2. The evaluation method and system for the retrieval results of large models according to claim 1, characterized in that: The method further includes a result matching and scoring step, and the result matching and scoring step includes: Data parsing: Reading the test set Excel file configured by the user from a specified location, and parsing the questions, standard answers, and source document information therein; Matching algorithm: Using a string matching algorithm to compare the retrieval results obtained by the large model with the content of the standard answers in the test set, and performing a strict matching check on the source documents; Matching determination: Only when the content of the standard answers in the retrieval results is exactly the same and the source document completely matches the source document of the standard answer, it is regarded as a successful match; Automatic scoring: Automatically scoring according to the scoring rules set by the user in the step of dynamically configuring parameters, calculating scores for the successfully matched retrieval results, and setting a full score and a passing score line; Result summary and feedback: Summarizing all successfully matched retrieval results and their corresponding scores, calculating the overall accuracy rate, and helping users identify the optimal parameter combination.

3. The evaluation method and system for large model retrieval results according to claim 2, wherein: The method further includes a performance index calculation step, and the performance index calculation step includes: Qualified result screening: Screening out the retrieval results with scores reaching the passing score line from all successfully matched and scored retrieval results; Result number statistics: Automatically counting the number of qualified results; Accuracy rate calculation: Calculating the proportion of the number of qualified results to the total number of test sets to obtain the accuracy rate; Parameter combination performance evaluation: For each pair of different score threshold and recall number combinations configured by the user, repeating the accuracy rate calculation process, and recording the accuracy rate data of each group; Performance comparison: Demonstrating the performance differences of different configurations by comparing the accuracy rates under different parameter combinations.

4. The evaluation method and system for the retrieval results of a large model according to claim 3, wherein: The method further includes an optimal parameter recommendation step, and the optimal parameter recommendation step includes: Accuracy rate comparison: Comparing the accuracy rates under all parameter combinations to find the parameter combination with the highest accuracy rate; Optimal parameter recommendation: Recommending the parameter combination with the highest accuracy rate to the user as the optimal parameter, and comprehensively evaluating considering factors such as computing resource consumption and time efficiency; Visualization display: Designing a chart to clearly display the accuracy rate data under different parameter combinations, and implementing a highlighting logic in the chart to automatically mark the parameter combination with the highest accuracy rate.

5. The evaluation method and system for large model retrieval results according to claim 4, characterized in that: The instant preview function in the dynamic configuration parameter step allows users to fine-tune the configuration to ensure that the parameters are reasonable and meet expectations; and the configuration saving mechanism supports users in naming different parameter configuration schemes for differentiation and reuse.

6. A system for the evaluation method of the retrieval results of a large model according to claim 5, characterized in that: It includes a dynamic configuration parameter module, and the dynamic configuration parameter module includes: A visual configuration interface for providing a user interaction environment, and the interface is provided with a scoring rule setting area, a threshold configuration area, and a test set import area; The scoring rule setting area is configured to allow users to set the specific conditions for adding or subtracting points by means of a drop-down menu, a slider, or directly entering a value; The threshold configuration area is configured to allow users to customize multiple score thresholds, each threshold representing a scoring segment, and support setting a set of different recall numbers; The test set import area is configured with a file selector or a path input box for specifying the path to store the test set Excel file and automatically parsing the content of the Excel file; The instant preview function is used to instantaneously display the impact of the currently set scoring rules, thresholds, and recall configurations on the scoring of retrieval results after the configuration is completed; The configuration saving mechanism is used to allow users to save multiple different parameter configuration schemes and provide a configuration history function.

7. A system according to claim 6, characterized in that: The system further includes a result matching and scoring module, and the result matching and scoring module includes: A data parsing unit for reading the test set Excel file configured by the user from a specified location and parsing the questions, standard answers, and source document information therein; A matching algorithm unit that uses a string matching algorithm to compare the retrieval results obtained by the large model with the content of the standard answers in the test set and performs a strict matching check on the source documents; The matching determination unit is configured to consider a match successful only when the content of the standard answer in the retrieval result is exactly the same and the source document exactly matches the source document of the standard answer; An automatic scoring unit that automatically scores according to the scoring rules set by the user in the dynamic configuration parameter module, calculates the scores for the retrieval results with successful matches, and sets a full score and a passing score line; A result summary and feedback unit for summarizing all retrieval results with successful matches and their corresponding scores, calculating the overall accuracy rate, and helping users identify the optimal parameter combination.

8. A system according to claim 7, wherein: The system further includes a performance index calculation module, and the performance index calculation module includes: A qualified result screening unit for screening out the retrieval results with scores reaching the passing score line from all the retrieval results with successful matches and having been scored; A result number statistics unit for automatically counting the number of qualified results; An accuracy rate calculation unit for calculating the proportion of the number of qualified results to the total number of test sets to obtain the accuracy rate; A parameter combination performance evaluation unit that repeats the accuracy rate calculation process for each pair of different score threshold and recall number combinations configured by the user and records the accuracy rate data for each group; A performance comparison unit for showing the performance differences of different configurations by comparing the accuracy rates under different parameter combinations.

9. A system according to claim 8, wherein: The system further includes an optimal parameter recommendation module, and the optimal parameter recommendation module includes: An accuracy comparison unit for comparing the accuracies under all parameter combinations and finding the parameter combination with the highest accuracy; An optimal parameter recommendation unit for recommending the parameter combination with the highest accuracy as the optimal parameter to the user and comprehensively evaluating it considering factors such as computing resource consumption and time efficiency; A visualization display unit for designing a chart to clearly display the accuracy data under different parameter combinations and implementing a highlighting logic in the chart to automatically mark the parameter combination with the highest accuracy.

10. A system according to claim 9, wherein: The system further includes: A fine-tuning support unit that allows the user to fine-tune the configuration in the instant preview function to ensure that the parameters are reasonable and meet expectations; A configuration naming and reuse unit that supports the user in naming different parameter configuration schemes in the configuration saving mechanism for differentiation and reuse; A hardware support structure including a processor, a memory, an input / output interface, and a power module for supporting the software operation of the system and ensuring the stability and compatibility of the system; A security protection unit with a built-in firewall and security software for protecting the system from malicious attacks.

Images