Test methods and electronic equipment for retrieval enhancement generation systems

Abstract

This application discloses a testing method and electronic device for a retrieval enhancement generation system, relating to the field of testing technology. The method includes obtaining runtime time series vectors of each retrieval enhancement generation system, containing multiple preset test event durations, under different concurrency levels, and determining a baseline runtime time series vector based on these vectors. The runtime time series vectors of each retrieval enhancement generation system under different concurrency levels and the baseline runtime time series vector are normalized. Based on the normalized runtime time series vectors and the normalized baseline runtime time series vectors, event scores for each retrieval enhancement generation system under different concurrency levels are determined to obtain a comprehensive score for each retrieval enhancement generation system, thereby determining the test result for each retrieval enhancement generation system. This application takes into account both the performance of the retrieval enhancement generation system under different concurrency levels and the operational characteristics of multiple test events, improving the accuracy of the test results for each retrieval enhancement generation system.

Description

Technical Field

[0001] This application relates to the field of testing technology, and in particular to a testing method and electronic device for a retrieval enhancement generation system. Background Technology

[0002] Retrieval-Augmented Generation (RAG) is an artificial intelligence framework that integrates a retrieval engine and a generator. It connects large language models to external knowledge bases, receives queries, matches relevant information, and then generates accurate answers, making it widely used in question-answering scenarios.

[0003] In related technologies, when testing multiple search enhancement generation systems, the tests are usually conducted using a single performance metric for each system. However, the test results obtained using this method are not accurate enough. Summary of the Invention

[0004] This application provides a testing method and electronic device for a retrieval enhancement generation system, to at least solve the problem of inaccurate test results in related technologies.

[0005] This application provides a testing method for a retrieval enhancement generation system, including:

[0006] Obtain multiple search enhancement generation systems to be tested;

[0007] Tests were conducted on each retrieval enhancement generation system under different concurrency levels to obtain runtime vectors of each retrieval enhancement generation system under different concurrency levels. The runtime vectors contain test duration information for multiple preset test events.

[0008] Based on the runtime time series vectors of each retrieval enhancement generation system under different concurrency levels, determine the baseline runtime time series vector;

[0009] The running time series vectors of each retrieval enhancement generation system under different concurrency levels and the baseline running time series vector are normalized to obtain the normalized running time series vectors of each retrieval enhancement generation system under different concurrency levels and the normalized baseline running time series vector.

[0010] For any retrieval enhancement generation system, the event score of the retrieval enhancement generation system under different concurrency levels is determined based on the normalized baseline runtime time series vector and the normalized runtime time series vector of the retrieval enhancement generation system under different concurrency levels.

[0011] The overall score of each retrieval enhancement generation system is determined based on the event score of each system under different concurrency levels.

[0012] The test results for each retrieval enhancement generation system are determined based on the comprehensive scores of each system.

[0013] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the test method of any of the above-described retrieval enhancement generation systems.

[0014] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the test method for any of the above-described retrieval enhancement generation systems.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the test method for any of the above-described retrieval enhancement generation systems.

[0016] This application obtains runtime time series vectors containing multiple preset test event durations for each retrieval enhancement generation system by testing under different concurrency levels. A baseline runtime time series vector is determined based on these vectors. The runtime time series vectors of each retrieval enhancement generation system under different concurrency levels and the baseline vector are then normalized to obtain normalized runtime time series vectors for each system under different concurrency levels and a normalized baseline runtime time series vector. For any retrieval enhancement generation system, an event score is determined based on the normalized baseline and normalized runtime time series vectors. Finally, a comprehensive score is determined for each retrieval enhancement generation system based on its event score, and the test result for each system is determined based on this comprehensive score. The method in this application takes into account the performance of the retrieval enhancement generation system under different load scenarios, i.e. different concurrency, and the running characteristics of multiple test events, which greatly improves the accuracy of the test results of each retrieval enhancement generation system compared with single index testing. Attached Figure Description

[0017] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of a hardware architecture provided for this application;

[0019] Figure 2 A flowchart illustrating a testing method for a retrieval enhancement generation system provided in an embodiment of this application;

[0020] Figure 3 A schematic diagram of a test architecture provided for an embodiment of this application;

[0021] Figure 4 A schematic diagram of an interface function provided in an embodiment of this application;

[0022] Figure 5 A schematic diagram of an event structure provided in an embodiment of this application;

[0023] Figure 6 A schematic diagram of an event type provided in an embodiment of this application;

[0024] Figure 7 A schematic diagram of an event state provided for an embodiment of this application;

[0025] Figure 8 A schematic diagram of a testing process provided in an embodiment of this application;

[0026] Figure 9 A flowchart illustrating a method for obtaining runtime vectors of various retrieval enhancement generation systems under different concurrency levels, provided in an embodiment of this application;

[0027] Figure 10 A flowchart illustrating another method for obtaining runtime time series vectors of a retrieval enhancement generation system under different concurrency levels, provided in an embodiment of this application;

[0028] Figure 11 A flowchart illustrating a method for determining event scoring scores of various retrieval enhancement generation systems under different concurrency levels, provided as an embodiment of this application;

[0029] Figure 12 This is a schematic diagram of a normalization process provided in an embodiment of this application;

[0030] Figure 13 A schematic diagram of the structure of a test device for a retrieval enhancement generation system provided in an embodiment of this application;

[0031] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0032] Explanation of reference numerals in the attached figures:

[0033] 01-Processor; 02-Memory; 03-Communication components; 04-Bus. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, other embodiments obtained by those of ordinary skill in the art without creative effort are all within the protection scope of this application.

[0035] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0036] The retrieval-enhanced generation system is an innovative technical architecture that deeply integrates information retrieval technology and generative artificial intelligence. The application of this system has entered the stage of large-scale implementation. Its core value lies in its ability to effectively compensate for the inherent shortcomings of generative artificial intelligence models in terms of knowledge coverage and timeliness by dynamically retrieving massive external knowledge sources, thereby significantly improving the accuracy, reliability and timeliness of the output content of the Large Language Model (LLM).

[0037] From a technical architecture perspective, the software architecture of the retrieval enhancement generation system exhibits typical characteristics of multi-component collaboration, primarily comprising five core parts: an embedding model, a vector database, a reordering model, a large language model inference engine, and the large language model itself. The system's general workflow follows a closed-loop logic of data preprocessing, retrieval, reordering, and generation. First, the user completes the construction of the retrieval enhancement generation system. The embedding model transforms the text data from the retrieval enhancement generation system into computer-recognizable vector forms, which are then stored in the vector database. When a user initiates a specific question-and-answer request, the retrieval enhancement generation system, based on the user's pre-specified retrieval strategy (e.g., vector retrieval, full-text retrieval, or a hybrid strategy combining the advantages of both), quickly retrieves a set of documents highly relevant to the question-and-answer request from the vector database and forms a candidate document set through initial screening. The retrieval enhancement generation system then calls the reordering model to further refine the ranking of the candidate document set, selecting the most relevant core documents. These ranked core documents are then input as supplementary knowledge into the large language model inference engine. The large language model inference engine is responsible for format adaptation, inference scheduling, and resource allocation of supplementary knowledge. It then efficiently inputs the processed supplementary knowledge into the large language model. The large language model combines its own knowledge base with the supplementary knowledge to generate a targeted answer. The large language model inference engine simultaneously optimizes the output and schedules the transmission of the answer, and finally returns the answer to the user.

[0038] To better utilize the search enhancement generation system, we conducted comprehensive and accurate performance tests on it, which will facilitate more precise guidance in the selection of different search enhancement generation systems.

[0039] In related technologies, when testing multiple search enhancement generation systems, the tests are usually conducted using a single performance metric corresponding to each system. This single metric could be a first-token latency metric, a single-token generation time metric, or the number of tokens generated per second per user.

[0040] While the aforementioned single metrics can reflect some operational characteristics of a retrieval enhancement generation system within a specific dimension, they have significant limitations: Firstly, these single metrics can only represent the system's local performance and cannot comprehensively quantify the system's overall capabilities under multi-component collaboration and complex business scenarios. Secondly, the single metric data from different test scenarios for various retrieval enhancement generation systems lack comparability. Therefore, the testing methods in related technologies are difficult to obtain accurate detection results for different retrieval enhancement generation systems.

[0041] Therefore, addressing the problems in related technologies, this application, based on the collaborative operation characteristics of multi-component search enhancement generation systems and the actual operational needs under complex business scenarios, introduces different concurrency levels to simulate real load scenarios. It comprehensively captures the runtime information of multiple preset test events for each search enhancement generation system under different load pressures, forming a runtime time series vector. Using this vector as a basis, a benchmark runtime time series vector is constructed. After normalization to eliminate data differences, the event-level performance of each search enhancement generation system under different concurrency scenarios is quantified, and event scores are obtained. A comprehensive score is then derived based on these event scores, ultimately presenting the overall performance level of each search enhancement generation system intuitively through the comprehensive score. This overcomes the limitations of single-indicator testing and the incomparability of data, thereby obtaining accurate test results for multiple search enhancement generation systems.

[0042] The specific hardware architecture upon which the test method for the retrieval enhancement generation system depends is described here. (References) Figure 1 , Figure 1 A hardware architecture diagram provided for this application includes a processor 01 and a memory 02, wherein the processor 01 and the memory 02 are connected.

[0043] The memory 02 is used to store a computer program, and the processor 01 is used to execute the computer program to implement the steps of the test method of the search enhancement generation system of this application.

[0044] It is understood that this application does not limit the number or type of processor 01 and memory 02, which can be determined according to the actual application.

[0045] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Please see Figure 2 , Figure 2 This is a flowchart illustrating a testing method for a retrieval enhancement generation system provided in an embodiment of this application. The execution entity of this method can be a testing device for the retrieval enhancement generation system. This testing device can be implemented using a computer program, a medium storing the relevant computer program (such as an optical disc), or a physical device integrating or installing the relevant computer program, such as an electronic device like a server, server cluster, or smart terminal. The method may include the following steps:

[0047] S201. Obtain multiple search enhancement generation systems to be tested.

[0048] To facilitate understanding of this embodiment, the test architecture will first be described. Please refer to [link to documentation]. Figure 3 , Figure 3 This is a schematic diagram of a test architecture provided for an embodiment of the present application, including a test system and multiple search enhancement generation systems to be tested.

[0049] The testing system may include a request module and an analysis module, while each retrieval enhancement generation system to be tested may include an event recording module and an event log module.

[0050] The request module accesses the services of the corresponding search enhancement generation system through the external service interface, i.e., the application programming interface (API), provided by each search enhancement generation system to be tested.

[0051] The event logging module captures and records the time points of preset test events during the response process of each retrieval enhancement generation system under test. These preset test events can be key events that affect the test results.

[0052] The event log module is used to store the time node information of the preset test events corresponding to each search enhancement generation system.

[0053] The analysis module is used to read the time node information of the preset test events corresponding to each retrieval enhancement generation system from the event log module, and to analyze the time node information of the preset test events corresponding to each retrieval enhancement generation system to obtain the test results of each retrieval enhancement generation system.

[0054] S202. Test each retrieval enhancement generation system under different concurrency levels to obtain the runtime time series vector of each retrieval enhancement generation system under different concurrency levels. The runtime time series vector contains test duration information of multiple preset test events.

[0055] To ensure that all search enhancement generation systems operate under the same testing scenario, this embodiment sets up the same concurrency test scenario for each system. The concurrency test scenario includes single-user and multi-user concurrent test scenarios. The multi-user concurrent test scenario further includes multiple sets of different concurrency levels, used to indicate low-concurrency, medium-concurrency, and high-concurrency scenarios, respectively.

[0056] The preset test events can be vector retrieval events, full-text retrieval events, fine-grained ranking events of the re-ranking model, and inference events of the large language model. It is understood that the above-mentioned preset test events are for illustrative purposes only and do not limit this application.

[0057] Taking the aforementioned preset test events as examples, the test duration information for the vector retrieval event can be the total duration of the vector retrieval process from start to finish. The test duration information for the full-text retrieval event can be the total duration of the full-text retrieval process from start to finish. The test duration information for the re-ranking model's refined sorting event can be the total duration of the re-ranking model's re-sorting of the initial screening results from start to finish. The test duration information for the large language model's inference event can be the first token generation duration and the average generation duration per token. Specifically, the first token generation duration is the time from when the large language model receives the input until the first token is generated, and the average generation duration per token is the average generation time of each token from when the large language model generates the first token until the last token is generated.

[0058] The test duration information for each preset test event corresponding to the search enhancement generation system can be recorded by the event logging module in the following ways.

[0059] The event logging module is pre-configured with a set of interface rules. Each search enhancement generation system can apply these rules and call the interface functions within them. These interface functions can be configured as follows: Figure 4 As shown, Figure 4 This application provides an example of an interface function diagram, which includes an initialization function and an event recording function.

[0060] The `Initialize` function initializes the event logging functionality, creates a log file, and returns a file descriptor. Its input parameters are the path and filename of the event log file, and its return type is a file descriptor. The `RecordEvent` function records event logs, and its input parameters are the aforementioned file descriptor and an event structure (Struct Event).

[0061] Please see Figure 5 , Figure 5 This is a schematic diagram of an event structure provided in an embodiment of this application, including the identity information of the question-and-answer request (RequestID), event type (EventType), event status (EventStatus), event occurrence time (EventTimestamp), and other additional attributes. The identity information can be a string, the event occurrence time can be a double-precision floating-point number, the event type and event status can be enumeration types, and other additional attributes can be user-defined.

[0062] Please see Figure 6 , Figure 6 This is a schematic diagram of an event type provided in an embodiment of this application. Taking the above-mentioned preset event as an example, it may include vector retrieval event, full-text retrieval event, reranking model fine-grained ranking event, and large language model inference event.

[0063] Please see Figure 7 , Figure 7 This is a schematic diagram of an event state provided in an embodiment of the present application. It may include an event start state (Start), an event end state (Completion), and an intermediate state (Intermediate). Intermediate_1 indicates that there is only one intermediate state. In this embodiment, the number of intermediate states is not limited.

[0064] Therefore, based on the above configuration, the event logging module can record the test duration information of the preset test events corresponding to each retrieval enhancement generation system and store it in the event log module.

[0065] To facilitate a better understanding of the steps in this embodiment, the following will be combined with... Figure 8 , Figure 8 This is a schematic diagram illustrating a testing process provided in an embodiment of this application. Figure 8 As shown, the testing process can be divided into three stages: initialization stage, question and answer request testing stage, and calculation and analysis stage.

[0066] During the initialization phase, the test system initializes itself based on parameters pre-input by the user. These parameters include, but are not limited to: the search enhancement generation systems to be tested, the corresponding API interfaces for each system, a list of concurrency levels, preset weights for each concurrency level, the dataset file path, the number of questions and answers (question_num) for each question-answering request, and the time interval between receiving a response in one request and initiating the next request (interval). Each search enhancement generation system to be tested then initializes itself, starting the corresponding system and loading the event logging module based on the pre-input path and filename of each preset event log file.

[0067] During the question-and-answer request testing phase, the test system sequentially sends question-and-answer requests to each search enhancement generation system. For any given search enhancement generation system, a question-and-answer request is first sent to that system using a single-user test scenario from the concurrency list. Then, questions-and-answer requests are sent sequentially to that system using concurrency groups with different concurrency levels from the multi-user concurrent test scenarios in the concurrency list, until all concurrency levels in the concurrency list have been tested. At this point, the question-and-answer request test for that search enhancement generation system is complete, and the event logging module of that system records the test duration information for multiple preset test events. This process is repeated until all search enhancement generation systems have been tested.

[0068] During the calculation and analysis phase, the test system obtains the test duration information of multiple preset test events from the event log module corresponding to each retrieval enhancement generation system. Then, based on the test duration information of multiple preset test events corresponding to each retrieval enhancement generation system, it obtains the runtime time series vector of each retrieval enhancement generation system under different concurrency levels.

[0069] S203. Determine the baseline running time series vector based on the running time series vectors of each retrieval enhancement generation system under different concurrency levels.

[0070] Based on the test duration information of multiple preset test events contained in the runtime time series vectors of each retrieval enhancement generation system under different concurrency levels, the minimum test duration information of each preset test event is determined.

[0071] A baseline runtime time series vector is generated based on the order of the preset test events in the runtime time series vector and the minimum test duration information of each preset test event.

[0072] S204. Normalize the running time series vectors of each retrieval enhancement generation system under different concurrency levels and the baseline running time series vector to obtain the normalized running time series vectors of each retrieval enhancement generation system under different concurrency levels and the normalized baseline running time series vector.

[0073] Organize the runtime time series vectors of each retrieval enhancement generation system under different concurrency levels and the established baseline runtime time series vectors to ensure that the vector dimensions are consistent, i.e., all contain the duration information of the same preset test events.

[0074] The Min-Max normalization method is selected to adapt to the duration-based data, mapping the data to the [0,1] interval. Batch calculations are performed one by one along the vector dimension, normalizing the duration data of each duration data for the runtime time series vectors under different concurrency levels of each retrieval enhancement generation system and the baseline runtime time series vector to obtain the corresponding results. Finally, the results are integrated, combining the normalized duration data of the same retrieval enhancement generation system under different concurrency levels according to the original event order to generate the normalized runtime time series vector for the corresponding concurrency level of the retrieval enhancement generation system, while simultaneously outputting the normalized baseline runtime time series vector, thus completing the normalization process.

[0075] S205. For any retrieval enhancement generation system, determine the event score of the retrieval enhancement generation system under different concurrency levels based on the normalized baseline running time series vector and the normalized running time series vector of the retrieval enhancement generation system under different concurrency levels.

[0076] For any retrieval enhancement generation system, based on a preset algorithm or a pre-trained model, the event scoring scores of the retrieval enhancement generation system under different concurrency levels are finally determined according to the normalized baseline running time series vector and the normalized running time series vector of the retrieval enhancement generation system under different concurrency levels.

[0077] S206. Determine the comprehensive score of each retrieval enhancement generation system based on the event score of each retrieval enhancement generation system under different concurrency levels.

[0078] One possible implementation is that, during the calculation and analysis phase, the test system inputs the event score of each retrieval enhancement generation system under different concurrency levels and the preset weight coefficients under different concurrency levels into a preset weighting algorithm for weighted calculation, obtains the weighted result of each retrieval enhancement generation system, and determines the corresponding weighted result of each retrieval enhancement generation system as the comprehensive score of each retrieval enhancement generation system.

[0079] Optionally, the preset weighting algorithm can be a weighted geometric mean, as shown in formula (1):

[0080] (1)

[0081] Where i represents the i-th group of concurrency in the concurrency list; This represents the preset weight corresponding to the number of concurrent requests in the i-th group.

[0082] The comprehensive score for each search enhancement generation system can be determined using the formula described above.

[0083] Another possible implementation is to input the event score of each retrieval enhancement generation system under different concurrency levels into a preset mean scoring algorithm to obtain the mean score of each retrieval enhancement generation system, and then determine the mean score of each retrieval enhancement generation system as the comprehensive score of each retrieval enhancement generation system.

[0084] The preset mean scoring algorithm can be the arithmetic mean algorithm. For any search enhancement generation system, the event score of the search enhancement generation system under different concurrency is input into the configured mean scoring algorithm. The mean scoring algorithm sums up the event score under different concurrency and obtains the sum result. The sum result is divided by the number of event score, and the final output result is the comprehensive score of the search enhancement generation system.

[0085] Based on the steps described above, a comprehensive score can be obtained for each search enhancement generation system.

[0086] S207. Determine the test results of each retrieval enhancement generation system based on the comprehensive score of each retrieval enhancement generation system.

[0087] Optionally, the comprehensive scores of each search enhancement generation system are sorted in ascending order to obtain a ranking result that includes the identification information of each search enhancement generation system and its corresponding comprehensive score. The ranking result indicates the selection priority of each search enhancement generation system, and the selection priority indicates the test results of each search enhancement generation system.

[0088] Optionally, based on the comprehensive score of each search enhancement generation system, at least one scoring threshold is preset, and the performance level is divided according to the correspondence between the comprehensive score of each system and the scoring threshold. The information containing the comprehensive score and the performance level is used as the test result of each search enhancement generation system.

[0089] Optionally, based on the comprehensive scores of each search enhancement generation system, the mean or median of the comprehensive scores of each system under test is calculated, and the degree of performance difference is determined by the difference between the comprehensive scores of each system and the mean or median. The information including the comprehensive scores and the degree of performance difference is used as the test results of each search enhancement generation system.

[0090] Based on the sorting results, other operations besides selection can also be performed, such as determining which search enhancement generation systems need to be adjusted and which do not.

[0091] In the above embodiments of this application, runtime time series vectors containing multiple preset test event durations for each retrieval enhancement generation system are obtained by testing under different concurrency levels. A baseline runtime time series vector is determined based on these vectors. The runtime time series vectors of each retrieval enhancement generation system under different concurrency levels and the baseline vector are then normalized to obtain normalized runtime time series vectors and a normalized baseline runtime time series vector for each system under different concurrency levels. For any retrieval enhancement generation system, an event score is determined based on the normalized baseline and normalized runtime time series vectors. Finally, a comprehensive score is determined based on these event scores, and the test results for each retrieval enhancement generation system are determined according to their comprehensive scores. The method in this application takes into account the performance of the retrieval enhancement generation system under different load scenarios, i.e. different concurrency, and the running characteristics of multiple test events, which greatly improves the accuracy of the test results of each retrieval enhancement generation system compared with single index testing.

[0092] Furthermore, based on the above embodiments, the following embodiments illustrate the process of testing each retrieval enhancement generation system under different concurrency levels to obtain the runtime time series vectors of each retrieval enhancement generation system under different concurrency levels.

[0093] Please see Figure 9 , Figure 9 A flowchart illustrating a method for obtaining runtime vectors of various retrieval enhancement generation systems under different concurrency levels, provided in this application embodiment, includes the following steps:

[0094] S901. Obtain a preset list of concurrent connections. The list of concurrent connections includes multiple concurrent groups, and each concurrent group includes a different number of concurrent connections.

[0095] The concurrency list includes multiple concurrency groups, one of which contains only one concurrency and is used for single-user scenario testing, while the remaining concurrency groups contain different numbers of concurrency and are used for multi-user scenario testing.

[0096] S902. For any retrieval enhancement generation system, based on the number of concurrent users included in each concurrency group, test the retrieval enhancement generation system under different concurrency groups to obtain the runtime time series vector of the retrieval enhancement generation system under different concurrency numbers.

[0097] The following will combine Figure 10 For an explanation of this step, please refer to [link / reference]. Figure 10, Figure 10 A flowchart illustrating another method for obtaining runtime vectors of a retrieval enhancement generation system under different concurrency levels, provided in this application embodiment, may include the following steps:

[0098] S1001, Read the application programming interface of the currently detected enhanced generation system.

[0099] For any concurrent group of the enhanced generation system, the test system simultaneously sends the same number of question-and-answer requests to the retrieval enhanced generation system, based on the number of concurrent users included in the concurrent group. Each question-and-answer request is sent sequentially based on a preset number of question-and-answer attempts.

[0100] S1002, Conduct single-user scenario testing.

[0101] The testing system first conducts a single-user scenario test by initiating a question-and-answer request to the enhanced generation system and sending the corresponding number of questions and answers according to the preset number of questions and answers in the question-and-answer request.

[0102] It should be noted that after a response is received for a request, the next request will be sent after a preset interval, until the preset number of requests and responses is reached.

[0103] S1003, Conduct multi-user concurrent scenario testing.

[0104] The test system sequentially performs multi-user concurrent scenario tests, based on the execution order of the concurrent groups in the concurrency list. The number of concurrent users in each concurrent group increases sequentially according to the execution order.

[0105] The test system reads the concurrency array from the concurrency list in sequence. Based on the current concurrency count in the concurrency array, it simultaneously sends multiple question-and-answer requests to the enhanced generation system, each corresponding to the number of concurrency counts. Each question-and-answer request is sent a corresponding number of times according to the preset number of questions and answers.

[0106] S1004. Determine whether all the multiple sets of concurrency included in the multi-user concurrency scenario test have been tested.

[0107] The testing system records the number of concurrent groups and the execution order of each concurrent group corresponding to the multi-user scenario in the preset concurrency list. After completing the test of each group of concurrent groups in sequence, it marks its completion status. When the test of each group of concurrent groups is marked as completed, it is determined that the test of multiple groups of concurrent groups in the multi-user concurrent scenario has been completed.

[0108] S1005. If not, continue with S1003.

[0109] S1006. If so, then for any question-and-answer request, obtain the runtime sequence vector corresponding to the number of questions and answers for each question and answer request under each question-and-answer request of the retrieval enhancement generation system.

[0110] The test system concludes the test of the retrieval enhancement generation system and obtains the runtime time series vector corresponding to the number of questions and answers for each question and answer request from the event log module of the retrieval enhancement generation system.

[0111] S1007. Based on the runtime time series vector corresponding to the number of questions and answers under each question and answer request, determine the runtime time series vector of the retrieval enhancement generation system under each concurrency level.

[0112] Based on the test duration information of multiple preset test events contained in the runtime time series vector corresponding to the number of questions and answers under each question and answer request, the average test duration information of each preset test event is determined.

[0113] For example, the preset test events are the vector retrieval event, full-text retrieval event, reordering model fine-grained ranking event, and large language model inference event in the above embodiments.

[0114] The time node information of multiple preset test events contained in the runtime time series vector corresponding to the number of questions and answers under each question and answer request can be obtained in the following way:

[0115] For any number of question-and-answer sessions, at the start of the corresponding vector retrieval operation, obtain the current time (Time) and write the event information (identity information of the requesting question and answer, vector retrieval, start status, and start time) into the log.

[0116] When the vector retrieval operation ends, obtain the current time (Time) and write the event information (identity information of the requester, vector retrieval, end status, and end time) to the log.

[0117] When the full-text search operation begins, obtain the current time (Time) and write the event information (identity information of the requester, full-text search, start status, and start time) into the log.

[0118] When the full-text search operation ends, obtain the current time (Time) and write the event information (identity information of the requester, full-text search, end status, and end time) into the log.

[0119] When the fine-grained sorting operation of the reordering model begins, obtain the current time (Time) and write the event information (identity information of the requester, fine-grained sorting of the reordering model, start status, and start time) into the log.

[0120] When the fine-grained sorting operation of the reordering model ends, obtain the current time (Time) and write the event information (identity information of the requester, fine-grained sorting of the reordering model, end status, and end time) into the log.

[0121] When the large language model inference operation begins, obtain the current time (Time) and write the event information (identity information of the requester, large language model inference, start state, and start time) into the log.

[0122] During the large language model inference process, when the first token is returned, the current time (Time) corresponding to the return is obtained, and the event information (identity information of the requester, large language model inference, intermediate state, and time) is written to the log.

[0123] After the large language model inference operation is completed, obtain the current time (Time), obtain the total number of returned tokens (tokens_num), and write the event information (identity information of the requester, large language model inference, end status, end time, and total number of tokens) into the log.

[0124] For any number of question-and-answer sessions, the test duration information for each preset test event is determined based on the identity information of the requester corresponding to that number of question-and-answer sessions.

[0125] Specifically, the vector retrieval duration is obtained by subtracting the start time from the start time of the event information (identity information of the request and answer, vector retrieval, end status, end time) from the end time of the event information (identity information of the request and answer, vector retrieval, start status, start time).

[0126] Subtract the start time from the start time in the event information (identity information of the requester, full-text search, end status, end time) to obtain the full-text search duration.

[0127] Subtract the start time from the start time from the end time in the event information (identity information of the request / answer, fine-grained sorting of the reordering model, end state, end time) to obtain the reordering duration.

[0128] Subtract the start time from the start time in the event information (identity information of the request and answer, large language model inference, intermediate state, time) to obtain the generation time of the first token of the large model inference.

[0129] Divide the difference between the end time in the event information (identity information of the request and answer, large language model inference, end state, end time, total number of tokens) and the time in the event information (identity information of the request and answer, large language model inference, intermediate state, time) by the difference value, which is the total number of tokens minus 1, to obtain the average generation time per token for large model inference.

[0130] Using the above calculation method, we can obtain the test duration information for each preset test event corresponding to any number of question and answer attempts, namely, vector retrieval duration, full-text retrieval duration, reordering duration, first token generation duration of large model inference, and average generation duration per token of large model inference.

[0131] For any preset test event, the average test duration information of the preset test event is determined as the target test duration information of the preset test event.

[0132] For any preset test event, based on the test duration information corresponding to the number of questions and answers under each question and answer request for that preset test event, the mean of each test duration information is calculated, and the mean, i.e., the average test duration information, is determined as the target test duration information for that preset test event. Thus, the target test duration information corresponding to each preset test event is obtained.

[0133] Based on the arrangement order of each preset test event in the runtime time series vector, assuming the order is vector retrieval time, full-text retrieval time, reordering time, first token generation time of large model inference, and average generation time per token of large model inference, and the target test time information of each preset test event determined above, a target runtime time series vector is generated. This target runtime time series vector is the runtime time series vector of the retrieval enhancement generation system under the concurrency.

[0134] S1008. Determine whether all the search enhancement generation systems to be tested have completed the test.

[0135] The testing system pre-records the total number of search enhancement generation systems to be tested. After completing the testing process of a single search enhancement generation system, it marks its completion status. When the number of marked completed systems is consistent with the total number of systems to be tested, it is determined that all search enhancement generation systems to be tested have completed the test.

[0136] S1009. If all are completed, the test ends.

[0137] S1010. If not completed, proceed with the steps S1001 to S1008 for the retrieval enhancement generation system that has not completed the test.

[0138] In the above embodiments of this application, a pre-defined concurrency list containing multiple groups of different concurrency levels is used to initiate a corresponding number of concurrent question-and-answer requests for each retrieval enhancement generation system according to the concurrency group. Multi-dimensional data is collected based on a pre-defined number of question-and-answer sessions, and the average duration of each pre-defined test event is calculated based on the runtime time series vector of multiple rounds of requests. Finally, a target runtime time series vector under the corresponding concurrency level is generated. This method covers different load scenarios with multiple concurrency groups, reducing the bias of single-concurrency testing. By using multiple rounds of question-and-answer requests and mean calculation, random errors are reduced, improving the reliability of test data and providing an accurate basis for subsequent scoring.

[0139] Furthermore, based on any of the above embodiments, the following embodiments illustrate the process of determining the event scoring scores of each retrieval enhancement generation system under different concurrency levels based on the runtime time series vectors of each retrieval enhancement generation system under different concurrency levels.

[0140] Please see Figure 11 , Figure 11 A flowchart illustrating a method for determining event scores of various retrieval enhancement generation systems under different concurrency levels, provided in this application embodiment, includes the following steps:

[0141] S1101. Determine the baseline running time series vector based on the running time series vectors of each retrieval enhancement generation system under different concurrency levels.

[0142] Based on the runtime sequence vectors generated by each retrieval enhancement generation system in different concurrency tests, the test duration of each preset test event is extracted. For any preset test event, the test durations of each preset test event are compared, and the minimum value is determined as the minimum test duration information of the preset test event.

[0143] For example, the preset test events are also the vector retrieval event, full-text retrieval event, reordering model fine-grained ranking event, and large language model inference event in the above embodiments.

[0144] From the test duration information of each vector retrieval event, determine the minimum test duration information, i.e., Min{vector retrieval duration}.

[0145] From the test duration information of each full-text search event, determine the minimum test duration information, namely Min{full-text search duration}.

[0146] From the test duration information of each event in the refined sorting of the reordering model, determine the minimum test duration information, namely Min{reordering duration}.

[0147] From the first token generation time information of the large language model inference event, determine the minimum first token generation time information, i.e., Min{large model inference first token generation time}, and from the average per token generation time information, determine the minimum average per token generation time information, i.e., Min{large model inference average per token generation time}.

[0148] A baseline runtime time series vector is generated based on the order of the preset test events in the runtime time series vector and the minimum test duration information of each preset test event.

[0149] The baseline runtime sequence vector is: (Min{vector retrieval time}, Min{full-text retrieval time}, Min{reordering time}, Min{first token generation time for large model inference}, Min{average generation time per token for large model inference}).

[0150] S1102. Normalize the running time series vectors of each retrieval enhancement generation system under different concurrency levels and the baseline running time series vector to obtain the normalized running time series vectors of each retrieval enhancement generation system under different concurrency levels and the normalized baseline running time series vector.

[0151] To ensure that all components have a uniform scale, and to better measure the impact of each preset detection event on the retrieval enhancement generation system, such as... Figure 12 As shown, Figure 12 This is a schematic diagram of a normalization process provided in an embodiment of this application. Figure 12 Except for the last row, each row represents the runtime time series vector corresponding to different retrieval enhancement generation systems. The last row represents the baseline runtime time series vector, and each component is normalized. The method involves normalizing the components of the runtime time series vectors of each retrieval enhancement generation system and the baseline runtime time series vector at a certain concurrency level.

[0152] S1103. For any retrieval enhancement generation system, determine the event score of the retrieval enhancement generation system under different concurrency levels based on the normalized baseline running time series vector and the normalized running time series vector of the retrieval enhancement generation system under different concurrency levels.

[0153] For any concurrency level of the retrieval-enhanced generation system:

[0154] The normalized runtime time series vector under the concurrency level and the normalized baseline runtime time series vector are input into the preset Euclidean distance algorithm for calculation to obtain the first calculation result.

[0155] Alternatively, the normalized runtime time series vector under the concurrency level and the normalized baseline runtime time series vector can be input into a preset Manhattan distance algorithm for calculation to obtain the second calculation result.

[0156] Alternatively, the normalized runtime time series vector under the concurrency level and the normalized baseline runtime time series vector can be preprocessed to obtain the preprocessed normalized runtime time series vector under the concurrency level and the preprocessed normalized baseline runtime time series vector.

[0157] The preprocessed normalized running time series vector under the concurrency level and the preprocessed normalized baseline running time series vector are input into the pre-trained score prediction model, and the third calculation result is output.

[0158] The first, second, or third operation result is determined as the event score of the retrieval enhancement generation system under the concurrency level.

[0159] By using any of the methods—Euclidean distance, Manhattan distance, or fractional prediction models—the performance gap between the running time series vector of the enhanced retrieval system and the benchmark running time series vector is accurately quantified, providing reliable data for the final comprehensive score.

[0160] In the embodiments described above, a baseline runtime time series vector is determined based on the runtime time series vectors of each retrieval enhancement generation system under different concurrency levels. Both vectors are then normalized. Finally, the event score is calculated based on the normalized baseline runtime time series vector and the normalized runtime time series vectors of each retrieval enhancement generation system. This eliminates the dimensional differences in test event durations, improves the objectivity of the scoring and the comparability of the performance of each system, provides accurate intermediate data for subsequent comprehensive scoring, and ensures the reliability of multi-system performance evaluation results.

[0161] Please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a test apparatus for a retrieval enhancement generation system provided in an embodiment of this application. Figure 13 As shown, it includes:

[0162] The acquisition module is used to acquire multiple search enhancement generation systems to be tested.

[0163] The testing module is used to test each retrieval enhancement generation system under different concurrency levels, and obtain the runtime time series vector of each retrieval enhancement generation system under different concurrency levels. The runtime time series vector contains the test duration information of multiple preset test events.

[0164] The determination module is used to determine the baseline runtime time series vector based on the runtime time series vectors of each retrieval enhancement generation system under different concurrency levels.

[0165] The determination module is used to normalize the runtime time series vectors of each retrieval enhancement generation system under different concurrency levels and the baseline runtime time series vector, so as to obtain the normalized runtime time series vectors of each retrieval enhancement generation system under different concurrency levels and the normalized baseline runtime time series vector.

[0166] The determination module is used to determine the event score of the retrieval enhancement generation system under different concurrency levels, based on the normalized baseline runtime time series vector and the normalized runtime time series vector of the retrieval enhancement generation system under different concurrency levels.

[0167] The determination module is also used to determine the comprehensive score of each retrieval enhancement generation system based on the event score of each retrieval enhancement generation system under different concurrency levels.

[0168] The determination module is also used to determine the test results of each retrieval enhancement generation system based on the comprehensive score of each retrieval enhancement generation system.

[0169] For a description of the features in the embodiment corresponding to the test apparatus of the retrieval enhancement generation system, please refer to the relevant description of the embodiment corresponding to the test method of the retrieval enhancement generation system, which will not be repeated here.

[0170] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 14 As shown, the electronic device provided in this embodiment includes, optionally, a communication component 03 in addition to a processor 01 and a memory 02. The processor 01, memory 02, and communication component 03 are connected via a bus 04.

[0171] In the specific implementation process, at least one processor 01 executes computer execution instructions stored in memory 02, causing at least one processor 01 to execute the above-described test method embodiment of the retrieval enhancement generation system.

[0172] The specific implementation process of processor 01 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0173] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0174] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0175] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0176] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the test method embodiments of any of the above-described retrieval enhancement generation systems when run.

[0177] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0178] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described test method embodiments for a retrieval enhancement generation system.

[0179] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described test method embodiments for a retrieval enhancement generation system.

[0180] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] The foregoing has provided a detailed description of the testing method and electronic device for a retrieval enhancement generation system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A testing method for a retrieval enhancement generation system, characterized in that, include: Obtain multiple search enhancement generation systems to be tested; Obtain a preset list of concurrent connections, which includes multiple concurrent groups, each containing a different number of concurrent connections; For any retrieval enhancement generation system, based on the number of concurrent processes included in each concurrency group, the retrieval enhancement generation system is tested under different concurrency groups to obtain the test duration information of multiple preset test events under each of the multiple concurrency groups. Based on the test duration information of the multiple preset test events, the running time series vector of the retrieval enhancement generation system under each concurrency group is obtained; the multiple preset test events include vector retrieval events, full-text retrieval events, re-ranking model fine-grained ranking events, and large language model inference events; Based on the running time series vectors of each retrieval enhancement generation system under different concurrency levels, a baseline running time series vector is determined; The running time series vectors of each retrieval enhancement generation system under different concurrency levels and the benchmark running time series vector are normalized to obtain the normalized running time series vectors of each retrieval enhancement generation system under different concurrency levels and the normalized benchmark running time series vector. For any retrieval enhancement generation system, the event score of the retrieval enhancement generation system under different concurrency levels is determined based on the normalized baseline runtime time series vector and the normalized runtime time series vector of the retrieval enhancement generation system under different concurrency levels. The overall score of each retrieval enhancement generation system is determined based on the event score of each system under different concurrency levels. The test results for each retrieval enhancement generation system are determined based on the comprehensive scores of each system.

2. The method according to claim 1, characterized in that, For any retrieval enhancement generation system, based on the number of concurrent users included in each concurrency group, the retrieval enhancement generation system is tested under different concurrency groups to obtain the runtime vector of the retrieval enhancement generation system under different concurrency levels, including: For any concurrent group of the search enhancement generation system, the number of question-and-answer requests are simultaneously sent to the search enhancement generation system according to the number of concurrent users included in the concurrent group, and each question-and-answer request is sent sequentially based on a preset number of question-and-answer requests; For any question-and-answer request, obtain the runtime time series vector corresponding to the number of questions and answers under the question-and-answer request of the retrieval enhancement generation system; Based on the runtime time series vector corresponding to the number of questions and answers under each question and answer request, the runtime time series vector of the retrieval enhancement generation system under the concurrency is determined.

3. The method according to claim 2, characterized in that, The step of determining the runtime vector of the retrieval enhancement generation system under the concurrency level based on the runtime vector corresponding to the number of questions and answers under each question and answer request includes: Based on the test duration information of multiple preset test events contained in the runtime time series vector corresponding to the number of questions and answers under each question and answer request, the average test duration information of each preset test event is determined. For any preset test event, the average test duration information of the preset test event is determined as the target test duration information of the preset test event; Based on the arrangement order of each preset test event in the runtime time series vector and the target test duration information of each preset test event, a target runtime time series vector is generated. The target runtime time series vector is the runtime time series vector of the retrieval enhancement generation system under the specified concurrency.

4. The method according to claim 1, characterized in that, The step of determining the baseline runtime vector based on the runtime vectors of each retrieval enhancement generation system under different concurrency levels includes: Based on the test duration information of multiple preset test events contained in the runtime time series vector of each retrieval enhancement generation system under different concurrency levels, the minimum test duration information of each preset test event is determined. The baseline running time series vector is generated based on the arrangement order of each preset test event in the running time series vector and the minimum test duration information of each preset test event.

5. The method according to claim 1, characterized in that, For any retrieval enhancement generation system, based on the normalized baseline runtime time series vector and the normalized runtime time series vector of the retrieval enhancement generation system under different concurrency levels, the event scoring score of the retrieval enhancement generation system under different concurrency levels is determined, including: For any concurrency level of the retrieval enhancement generation system: The normalized running time series vector under the concurrency number and the normalized baseline running time series vector are input into a preset Euclidean distance algorithm for calculation to obtain the first calculation result; Alternatively, the normalized runtime time series vector under the concurrency level and the normalized baseline runtime time series vector can be input into a preset Manhattan distance algorithm for calculation to obtain a second calculation result; Alternatively, the normalized runtime time series vector under the stated concurrency level and the normalized baseline runtime time series vector can be preprocessed to obtain the preprocessed normalized runtime time series vector under the stated concurrency level and the preprocessed normalized baseline runtime time series vector. The preprocessed normalized running time series vector under the concurrency level and the preprocessed normalized baseline running time series vector are input into the pre-trained score prediction model, and the third calculation result is output. The first calculation result, the second calculation result, or the third calculation result is determined as the event score of the retrieval enhancement generation system under the number of concurrent requests.

6. The method according to any one of claims 1-5, characterized in that, The comprehensive score of each retrieval enhancement generation system is determined based on the event score of each system under different concurrency levels, including: Obtain the preset weight coefficients for different concurrency levels; The event score of each search enhancement generation system under different concurrency levels and the preset weight coefficients under the different concurrency levels are input into a preset weighting algorithm for weighted calculation to obtain the weighted result of each search enhancement generation system; The weighted results of each search enhancement generation system are used to determine the comprehensive score of each search enhancement generation system.

7. The method according to any one of claims 1-5, characterized in that, The comprehensive score of each retrieval enhancement generation system is determined based on the event score of each system under different concurrency levels, including: The event score of each retrieval enhancement generation system under different concurrency levels is input into the preset mean score algorithm to obtain the mean score of each retrieval enhancement generation system. The average score of each search enhancement generation system is used as the comprehensive score for each search enhancement generation system.

8. The method according to claim 1, characterized in that, The process of determining the test results of each retrieval enhancement generation system based on the comprehensive score of each system includes: The comprehensive scores of each search enhancement generation system are sorted in ascending order to obtain a sorting result that includes the identification information of each search enhancement generation system and its corresponding comprehensive score; The sorting results are used to indicate the selection priority of each retrieval enhancement generation system, and the selection priority is used to indicate the test results of each retrieval enhancement generation system.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the test method for the retrieval enhancement generation system as described in any one of claims 1 to 8 when executing the computer program.

Images