Engine performance test method, system, equipment and medium

Abstract

The invention provides an engine performance test method, system and device and a medium, and relates to the technical field of tests.The method comprises the steps that a test task is created according to test requirement information, and task parameters of the test task comprise a test data set, an evaluation scheme and a to-be-tested engine list; according to each test data in the test data set, performing performance test on each to-be-tested engine in the to-be-tested engine list to obtain a test output result of each to-be-tested engine; and outputting a test report corresponding to the test task according to the task type of the test task, the test output result and the evaluation scheme. According to the method, full-automatic execution of the performance test process of each engine is realized, the test efficiency is remarkably improved, the accuracy and objectivity of the test result are ensured, meanwhile, the traceability of the result is enhanced by the automatically generated test report, subsequent analysis and problem reproduction are facilitated, and the test efficiency is improved. The problems that traditional manual testing is low in efficiency and inaccurate in result are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of testing technology, and in particular to an engine performance testing method, system, equipment and medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, AI engines for various natural language processing tasks (such as speech translation, recognition, and synthesis) have been widely used in various natural language processing scenarios. The performance of these AI engines is directly related to task performance and user experience. Therefore, how to perform performance testing of AI engines has become particularly important.

[0003] Currently, performance testing of AI engines mostly requires manual testing by testers, meaning they manually execute the test process. This testing method is not only inefficient, but also prone to human interference, making it difficult to ensure accurate testing. Summary of the Invention

[0004] The present invention provides an engine performance testing method, system, device and medium to solve the defects in the prior art.

[0005] The present invention provides an engine performance testing method, comprising: Create a test task based on the test requirement information, where the task parameters of the test task include a test data set, an evaluation plan, and a list of engines to be tested; Performing a performance test on each engine to be tested in the list of engines to be tested according to each test data in the test data set, and obtaining a test output result of each engine to be tested; Output a test report corresponding to the test task according to the task type of the test task, the test output result and the evaluation plan.

[0006] According to a performance testing method for an engine provided by the present invention, outputting a test report corresponding to the test task according to the task type of the test task, the test output result, and the evaluation scheme includes: When the task type is a translation task, obtaining an evaluation mode according to the test requirement information, wherein the evaluation mode includes a mode of independently evaluating each engine and / or a mode of comparatively evaluating multiple engines; Generate first prompt information according to the evaluation mode and the evaluation plan; The first prompt information, each of the test data, and the test output result are input into a large language model to obtain the test report output by the large language model.

[0007] According to a performance testing method for an engine provided by the present invention, outputting a test report corresponding to the test task according to the task type of the test task, the test output result, and the evaluation scheme includes: When the task type is a speech synthesis task, an image-text recognition task, or a listening and speaking task, determining a target engine with the best performance corresponding to the task type in the engine storage space; The test report is output according to the evaluation plan, the test output results of each of the engines to be tested, and the test output results of the target engine.

[0008] According to a performance testing method for an engine provided by the present invention, the method performs a performance test on each engine to be tested in the list of engines to be tested based on each test data in the test data set to obtain a test output result of each engine to be tested, including: Dividing the test data set into a plurality of sub-data sets according to data attribute information of each test data in the test data set; According to the data set division result, each subtask under the test task is created; the task parameters of each subtask include a sub-data set, the evaluation plan and the list of engines to be tested; Determine the test process of each subtask according to the task status corresponding to each subtask; Based on the threads corresponding to the subtasks, calling the engines to be tested to execute the test operations corresponding to the subtasks in parallel, and obtaining the test output results of the engines to be tested; The test operations corresponding to the subtasks include, according to the test procedures of the subtasks, executing operations on the test data in the sub-data sets corresponding to the subtasks.

[0009] According to a method for testing engine performance provided by the present invention, the steps of calling each engine to be tested include: Locating the engine folder of each engine to be tested according to the engine type of each engine to be tested; Based on the engine folder, determining the calling scheme of each engine to be tested according to the reflection mechanism; According to the calling scheme, each of the engines to be tested is called.

[0010] According to a performance testing method for an engine provided by the present invention, creating a test task according to test requirement information includes: Parsing the test requirement information to obtain data item requirement information, evaluation requirement information, and engine requirement information; In the data storage space, a plurality of test data whose data attribute information matches the data item requirement information is acquired to construct the test data set; Obtaining the evaluation plan according to the evaluation requirement information; In the engine storage space, multiple engines whose engine types and versions match the engine requirement information are obtained, and a list of engines to be tested is constructed; The test task is created with the test data set, the evaluation plan and the list of engines to be tested as task parameters.

[0011] According to a performance testing method for an engine provided by the present invention, the step of storing any test data in the data storage space includes: generating second prompt information according to the data attribute information of any test data; Inputting the second prompt information and the content description information of the any test data into a large language model to obtain generated data of the any test data output by the large language model; According to the generated data and the data attribute information of the any test data, the any test data is written into the data storage space.

[0012] The present invention also provides an engine performance testing system, comprising: A task creation unit, configured to create a test task according to the test requirement information, wherein the task parameters of the test task include a test data set, an evaluation plan, and a list of engines to be tested; A first testing unit is configured to perform a performance test on each engine to be tested in the list of engines to be tested according to each test data in the test data set, and obtain a test output result of each engine to be tested; The second testing unit is used to output a test report corresponding to the test task according to the task type of the test task, the test output result and the evaluation plan.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described engine performance testing methods when executing the computer program.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned engine performance testing methods.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned engine performance testing methods.

[0016] The engine performance testing method, system, device and medium provided by the present invention automatically create a test task containing task parameters based on test requirement information. These parameters cover a test data set, an evaluation scheme and a list of engines to be tested. Then, the test data set is used to automatically perform a performance test on each engine to be tested in the list of engines to be tested to obtain the test output results of each engine, and a corresponding test report is intelligently generated based on the test task type, output results and evaluation scheme. This achieves the full automation of the engine performance testing process, significantly improves test efficiency, reduces manual intervention, and thus reduces errors caused by human factors, ensuring the accuracy and objectivity of the test results. At the same time, the automatically generated test report enhances the traceability of the results, facilitates subsequent analysis and problem reproduction, and effectively solves the problems of low efficiency and inaccurate results of traditional manual testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow charts of the engine performance testing method provided by the present invention.

[0019] Figure 2 This is the second flow chart of the engine performance testing method provided by the present invention.

[0020] Figure 3 This is the third flow chart of the engine performance testing method provided by the present invention.

[0021] Figure 4 This is the fourth flow chart of the engine performance testing method provided by the present invention.

[0022] Figure 5 This is the fifth flow chart of the engine performance testing method provided by the present invention.

[0023] Figure 6 This is the sixth flow chart of the engine performance testing method provided by the present invention.

[0024] Figure 7 It is a structural diagram of the engine performance testing system provided by the present invention.

[0025] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0027] In recent years, with the vigorous development of artificial intelligence technology, AI engines for various natural language processing tasks (such as speech translation, recognition, synthesis, etc.) have been widely used in various natural language processing scenarios.

[0028] For translators, voice recorders, and other natural language processing-related products on the market, AI engine performance is their core capability. Integrating the best engines on the market into products in a timely manner can effectively improve product performance and enhance business reputation. Furthermore, accurately understanding the capabilities of different AI engine versions helps to select appropriate models for different application scenarios, thereby enhancing the quality of the product's natural language processing tasks. Furthermore, end-to-end AI engine testing can also expose engine issues, allowing for timely targeted improvements and optimizations to improve engine performance and iteration speed. Therefore, efficiently and accurately testing the natural language processing capabilities of these AI engines is particularly important.

[0029] In existing technologies, performance testing of AI engines often requires manual testing by testers. Specifically, testers need to manually perform the following process: The first step is to manually collect test data and organize it to build a test set. Test data collection methods typically include native speaker collection and online collection. However, native speaker collection is costly and time-consuming. Online data collection is quite disorganized in content and format, making it easier to obtain data that has been trained on large models. Furthermore, the collected data requires manual review and labeling to distinguish data usage scenarios. The second step is to manually call the engine to execute the test set and organize the data generated by the engine; The third step involves manually evaluating the data generated by each AI engine using different evaluation methods. For example, translation engines require translation performance evaluation using translation evaluation metrics such as the Bilingual Evaluation Understudy (BLEU), the improved character n-gram F-score (chrF++), and the Transformer-based Bidirectional Encoder Representation Score (BERTScore). The basic logic is to compare the similarity between the reference translation and the translated text and calculate a score. A higher score indicates better translation performance. For audio dictation and image recognition engines, evaluation methods primarily include word accuracy and word precision. The overall evaluation logic compares the recognized text with the reference text, using the following calculation formulas: Accuracy = Number of Correct Characters / Total Number of Characters; Precision = True Positives / (True Positives + False Positives), where true positives are the number of characters correctly recognized by the model, and false positives are the number of characters incorrectly recognized as the target character. For audio synthesis engines, manual evaluation is primarily required.

[0030] The fourth step is to manually organize the evaluation data and write a test report.

[0031] In summary, each step in the existing AI engine performance testing process is often independent, requiring testers to manually execute or write code to execute each step and manually save the data generated by each step. Furthermore, the test data format is chaotic, and different engines have different input and output format requirements, requiring testers to manually or code to adjust the input and output format of each engine. As a result, the entire testing process requires a high investment in human resources, and the evaluation results are easily affected by human factors. This leads to problems such as low testing efficiency, error-prone and inaccurate evaluation results.

[0032] To this end, this embodiment provides an engine performance testing method, which systematically manages test data, test data sets, test tasks, AI engines, evaluation methods and evaluation results to achieve the interconnectedness of various modules such as test results, test tasks, test data sets and AI engines, and realizes the full automation of the engine testing process through automated creation of test tasks, automated execution of engine performance tests and automated generation of test reports, thereby reducing manual operations and significantly improving testing efficiency. At the same time, it eliminates human interference factors and ensures the objectivity and accuracy of the test results.

[0033] Figure 1This is one of the flow charts of the engine performance testing method provided by the present invention. The method provided in this embodiment can be applied to various AI engine testing scenarios, such as testing after version upgrades of AI engine capabilities such as transcription, translation, synthesis, and image-text conversion, testing after architecture changes, and rapid testing after the introduction of third-party capabilities.

[0034] The executor of the method provided in this embodiment can be a performance testing system, which can be an electronic device such as a voice recorder, a translator, a mobile phone, a computer, a server, a server cluster, or a specially designed intelligent engine testing device, or it can be an AI engine testing device set in the electronic device. The AI engine testing device can be implemented through software, hardware, or a combination of both.

[0035] like Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0036] Step 110: Create a test task according to the test requirement information. The task parameters of the test task include a test data set, an evaluation plan, and a list of engines to be tested.

[0037] The test requirement information herein is triggered and acquired when an engine test is required, and includes information indicating the test data requirements, test engine scope, and evaluation requirements for the test; it can be obtained by parsing the user input entered by the user on the front-end interface, or by parsing the information generated by other controllers or under certain conditions, such as a test instruction triggered by a timer. The so-called user input can be information input through a command line interface, a graphical interface, touch input, a drop-down selection input, voice input, etc., and this embodiment does not specifically limit this.

[0038] Optionally, after obtaining the test requirement information, a corresponding test task can be created based on the test requirement information. The task parameters of this test task include at least a test dataset, an evaluation plan, and a list of engines to be tested. Other information may also be included. For example, for a translation task, the task parameters also include the source language (or source language) and the target language (or target language). For other tasks, the task parameters also include the source language, etc. This embodiment does not specifically limit this.

[0039] Among them, each test data in the test data set is data used for engine testing, and the test data can be generated by batch uploading through user input, batch downloading from the website, or automatic generation of large language models; the data type of the test data can be text, picture, or audio, etc., which is determined specifically according to the test requirement information. The evaluation plan is a plan for evaluating the performance of the engine, in which corresponding evaluation indicators and evaluation logic are configured for different test requirement information. The list of engines to be tested contains multiple versions of engines that need to be tested that meet the test requirement information; preferably, in order to simplify the description, the list of engines to be tested here can include multiple engines with the same function (engine type) but different versions.

[0040] Large language models (also referred to as "big models") typically refer to deep learning models with billions to tens of billions of parameters, used to process and generate natural language text. These models are pre-trained on a large scale. By learning the structure, semantics, and grammatical rules of a language from massive amounts of text data, they excel in a variety of language understanding and generation tasks. They not only understand the surface meaning of input text but also infer its underlying semantics and logic. Their powerful language understanding and context modeling capabilities provide a strong foundation for end-to-end natural language text processing.

[0041] It should be noted that in the task creation process, it can be specifically based on the association relationship between the test requirement information and the test task, directly mapping to obtain the associated test task; it can also be based on the association relationship between the test requirement information and the test data, solution and engine, mapping to obtain the associated test data, associated solution and associated engine, and using the test data set integrated with the associated test data, the associated solution, the list of engines to be tested integrated with the associated engine as task parameters to construct and obtain the associated test task, etc. This embodiment does not make specific restrictions on this.

[0042] The task type of the test task here may be a translation task, a speech synthesis task, a graphic and text recognition task, or a listening and speaking task, etc., which is not specifically limited in this embodiment.

[0043] Step 120 : performing a performance test on each engine to be tested in the list of engines to be tested according to each test data in the test data set, and obtaining a test output result of each engine to be tested.

[0044] Optionally, after obtaining the test task, the test task can be executed as a whole, that is, each engine to be tested in the list of engines to be tested is called, and all test data in the test data set are batch processed to obtain the test output results of each engine to be tested; or the test task can be divided into multiple subtasks, each subtask corresponding to a subset of the test data set, and then, each engine to be tested in the list of engines to be tested is called, and the test data in each subdata set are batch processed in parallel to obtain the test output results of each engine to be tested, etc. This embodiment does not make specific limitations on this.

[0045] Step 130: Output a test report corresponding to the test task according to the task type of the test task, the test output result, and the evaluation plan.

[0046] Optionally, after obtaining the test output results of each engine to be tested, the corresponding evaluation tool (such as a large language model or the engine with the best performance corresponding to the task type) can be determined according to the task type of the test task. Then, using the evaluation tool, the test output results of each engine to be tested are performance evaluated according to the evaluation plan, and the corresponding test report is automatically generated based on the performance evaluation results through the large language model.

[0047] The test report includes overview evaluation results and / or detailed evaluation results; the overview evaluation results include the total performance evaluation results of each engine under test in each sub-dataset (such as averaging the performance evaluation values of each engine under all test data in the sub-dataset to obtain the total performance evaluation results of each engine under the sub-dataset), as well as comparative analysis results of the total performance evaluation results of each engine under test in different sub-datasets; the detailed evaluation results include the detailed performance evaluation results and detailed performance analysis results of each engine under test in each test data in each sub-dataset. Based on this test report, users can view the corresponding evaluation results of each engine under test by clicking on a sub-dataset or a test data, or they can export all evaluation results in the test report at once by clicking the export button. This improves the reliability of the evaluation while allowing users to more intuitively view the performance of different engines under test in the list of engines under test, as well as to view the engines under test with the best performance under each sub-dataset.

[0048] The method provided in this embodiment automatically creates a test task containing task parameters based on the test requirement information. These parameters include a test data set, an evaluation plan, and a list of engines to be tested. Then, the test data set is used to automatically perform performance testing on each engine to be tested in the list of engines to be tested, so as to obtain the test output results of each engine, and a corresponding test report is intelligently generated based on the test task type, output results, and evaluation plan. This achieves the full automation of the engine performance test process, significantly improves the test efficiency, reduces manual intervention, and thus reduces errors caused by human factors, ensuring the accuracy and objectivity of the test results. At the same time, the automatically generated test report enhances the traceability of the results, facilitates subsequent analysis and problem reproduction, and effectively solves the problems of low efficiency and inaccurate results of traditional manual testing.

[0049] Based on the above embodiments, Figure 2 This is the second flow chart of the engine performance testing method provided by the present invention. Figure 2 As shown, the method further includes: step 210, step 220 and step 230.

[0050] Step 210: Create a test task according to the test requirement information. The task parameters of the test task include a test data set, an evaluation plan, and a list of engines to be tested.

[0051] The test requirement information here is triggered and obtained when the engine test is required, including information used to indicate the test data requirements, test engine scope, evaluation requirements, etc.; it can be obtained by parsing the user input entered by the user on the front-end interface, or it can be obtained by parsing other controllers or information generated under certain conditions, such as test instructions triggered by time.

[0052] Optionally, after obtaining the test requirement information, a corresponding test task can be created based on the test requirement information. The task parameters of this test task include at least a test dataset, an evaluation plan, and a list of engines to be tested. Other information may also be included. For example, for a translation task, the task parameters also include the source language (or source language) and the target language (or target language). For other tasks, the task parameters also include the source language, etc. This embodiment does not specifically limit this.

[0053] Among them, each test data in the test data set is data used for engine testing, and the test data can be generated by batch uploading through user input, batch downloading from the website, or automatic generation of large language models; the data type of the test data can be text, picture, or audio, etc., which is determined specifically according to the test requirement information. The evaluation plan is a plan for evaluating the performance of the engine, in which corresponding evaluation indicators and evaluation logic are configured for different test requirement information. The list of engines to be tested contains multiple versions of engines that need to be tested that meet the test requirement information; preferably, in order to simplify the description, the list of engines to be tested here can include multiple engines with the same function (engine type) but different versions.

[0054] It should be noted that in the task creation process, it can be specifically based on the association relationship between the test requirement information and the test task, directly mapping to obtain the associated test task; it can also be based on the association relationship between the test requirement information and the test data, solution and engine, mapping to obtain the associated test data, associated solution and associated engine, and using the test data set integrated with the associated test data, the associated solution, the list of engines to be tested integrated with the associated engine as task parameters to construct and obtain the associated test task, etc. This embodiment does not make specific restrictions on this.

[0055] Step 220 : performing a performance test on each engine to be tested in the list of engines to be tested according to each test data in the test data set, and obtaining a test output result of each engine to be tested.

[0056] Optionally, after obtaining the test task, the test task can be executed as a whole, that is, each engine to be tested in the list of engines to be tested is called, and all test data in the test data set are batch processed to obtain the test output results of each engine to be tested; or the test task can be divided into multiple subtasks, each subtask corresponding to a subset of the test data set, and then, each engine to be tested in the list of engines to be tested is called, and the test data in each subdata set are batch processed in parallel to obtain the test output results of each engine to be tested, etc. This embodiment does not make specific limitations on this.

[0057] Step 230, when the task type is a translation task, obtain an evaluation mode according to the test requirement information, the evaluation mode including a mode for independently evaluating each engine and / or a mode for comparative evaluation of multiple engines; generate a first prompt message according to the evaluation mode and the evaluation plan; input the first prompt message, each of the test data, and the test output result into a large language model to obtain the test report output by the large language model.

[0058] Optionally, if the task type is a translation task, the test data in the test dataset is the source language text to be translated, and the evaluation scheme is a scheme for evaluating translation performance, including but not limited to evaluation methods such as BLEU, chrF++, and BERTScore that judge the engine's effectiveness based on the similarity between the engine's test output (i.e., the translated text) and the target language text, and / or translation quality evaluation methods configured with evaluation logic based on different scoring scales, etc. This embodiment does not specifically limit this. The engines to be tested included in the list of engines to be tested are translation engines of different versions.

[0059] Accordingly, the steps for generating a test report include: An evaluation mode is obtained based on the test requirement information. If the test requirement information includes requirement information indicating a single-engine version evaluation, the corresponding evaluation mode is a mode for independently evaluating each engine; if the test requirement information includes requirement information indicating a multi-engine version evaluation, the corresponding evaluation mode is a mode for comparative evaluation of multiple engines; if the test requirement information includes requirement information indicating both a single-engine version evaluation and a multi-engine version evaluation, the corresponding evaluation mode includes both a mode for independently evaluating each engine and a mode for comparative evaluation of multiple engines.

[0060] After obtaining the corresponding evaluation mode, prompt information can be generated according to the evaluation mode and evaluation plan to obtain first prompt information. The first prompt information is used to prompt the large language model to perform performance evaluation of the engine to be tested according to the corresponding evaluation mode and evaluation plan.

[0061] It should be noted that, in the process of generating prompt information, the evaluation mode and evaluation plan can be updated to the engine test prompt template to obtain the first prompt information, or the evaluation mode and evaluation plan can be input into the big model, and the big model can dynamically generate the first prompt information based on the text generation capability, etc. This embodiment does not make specific limitations on this.

[0062] After obtaining the first prompt information, the first prompt information, each test data (that is, the source language text in the case of a translation task), and the test output results of each engine to be tested (that is, the target language text in the case of a translation task) can be used as input information and input into the large language model, so that the large language model can evaluate the performance of each engine to be tested based on the corresponding input information to automatically generate a corresponding test report. Specifically, the test report can provide an overview of the score or rating (poor, medium, good, excellent, or perfect, etc.) of the total performance evaluation results of each engine to be tested under each sub-data set, and / or a distributed display of the detailed performance evaluation results and detailed performance analysis results of each engine to be tested under each test data in each sub-data set, etc. This embodiment does not specifically limit this.

[0063] For example, in the evaluation mode of comparing and evaluating multiple engines, the code content of an exemplary input information is as follows: { content = ("Evaluate the translation quality of the %s source text '%s' into multiple %s translated texts '%s' and assign scores to each translation. Use a continuous rating scale from 0 to 5, where 0 means "no meaning is retained" and 5 means "semantically and grammatically perfect"); Specific scoring requirements are as follows:\n 0-1 points: The original message is not conveyed, the translation is not complete and does not make sense;\n 1.1~2 points: Only a small part of the information is conveyed, the translation is confusing, and the overall translation needs to be revised by more than 80%;\n 2.1~3 points: The main content is not fully conveyed, the translation contains serious translation errors, and the meaning of the sentences is difficult to understand without reading the original text. The translation as a whole needs to be revised by more than 50%;\n 3.1~4 points: The main content is basically conveyed, but there are some translation errors, and the overall translation needs to be revised by more than 10%;\n 4.1-4.5 points: The original message is accurately conveyed, the translation has 1-2 minor deficiencies, and the language expression is smooth and fluent (subjective feeling: OK); 4.6-5 points: The original message is accurately conveyed, the translation is error-free, and the language expression is very natural and authentic, reaching the level of "elegance" (subjective feeling: amazing); Compare multiple translations and try not to give them the same score;\n To give only specific scores, simply place the score digits in order within a []; the scores given should be specific values, not fraction segments. ")% (src_lang, src_text, tar_lang, str(trans_text_list))}.

[0064] Among them, src_lang is the source language, src_text is the original text (that is, the source language text), tar_lang is the target language, and trans_text_list is a list of multiple translated texts (that is, the target language text).

[0065] The method provided in this embodiment, when the test task is a translation task, can automatically evaluate the translation performance of the engine and generate a standardized test report by combining prompt engineering and a large model, reducing the need for labeled data. It can also perform independent evaluation of a single engine or comparative evaluation of multiple engines based on the evaluation mode. Compared with the existing technology that requires reliance on manually labeled data for translation performance testing, it effectively realizes a flexible, efficient, accurate, comprehensive, and objective evaluation of AI engine performance, significantly improves test efficiency and the reliability of results, and reduces errors and subjectivity caused by manual intervention.

[0066] Based on the above embodiments, Figure 3 This is the third flow chart of the engine performance testing method provided by the present invention. Figure 3 As shown, the method further includes: step 310, step 320 and step 330.

[0067] Step 310: Create a test task according to the test requirement information. The task parameters of the test task include a test data set, an evaluation plan, and a list of engines to be tested.

[0068] The test requirement information here is triggered and obtained when the engine test is required, including information used to indicate the test data requirements, test engine scope, evaluation requirements, etc.; it can be obtained by parsing the user input entered by the user on the front-end interface, or it can be obtained by parsing other controllers or information generated under certain conditions, such as test instructions triggered by time.

[0069] Optionally, after obtaining the test requirement information, a corresponding test task can be created based on the test requirement information. The task parameters of this test task include at least a test dataset, an evaluation plan, and a list of engines to be tested. Other information may also be included. For example, for a translation task, the task parameters also include the source language (or source language) and the target language (or target language). For other tasks, the task parameters also include the source language, etc. This embodiment does not specifically limit this.

[0070] Among them, each test data in the test data set is data used for engine testing, and the test data can be generated by batch uploading through user input, batch downloading from the website, or automatic generation of large language models; the data type of the test data can be text, picture, or audio, etc., which is determined specifically according to the test requirement information. The evaluation plan is a plan for evaluating the performance of the engine, in which corresponding evaluation indicators and evaluation logic are configured for different test requirement information. The list of engines to be tested contains multiple versions of engines that need to be tested that meet the test requirement information; preferably, in order to simplify the description, the list of engines to be tested here can include multiple engines with the same function (engine type) but different versions.

[0071] It should be noted that in the task creation process, it can be specifically based on the association relationship between the test requirement information and the test task, directly mapping to obtain the associated test task; it can also be based on the association relationship between the test requirement information and the test data, solution and engine, mapping to obtain the associated test data, associated solution and associated engine, and using the test data set integrated with the associated test data, the associated solution, the list of engines to be tested integrated with the associated engine as task parameters to construct and obtain the associated test task, etc. This embodiment does not make specific restrictions on this.

[0072] Step 320 : Performing a performance test on each engine to be tested in the list of engines to be tested according to each test data in the test data set, and obtaining a test output result of each engine to be tested.

[0073] Optionally, after obtaining the test task, the test task can be executed as a whole, that is, each engine to be tested in the list of engines to be tested is called, and all test data in the test data set are batch processed to obtain the test output results of each engine to be tested; or the test task can be divided into multiple subtasks, each subtask corresponding to a subset of the test data set, and then, each engine to be tested in the list of engines to be tested is called, and the test data in each subdata set are batch processed in parallel to obtain the test output results of each engine to be tested, etc. This embodiment does not make specific limitations on this.

[0074] Step 330, when the task type is a speech synthesis task, an image-text recognition task, or a listening and speaking task, determine the target engine with the best performance corresponding to the task type in the engine storage space; output the test report based on the evaluation plan, the test output results of each of the engines to be tested, and the test output results of the target engine.

[0075] Optionally, when the task type is a speech synthesis task, an image and text recognition task, or a listening and speaking task, the target engine with the best performance corresponding to the task type can be loaded into the engine storage space to perform engine performance analysis based on the evaluation plan, the test output results of each engine to be tested, and the test output results of the target engine, thereby automatically outputting the corresponding test report.

[0076] For example, when the task type is a speech synthesis task, the engines to be tested in the list of engines to be tested are speech synthesis engines, the test data is the original text, and the test output results of the engines to be tested are the synthesized speech corresponding to the original text. At this time, the audio-visual synthesis engine with the best performance can be loaded into the engine storage space as the target engine. Then, the synthesized speech output by each engine to be tested in the list of engines to be tested for speech synthesis of the original text is input into the audio-visual synthesis engine with the best performance to obtain the test output result of the audio-visual synthesis engine with the best performance, that is, the converted text. Then, using the evaluation calculation logic indicated by the evaluation scheme (such as BLEU, etc.), the similarity between each converted text output by the audio-visual synthesis engine with the best performance and the original text is calculated, so that according to the similarity, the score or rating (poor, medium, good, excellent or perfect, etc.) of each engine to be tested in the overall performance evaluation results under each sub-data set is summarized in the test report, and / or the detailed performance evaluation results and detailed performance analysis results of each engine to be tested under each test data in each sub-data set are distributed and displayed, etc. This embodiment does not specifically limit this.

[0077] When the task type is a dictation task, the engines to be tested in the list of engines to be tested are dictation engines, the test data are voice data, and the test output results of the engines to be tested are text data corresponding to the voice data. At this time, the dictation engine with the best performance can be loaded into the engine storage space as the target engine. Then, the evaluation calculation logic indicated by the evaluation plan is used to calculate the similarity between the text data output by the listening and speaking processing of the voice data by each engine to be tested in the list of engines to be tested and the text data output by the listening and speaking processing of the voice data by the dictation engine with the best performance. Based on the similarity, the test report provides an overview of the scores or ratings (poor, medium, good, excellent or perfect, etc.) in the total performance evaluation results of each engine to be tested under each sub-data set, and / or the detailed performance evaluation results and detailed performance analysis results of each engine to be tested under each test data in each sub-data set are distributed and displayed. This embodiment does not specifically limit this.

[0078] Similarly, when the task type is an image and text recognition task, the performance evaluation of each engine to be tested can be carried out according to the evaluation process corresponding to the dictation task, and a corresponding test report can be output. This will not be repeated here.

[0079] The method provided in this embodiment, by introducing the optimal performance engine for automatic comparative evaluation, not only avoids text labeling and effectively saves labeling costs and labor costs, but also realizes efficient and accurate evaluation of engine performance of various task types (such as speech synthesis, image and text recognition, etc.), significantly improves test efficiency and reliability of results, and enhances the traceability and reproducibility of test results.

[0080] Based on the above embodiments, Figure 4 This is a fourth flow chart of the engine performance testing method provided by the present invention. Figure 4 As shown, the method further includes: step 410, step 420 and step 430.

[0081] Step 410: Create a test task according to the test requirement information. The task parameters of the test task include a test data set, an evaluation plan, and a list of engines to be tested.

[0082] The test requirement information here is triggered and obtained when the engine test is required, including information used to indicate the test data requirements, test engine scope, evaluation requirements, etc.; it can be obtained by parsing the user input entered by the user on the front-end interface, or it can be obtained by parsing other controllers or information generated under certain conditions, such as test instructions triggered by time.

[0083] Optionally, after obtaining the test requirement information, a corresponding test task can be created based on the test requirement information. The task parameters of this test task include at least a test dataset, an evaluation plan, and a list of engines to be tested. Other information may also be included. For example, for a translation task, the task parameters also include the source language (or source language) and the target language (or target language). For other tasks, the task parameters also include the source language, etc. This embodiment does not specifically limit this.

[0084] Among them, each test data in the test data set is data used for engine testing, and the test data can be generated by batch uploading through user input, batch downloading from the website, or automatic generation of large language models; the data type of the test data can be text, picture, or audio, etc., which is determined specifically according to the test requirement information. The evaluation plan is a plan for evaluating the performance of the engine, in which corresponding evaluation indicators and evaluation logic are configured for different test requirement information. The list of engines to be tested contains multiple versions of engines that need to be tested that meet the test requirement information; preferably, in order to simplify the description, the list of engines to be tested here can include multiple engines with the same function (engine type) but different versions.

[0085] It should be noted that in the task creation process, it can be specifically based on the association relationship between the test requirement information and the test task, directly mapping to obtain the associated test task; it can also be based on the association relationship between the test requirement information and the test data, solution and engine, mapping to obtain the associated test data, associated solution and associated engine, and using the test data set integrated with the associated test data, the associated solution, the list of engines to be tested integrated with the associated engine as task parameters to construct and obtain the associated test task, etc. This embodiment does not make specific restrictions on this.

[0086] Step 420: divide the test data set into multiple sub-data sets according to the data attribute information of each test data in the test data set; create each sub-task under the test task according to the data set division result; the task parameters of each sub-task include one sub-data set, the evaluation plan and the list of engines to be tested; determine the test process of each sub-task according to the task status corresponding to each sub-task; based on the thread corresponding to each sub-task, call each engine to be tested to execute the test operation corresponding to each sub-task in parallel to obtain the test output result of each engine to be tested; wherein, the test operation corresponding to each sub-task includes, according to the test process of each sub-task, executing the operation of each test data in the sub-data set corresponding to each sub-task.

[0087] Here, each test data is configured with corresponding attribute information based on the attribute configuration mode corresponding to its own data type; for example, for text-type test data used to evaluate translation engines, speech synthesis engines, or image synthesis task engines, its attribute information may include basic information, such as language information, industry type information (such as computer industry, medical industry, etc.), data source information (such as website download, AI generation, etc.), and file type. For image-type test data used to evaluate text recognition engines, in addition to basic information, that is, at least one of language information, industry type information, data source information, file type, etc., its attribute information also includes image attribute information, such as resolution, text layout information, etc.; For audio-type test data used to evaluate dictation engines, in addition to basic information, that is, at least one of language information, industry type information, data source information, and file type, its attribute information also includes audio attribute information, such as age group information, gender information (such as male or female voice), sampling rate, accent, speaking speed, and signal-to-noise ratio. Preferably, the attribute information of each test information includes at least language information and file type information, and other information can be configured according to actual needs.

[0088] Optionally, after obtaining the test task, in order to improve the test efficiency, the test data set can be divided into multiple sub-data sets according to the data attribute information of each test data in the test data set; for example, according to the language information, multiple test data belonging to the same language attribute in the test data set can be divided into a sub-data set to obtain multiple sub-data sets, or, according to the industry type information, multiple test data belonging to the same industry type in the test data set can be divided into a sub-data set to obtain multiple sub-data sets, etc. This embodiment does not make specific limitations on this.

[0089] Then, according to the data set division results, the test task is split into multiple subtasks; each subtask is bound to a sub-data set, an evaluation plan, and a list of engines to be tested.

[0090] In addition, a thread pool is created in the performance testing system, and each thread in the thread pool runs a subtask. The running steps of each subtask include: task status monitoring, test process determination, sub-dataset download, engine call and engine testing.

[0091] When monitoring the task status, the task status of each subtask can be monitored in real time based on the thread corresponding to each subtask. The status determines the operation authority and process of the subtask at different stages; the task status includes but is not limited to the invalid status (that is, no operation can be performed), the pending review status (selectable reviewer), the review status (waiting for review after selecting a reviewer), the review failed status (re-reviewed after editing), the pending execution status (executable), the execution status (waiting for execution completion), the execution success status (the result can be viewed), the execution failure status (execution failed, can be re-executed), the user termination status (the task is stopped, can be re-executed), the evaluation status, the evaluation failure status (can be re-evaluated or re-executed), the modified status (there is a version update of the sub-dataset), etc. This embodiment does not make specific restrictions on this.

[0092] After monitoring the task status of each subtask, the test process of each subtask can be determined based on the thread corresponding to each subtask and the task status; the test processes corresponding to different task statuses are different. For example, if the task status of any subtask is in the pending review state, the test process of the subtask is: first notify the user to select a reviewer, and after obtaining the reviewer, update the task status to the review state, and notify the reviewer to review the task parameters. After the review is passed, the task status is updated to the pending execution state. When calling each engine to be tested to execute the test operation corresponding to the subtask, the task status is updated to the execution state; after the test operation is executed, if a manual termination instruction is received, the task status is updated to the user termination state. In this state, the subtask can be edited or re-executed; after the test operation is completed, the task status is updated to the execution success state or the execution failure state; when the task status is the execution success state, the engine performance evaluation can be performed, and the task status is updated to the evaluation state. After the evaluation is completed, the task status is updated to the evaluation success state or the evaluation failure state. When the task status is determined to be a successful evaluation status, the evaluation results of the engine to be tested can be displayed in the form of a test report; when the task status is determined to be a failed evaluation status, the task status is updated to an evaluation status or a pending execution status to re-evaluate or re-execute the test operation.

[0093] After obtaining the test process of each subtask, all the test data in the sub-data set corresponding to each subtask can be downloaded to the test data storage space in the working interval corresponding to each subtask based on the thread corresponding to each subtask. If the sub-data set carries the annotated text of the test data, an annotated data storage space is created in the working interval corresponding to each subtask, and the annotated text of the test data is stored in the annotated data storage space, so that the corresponding annotated text can be easily accessed when calling each engine to be tested to execute separate test data.

[0094] Then, based on the thread corresponding to each subtask, each engine to be tested in the list of engines to be tested is called to execute the test operations corresponding to each subtask in parallel, that is, each engine to be tested is called to execute the test data in the sub-data set corresponding to each subtask according to the test process of each subtask to obtain the test output results of each engine to be tested.

[0095] In one possible implementation, the calling step of each engine to be tested includes: locating the engine folder of each engine to be tested according to the engine type of each engine to be tested; based on the engine folder, determining the calling scheme of each engine to be tested according to the reflection mechanism; and calling each engine to be tested according to the calling scheme.

[0096] Optionally, when calling each engine to be tested, the engine folder of the engine to be tested can be located in the database first through the engine type of the engine to be tested, and then, based on the located engine folder, the calling scheme of the engine to be tested is obtained through the reflection mechanism, so as to call the engine to be tested according to the calling scheme. Thus, through this engine calling method, the input and output of all engines of the same engine type are standardized, such as the audio synthesis engine receives text and outputs audio files, the text recognition engine receives image files and outputs text, etc.; and when adding a new type of engine, it is only necessary to add engine information and a new calling method to the database, without modifying the old files, which greatly reduces the program coupling.

[0097] Step 430: Output a test report corresponding to the test task according to the task type of the test task, the test output result, and the evaluation plan.

[0098] Optionally, after obtaining the test output results of each engine to be tested, the corresponding evaluation tool (such as a large language model or the engine with the best performance corresponding to the task type) can be determined according to the task type of the test task. Then, using the evaluation tool, the test output results of each engine to be tested are performance evaluated according to the evaluation plan, and the corresponding test report is automatically generated based on the performance evaluation results through the large language model.

[0099] The test report includes overview and / or detailed evaluation results. The overview includes the overall performance evaluation results of each engine under test under the sub-dataset corresponding to each subtask (e.g., averaging the performance evaluation values of each engine under all test data in the sub-dataset to obtain the overall performance evaluation results of each engine under the sub-dataset), as well as comparative analysis of the overall performance evaluation results of each engine under test under different subtasks. The detailed evaluation includes the detailed performance evaluation and performance analysis results of each engine under test under each test data in the sub-dataset corresponding to each subtask. This allows users to view the evaluation results of each engine under test by clicking on a sub-dataset or test data, or to export all evaluation results in the test report at once using the Export button.

[0100] For example, when the test report includes an overview of the evaluation results, the data format is as follows: Test Report #### Tester -yuxia9 #### Evaluation Methodology - Large model evaluation #### Task start time and end time - Start time : 2024-11-20 11:10:29 - End Time : 2024-11-20 11:10:32 #### Evaluation scores of each engine in different languages - A translation engine - Chinese-Cantonese->Vietnamese: 3.65 - Chinese-Cantonese->Indonesian: 3.35 - Chinese-Cantonese->Chinese-Mandarin: 4.67 - Chinese-Cantonese->English-US: 2.88 - B Translation Engine - Chinese-Cantonese->Vietnamese: 3.62 - Chinese-Cantonese->Indonesian: 3.55 - Chinese-Cantonese->Chinese-Mandarin: 4.67 - Chinese-Cantonese->English-US: 3.45 #### Analysis Results Since two engines (A and B) were tested for each language, we can compare the scores between different engines. The following is an analysis of the best and worst scoring engines for each language: - Chinese-Cantonese->Vietnamese - Best: A translation engine (3.65) - Worst: B Translation Engine (3.62) - Chinese-Cantonese->Indonesian - Best: A translation engine (3.35) - Worst: B Translation Engine (3.55) - Chinese-Cantonese -> Chinese-Mandarin - Best: Translation Engine A and Translation Engine B tied (4.67) - Worst: Translation Engine A and Translation Engine B tied (4.67) - Chinese-Cantonese -> English-American - Best: B Translation Engine (3.45) - Worst: A translation engine (2.88) #### Summarize In this evaluation, the performance of the two engines in different languages varies. Specifically: - For translation from Chinese to Cantonese to Vietnamese, translation engine A is slightly better than translation engine B - For translation from Chinese to Cantonese to Indonesian, translation engine A is slightly better than translation engine B - For Chinese-Cantonese to Chinese-Mandarin translation, both engines perform the same - For Chinese-Cantonese to English-Mei translation, the B translation engine is significantly better than the A translation engine}.

[0101] For example, when the test report includes detailed evaluation results, the data format is as follows: {### Evaluation Effect Analysis Report Test data ID 16398 Original text: Arrogant and self-assured Standard text: null Translation results: arrogance Review score: 2.1 analyze: - Original text: "Ao Ni Zi Ruo" is an idiom meaning an arrogant and natural attitude. - Translation results: "arrogant" in English means "arrogant", which only conveys part of the meaning and does not fully express the meaning of "confident". Problem explanation: The translation only conveys part of the meaning and does not fully express the original text. A rating of 2.1 is reasonable because the translation is not completely accurate. {#### Test data ID 16400 Original text: Men's Group A 100m sprint, first call Standard text: null Translation results: Men's first group 100m dash, first call Review score: 4.2 analyze: - Original text: Describes the first call for the men's group A 100-meter sprint - Translation results: The original meaning is basically conveyed, but the language expression is not fluent enough Problem explanation: The translation basically conveys the meaning of the original text, but the language expression is not smooth enough. The score of 4.2 may be too high because the accuracy and fluency of the translation still need to be improved. ### Summarize Overall, these translations have varying degrees of accuracy and fluency issues, and the ratings generally reflect these issues. However, some ratings may be slightly off, depending on the rating criteria and expectations of translation quality. For example, ID16400's rating of 4.2 may be too high because the accuracy and fluency of the translation still need to be improved. Overall Rating Taking into account the translation quality and rating of each record, the overall evaluation rationality score is 7 points (out of 10 points). This score reflects the accuracy and fluency of most translation results, but also takes into account the possible deviation of some ratings.

[0102] The method provided in this embodiment dynamically creates tasks through test requirement information, supports flexible combinations of multiple data types and engine types to adapt to different testing requirements, and significantly improves test efficiency and accuracy through data set division and parallel execution of multiple subtasks, reduces program coupling, and enhances the controllability and maintainability of the system through task status management and process automation.

[0103] Based on the above embodiments, Figure 5 This is a fifth flow chart of the engine performance testing method provided by the present invention, such as Figure 5As shown, the method further includes: step 510, step 520 and step 530.

[0104] Step 510, parse the test requirement information to obtain data item requirement information, evaluation requirement information and engine requirement information; in the data storage space, obtain multiple test data whose data attribute information matches the data item requirement information, and construct the test data set; according to the evaluation requirement information, obtain the evaluation plan; in the engine storage space, obtain multiple engines whose engine type and version information match the engine requirement information, and construct a list of engines to be tested; create the test task with the test data set, the evaluation plan and the list of engines to be tested as task parameters.

[0105] Optionally, when creating a task, if no associated test task is found based on the test requirement information, that is, if the existing test task cannot be matched with the test requirement information, the test requirement information can be parsed to obtain data item requirement information, evaluation requirement information, and engine requirement information. The data item requirement information at least includes the attribute requirement information of the data involved in the test.

[0106] Figure 6 This is the sixth flow chart of the engine performance test method provided by the present invention. Figure 6 As shown, after obtaining the data item requirement information, multiple test data whose data attribute information matches the data item requirement information can be searched in the test data stored in the data storage space (such as cloud storage space, etc.) to construct a test data set.

[0107] Among them, each test data in the data storage space can be generated by batch uploading by user input, batch downloading from the website, or automatic generation of a large language model.

[0108] For example, in one possible implementation, the step of storing any test data in the data storage space includes: upon receiving a write request for the test data, parsing the write request to obtain the attribute information and data content of the test data, and writing the test data into the data storage space based on the attribute information and data content of the test data. It should be noted that when writing the test data, the test data may be written to the data storage space in a batch along with other test data with the same attribute information, or the test data may be compressed with other test data with the same attribute information and written into the data storage space as a compressed package, etc. This embodiment does not specifically limit this.

[0109] In addition, in another possible implementation method, the step of storing any test data in the data storage space also includes: generating second prompt information based on the data attribute information of any test data; inputting the second prompt information and the content description information of any test data into the large language model to obtain the generated data of any test data output by the large language model; and writing any test data into the data storage space based on the generated data and the data attribute information of any test data.

[0110] Optionally, any test data in the data storage space can be automatically generated based on the macro model. Specifically, a second prompt message can be generated based on the data attribute information of the test data, prompting the macro model to generate data that matches the data attribute information. For example, the second prompt message could be "Please generate a piece of text data in Chinese related to the medical field. The content must include professional terminology and medical record format, such as patient symptom descriptions and diagnosis results."

[0111] During the prompt information generation process, the data attribute information of the test data can be updated to the data generation prompt template to obtain the second prompt information, or the data attribute information of the test data can be input into the large model, and the model can dynamically generate the second prompt information based on the text generation capability, etc. This embodiment does not make specific limitations on this.

[0112] Then, the second prompt information and the content description information of the test data are input into the large language model, so that the large language model generates data according to the input information to obtain the generated data of the test data. After obtaining the generated data, it is necessary to further determine whether the generated data needs to be further processed based on the data type of the test data to obtain the test data. For example, if the data type of the test data is a picture or text, the test data is written in the data storage space based on the data content of the generated data and the data attribute information of the test data; if the data type of the test data is audio data, it is necessary to use the audio synthesis engine with the best performance to perform audio synthesis on the generated data, and write the test data in the data storage space based on the data content of the synthesized data and the data attribute information of the test data. In addition, when the large language model starts to generate data, the performance testing system can also establish another thread to analyze the input information of the large language model to automatically update and supplement the test data to add a new form so that the user can view and verify the data attribute information and content description information of the updated test data in real time.

[0113] Therefore, by using large models to automatically generate test data that matches specific attributes, manual participation in collecting test data is reduced, efficient generation and management of test data is achieved, the flexibility of data generation is improved, and effective data support is provided for improving engine testing efficiency and test quality.

[0114] It should be noted that after the test data is written to the data storage space, it can also be viewed, downloaded and edited by users online. When editing the test data, if the test data has been added to the test data set of a task, the test data set will be automatically updated based on the editing information of the test data, and the task status of the task to which the test data set belongs will be updated to the modified status.

[0115] like Figure 6 As shown, after parsing the test requirement information to obtain the evaluation requirement information, a corresponding evaluation plan can be generated based on the evaluation requirement information to indicate how to quantitatively evaluate the performance of the engine under test. For example, the corresponding evaluation plan can be obtained directly based on the mapping relationship between the evaluation requirement information and the evaluation plan; or the evaluation requirement information can be input into the big model, which uses its text generation capabilities to generate the corresponding evaluation plan, etc. This embodiment does not specifically limit this.

[0116] Furthermore, after obtaining the engine requirement information from the test requirement information, the engine type and version information of each engine in the engine storage space can be matched with the engine requirement information to obtain multiple engines whose engine type and version information match the engine requirement information, and construct a list of engines to be tested.

[0117] After obtaining the test data set, evaluation scheme and list of engines to be tested that match the test requirement information, a corresponding test task can be created using the test data set, the evaluation scheme and the list of engines to be tested as task parameters.

[0118] Step 520 : Perform performance tests on each engine to be tested in the list of engines to be tested according to each test data in the test data set, and obtain test output results of each engine to be tested.

[0119] Optionally, after obtaining the test task, the test task can be executed as a whole, that is, each engine to be tested in the list of engines to be tested is called, and all test data in the test data set are batch processed to obtain the test output results of each engine to be tested; or the test task can be divided into multiple subtasks, each subtask corresponding to a subset of the test data set, and then, each engine to be tested in the list of engines to be tested is called, and the test data in each subdata set are batch processed in parallel to obtain the test output results of each engine to be tested, etc. This embodiment does not make specific limitations on this.

[0120] Step 530: Output a test report corresponding to the test task according to the task type of the test task, the test output result, and the evaluation plan.

[0121] Optionally, if it is determined that the test task execution fails, return to step 520 to continue task execution; if it is determined that the test task execution is successful, that is, after obtaining the test output results of each engine to be tested, the corresponding evaluation tool (such as a large language model or the engine with the best performance corresponding to the task type) can be determined according to the task type of the test task. Thus, after using the evaluation tool to perform performance evaluation on the test output results of each engine to be tested according to the evaluation plan, the corresponding test report is automatically generated according to the performance evaluation results through the large language model, so that the user can view the performance evaluation results of each engine to be tested in real time.

[0122] The test report includes overview evaluation results and / or detailed evaluation results; the overview evaluation results include the total performance evaluation results of each engine under test in each sub-dataset (such as averaging the performance evaluation values of each engine under all test data in the sub-dataset to obtain the total performance evaluation results of each engine under the sub-dataset), as well as comparative analysis results of the total performance evaluation results of each engine under test in different sub-datasets; the detailed evaluation results include the detailed performance evaluation results and detailed performance analysis results of each engine under test data in each sub-dataset. Based on this test report, users can view the evaluation results corresponding to each engine under test by clicking on a sub-dataset or a test data, or they can export all evaluation results in the test report at once by clicking the Export button. This improves the reliability of the evaluation while allowing users to more intuitively obtain the performance of different engines under test in the list of engines under test.

[0123] The method provided in this embodiment significantly improves the efficiency and quality of engine testing by automatically parsing test requirements, generating test data matching specific attributes, flexibly creating and executing test tasks, and intelligently generating test reports, while enhancing the reliability of evaluation results and the convenience for users to obtain performance information.

[0124] The engine performance testing system provided by the present invention is described below. The engine performance testing system described below and the engine performance testing method described above can be referenced to each other.

[0125] Figure 7 FIG. 1 is a schematic diagram of the structure of the engine performance test system provided by the present invention; FIG. Figure 7 As shown, the system includes: The task creation unit 710 is used to create a test task according to the test requirement information. The task parameters of the test task include a test data set, an evaluation plan, and a list of engines to be tested. The first testing unit 720 is configured to perform a performance test on each engine to be tested in the list of engines to be tested according to each test data in the test data set, and obtain a test output result of each engine to be tested; The second testing unit 730 is used to output a test report corresponding to the test task according to the task type of the test task, the test output result and the evaluation plan.

[0126] The system provided in this embodiment automatically creates a test task containing task parameters based on the test requirement information. These parameters include a test data set, an evaluation plan, and a list of engines to be tested. Then, the test data set is used to automatically perform performance testing on each engine to be tested in the list of engines to obtain the test output results of each engine, and a corresponding test report is intelligently generated based on the test task type, output results, and evaluation plan. This achieves the full automation of the engine performance test process, significantly improves test efficiency, reduces manual intervention, and thus reduces errors caused by human factors, ensuring the accuracy and objectivity of the test results. At the same time, the automatically generated test report enhances the traceability of the results, facilitates subsequent analysis and problem reproduction, and effectively solves the problems of low efficiency and inaccurate results of traditional manual testing.

[0127] In some embodiments, the second testing unit is specifically used to: when the task type is a translation task, obtain an evaluation mode according to the test requirement information, wherein the evaluation mode includes a mode for independently evaluating each engine and / or a mode for comparative evaluation of multiple engines; generate a first prompt message according to the evaluation mode and the evaluation scheme; input the first prompt message, each of the test data, and the test output result into a large language model to obtain the test report output by the large language model. In some embodiments, the second test unit is further used to: when the task type is a speech synthesis task, an image and text recognition task, or a listening and speaking task, determine the target engine with the best performance corresponding to the task type in the engine storage space; and output the test report according to the evaluation plan, the test output results of each of the engines to be tested, and the test output results of the target engine.

[0128] In some embodiments, the first test unit is specifically used to: divide the test data set into multiple sub-data sets according to the data attribute information of each test data in the test data set; create each sub-task under the test task according to the data set division result; the task parameters of each sub-task include one of the sub-data sets, the evaluation plan and the list of engines to be tested; determine the test process of each sub-task according to the task status corresponding to each sub-task; based on the thread corresponding to each sub-task, call each engine to be tested to execute the test operations corresponding to each sub-task in parallel to obtain the test output results of each engine to be tested; wherein, the test operation corresponding to each sub-task includes, according to the test process of each sub-task, executing the operation of each test data in the sub-data set corresponding to each sub-task.

[0129] In some embodiments, the first test unit is further used to: locate the engine folder of each engine to be tested according to the engine type of each engine to be tested; based on the engine folder, determine the calling scheme of each engine to be tested according to the reflection mechanism; and call each engine to be tested according to the calling scheme.

[0130] In some embodiments, the task creation unit is specifically used to: parse the test requirement information to obtain data item requirement information, evaluation requirement information and engine requirement information; obtain multiple test data whose data attribute information matches the data item requirement information in the data storage space, and construct the test data set; obtain the evaluation plan based on the evaluation requirement information; obtain multiple engines whose engine type and version information match the engine requirement information in the engine storage space, and construct a list of engines to be tested; create the test task with the test data set, the evaluation plan and the list of engines to be tested as task parameters.

[0131] In some embodiments, the system also includes a data storage unit, which is specifically used to: generate second prompt information based on data attribute information of any test data; input the second prompt information and content description information of any test data into a large language model to obtain generated data of any test data output by the large language model; and write any test data into the data storage space based on the generated data and the data attribute information of any test data.

[0132] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0133] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call logic instructions in the memory 830 to execute an engine performance testing method, which includes: creating a test task based on test requirement information, wherein the task parameters of the test task include a test data set, an evaluation scheme, and a list of engines to be tested; performing a performance test on each engine to be tested in the list of engines to be tested based on each test data in the test data set, and obtaining a test output result for each engine to be tested; and outputting a test report corresponding to the test task based on the task type of the test task, the test output result, and the evaluation scheme.

[0134] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0135] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the engine performance testing method provided by the above methods, the method including: creating a test task according to test requirement information, the task parameters of the test task including a test data set, an evaluation scheme and a list of engines to be tested; performing a performance test on each engine to be tested in the list of engines to be tested according to each test data in the test data set, and obtaining the test output results of each engine to be tested; outputting a test report corresponding to the test task according to the task type of the test task, the test output results and the evaluation scheme.

[0136] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the performance testing method for the engine provided by the above-mentioned methods, the method comprising: creating a test task according to test requirement information, the task parameters of the test task including a test data set, an evaluation scheme and a list of engines to be tested; performing a performance test on each engine to be tested in the list of engines to be tested according to each test data in the test data set, and obtaining a test output result of each engine to be tested; outputting a test report corresponding to the test task according to the task type of the test task, the test output result and the evaluation scheme.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0138] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A performance testing method for an engine, characterized in that: include: Create a test task based on the test requirement information, where the task parameters of the test task include a test data set, an evaluation plan, and a list of engines to be tested; Performing a performance test on each engine to be tested in the list of engines to be tested according to each test data in the test data set, and obtaining a test output result of each engine to be tested; Output a test report corresponding to the test task according to the task type of the test task, the test output result and the evaluation plan.

2. The engine performance testing method according to claim 1, characterized in that: Outputting a test report corresponding to the test task according to the task type of the test task, the test output result, and the evaluation plan includes: When the task type is a translation task, obtaining an evaluation mode according to the test requirement information, wherein the evaluation mode includes a mode of independently evaluating each engine and / or a mode of comparatively evaluating multiple engines; Generate first prompt information according to the evaluation mode and the evaluation plan; The first prompt information, each of the test data, and the test output result are input into a large language model to obtain the test report output by the large language model.

3. The engine performance testing method according to claim 1, characterized in that: Outputting a test report corresponding to the test task according to the task type of the test task, the test output result, and the evaluation plan includes: When the task type is a speech synthesis task, an image-text recognition task, or a listening and speaking task, determining a target engine with the best performance corresponding to the task type in the engine storage space; The test report is output according to the evaluation plan, the test output results of each of the engines to be tested, and the test output results of the target engine.

4. The engine performance testing method according to any one of claims 1 to 3, characterized in that: The step of performing a performance test on each engine to be tested in the list of engines to be tested based on each test data in the test data set to obtain a test output result of each engine to be tested includes: Dividing the test data set into a plurality of sub-data sets according to data attribute information of each test data in the test data set; Creating subtasks under the test task based on the data set division result; task parameters of each subtask include a sub-data set, the evaluation plan, and the list of engines to be tested; Determine the test process of each subtask according to the task status corresponding to each subtask; Based on the threads corresponding to the subtasks, calling the engines to be tested to execute the test operations corresponding to the subtasks in parallel, and obtaining the test output results of the engines to be tested; The test operations corresponding to the subtasks include, according to the test procedures of the subtasks, executing operations on the test data in the sub-data sets corresponding to the subtasks.

5. The engine performance testing method according to claim 4, characterized in that: The steps of calling each of the engines to be tested include: Locating the engine folder of each engine to be tested according to the engine type of each engine to be tested; Based on the engine folder, determining the calling scheme of each engine to be tested according to the reflection mechanism; According to the calling scheme, each of the engines to be tested is called.

6. The engine performance testing method according to any one of claims 1 to 3, characterized in that: The step of creating a test task based on the test requirement information includes: Parsing the test requirement information to obtain data item requirement information, evaluation requirement information, and engine requirement information; In the data storage space, a plurality of test data whose data attribute information matches the data item requirement information is acquired to construct the test data set; Obtaining the evaluation plan according to the evaluation requirement information; In the engine storage space, multiple engines whose engine types and versions match the engine requirement information are obtained, and a list of engines to be tested is constructed; The test task is created with the test data set, the evaluation plan and the list of engines to be tested as task parameters.

7. The engine performance testing method according to claim 6, characterized in that: The step of storing any test data in the data storage space includes: generating second prompt information according to the data attribute information of any test data; Inputting the second prompt information and the content description information of the any test data into a large language model to obtain generated data of the any test data output by the large language model; According to the generated data and the data attribute information of the any test data, the any test data is written into the data storage space.

8. An engine performance testing system, characterized in that: include: A task creation unit, configured to create a test task according to the test requirement information, wherein the task parameters of the test task include a test data set, an evaluation plan, and a list of engines to be tested; A first testing unit is configured to perform a performance test on each engine to be tested in the list of engines to be tested according to each test data in the test data set, and obtain a test output result of each engine to be tested; The second testing unit is used to output a test report corresponding to the test task according to the task type of the test task, the test output result and the evaluation plan.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the engine performance testing method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the engine performance testing method according to any one of claims 1 to 7 is implemented.

Images