A method and device for testing consistency of a large language model inference

Abstract

This application provides a method and apparatus for testing the inference consistency rate of a large language model, relating to the field of artificial intelligence technology. The method includes: sending preset test cases to an inference server, causing the inference server to execute baseline inference and comparison inference respectively, obtaining baseline inference results and comparison inference results; standardizing the baseline inference results and comparison inference results; generating inference precision and inference hit rate based on the standardized comparison inference results and standardized baseline inference results; and calculating the inference consistency rate of the large language model based on the inference precision and inference hit rate. The method and apparatus for testing the inference consistency rate of a large language model provided in this application achieve quantifiable, tolerant, and comparable evaluation of inference consistency, significantly improving the accuracy, interpretability, and adaptability of inference consistency verification.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for testing the reasoning consistency rate of a large language model. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in text generation, intelligent question answering, and automated reasoning, the consistency and reproducibility of their reasoning results have gradually become important concerns in model development, system deployment, and engineering testing. In the reasoning process of LLMs, the final output result is typically generated based on the unnormalized score (logits) of the model output, combined with a sampling strategy. Common sampling methods include probabilistic sampling mechanisms that introduce a temperature parameter. The temperature parameter is used to adjust the randomness of the output result. When the temperature is greater than zero, the model smooths the probability distribution, allowing low-probability tokens to still have the possibility of being selected, thus introducing a certain degree of randomness. When the temperature equals zero, the sampling process degenerates into a greedy decoding method, meaning the model selects the token with the highest probability at each step, theoretically resulting in a completely consistent sequence of outputs under the same input conditions. Here, a token refers to the smallest unit of text data after word segmentation.

[0003] Under deterministic inference conditions with a zero temperature, performing multiple inference consistency tests on the model is not intended to assess output differences caused by randomness, but rather to detect the consistency and reproducibility of the model's internal computational logic across multiple inference processes, as well as the stability and reliability of external hardware and software systems (such as inference engines, drivers, and computing accelerator cards) under deterministic conditions. Furthermore, this type of test can also be used to discover non-logical deviations or potential self-contradictory output problems in the inference execution path.

[0004] Existing consistency detection methods typically employ direct string comparison of model outputs or convert the outputs into byte sequences and calculate hash values ​​(e.g., MD5) for verification. These methods can determine whether multiple inference results are completely consistent at the byte level, thus detecting inference anomalies. However, these methods only provide a binary "consistency" result and cannot reflect the degree of difference between different inference results. When inference results have slight differences but are still within acceptable limits, they may still be misjudged as anomalies, leading to unnecessary investigation and verification work and increasing R&D and testing costs.

[0005] Furthermore, in asynchronous execution, batch inference, or cross-device and cross-stage inference testing scenarios, existing methods based on real-time comparison or simple hash verification also have certain limitations in terms of applicability and flexibility, and the above problems urgently need to be solved.

[0006] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0007] To address the problems in the prior art, this application provides a method and apparatus for testing the consistency rate of reasoning in large language models. This method and apparatus can solve the problems that existing consistency detection methods can only qualitatively determine whether the reasoning results are consistent, cannot quantify the degree of difference, and are prone to misjudgment due to minor format differences, thus making it difficult to truly evaluate the stability and reproducibility of reasoning in large language models.

[0008] One aspect of the present invention provides a method for testing the reasoning consistency rate of a large language model, the method comprising: The preset test cases are sent to the inference server, so that the inference server performs the baseline inference and the inference to be compared respectively, and obtains the baseline inference result and the inference to be compared result. The baseline inference results and the inference results to be compared are standardized. The inference accuracy and inference hit rate are generated based on the standardized inference results to be compared and the standardized baseline inference results. The reasoning consistency rate of the large language model is calculated based on the reasoning accuracy and the reasoning hit rate.

[0009] Further, the inference server includes a first inference engine client, multiple second inference engine clients, an inference engine server, and a computing card; the step of sending preset test cases to the inference server, so that the inference server performs benchmark inference and comparison inference respectively, to obtain benchmark inference results and comparison inference results, includes: The test cases are sent to the first inference engine client, so that the first inference engine client calls the computing card through the inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result. The test cases are sent to the multiple second inference engine clients, so that the multiple second inference engine clients can call the computing card through the inference engine server to perform comparative inference using the target language model and obtain multiple comparative inference results.

[0010] Further, the inference server includes a third inference engine client, a fourth inference engine client, an inference engine server, and a computing card; the step of sending preset test cases to the inference server, so that the inference server performs baseline inference and comparison inference respectively, to obtain baseline inference results and comparison inference results, includes: The test cases are sent to the third inference engine client, so that the third inference engine client calls the computing card through the inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result. The test cases are repeatedly sent to the fourth inference engine client in multiple rounds, so that the fourth inference engine client calls the computing card through the inference engine server to perform multiple rounds of comparison inference using the target language model in sequence, and obtain multiple comparison inference results.

[0011] Further, the inference server includes a fifth inference engine client, a sixth inference engine client, a first inference engine server, a second inference engine server, a first computing card, and a second computing card; the step of sending preset test cases to the inference server, so that the inference server performs benchmark inference and comparison inference respectively, to obtain benchmark inference results and comparison inference results, includes: The test cases are sent to the fifth inference engine client, so that the fifth inference engine client calls the first computing card through the first inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result. The test cases are sent to the sixth inference engine client, so that the sixth inference engine client calls the second computing card through the second inference engine server to perform the comparison inference using the target language model and obtain the comparison inference result.

[0012] Further, the inference server includes a first inference server and a second inference server; the first inference server includes a seventh inference engine client, a third inference engine server, and a third computing card; the second inference server includes an eighth inference engine client, a fourth inference engine server, and a fourth computing card; the step of sending preset test cases to the inference server, so that the inference server performs benchmark inference and comparison inference respectively, to obtain benchmark inference results and comparison inference results, includes: The test cases are sent to the seventh inference engine client, so that the seventh inference engine client calls the third computing card through the third inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result. The test cases are sent to the eighth inference engine client, so that the eighth inference engine client calls the fourth computing card through the fourth inference engine server to perform the comparison inference using the target language model and obtain the comparison inference result.

[0013] Furthermore, the standardization process for the baseline inference result and the inference result to be compared includes: If the benchmark inference result and the inference result to be compared are Chinese inference results, then the benchmark inference result and the inference result to be compared are subjected to text segmentation. The baseline inference results and the inference results to be compared after text segmentation are subjected to case normalization. Sign removal is performed on the baseline inference results and the inference results to be compared after case normalization. The baseline inference result and the inference result to be compared are normalized by whitespace symbol normalization after symbol removal, so as to obtain the normalized inference result to be compared and the baseline inference result.

[0014] Furthermore, the standardization process for the baseline inference result and the inference result to be compared includes: If the baseline reasoning result and the reasoning result to be compared are in English, the English articles are removed from the baseline reasoning result and the reasoning result to be compared. The baseline inference results and the inference results to be compared after removing English articles are case-normalized. Sign removal is performed on the baseline inference results and the inference results to be compared after case normalization. The baseline inference result and the inference result to be compared are normalized by whitespace symbol normalization after symbol removal, so as to obtain the normalized inference result to be compared and the baseline inference result.

[0015] Furthermore, the generation of inference precision and inference hit rate based on the standardized inference results to be compared and the standardized baseline inference results includes: The inference accuracy is calculated based on the number of consistent lexical units between the standardized inference result to be compared and the standardized baseline inference result, and the total number of lexical units in the standardized inference result to be compared. The inference hit rate is calculated based on the number of consistent lexical units between the standardized inference result to be compared and the standardized baseline inference result, and the total number of lexical units in the standardized baseline inference result.

[0016] In another aspect, the present invention provides a testing device for the reasoning consistency rate of a large language model, the device comprising: The inference result generation unit is used to send preset test cases to the inference server, so that the inference server can perform benchmark inference and comparison inference respectively to obtain benchmark inference result and comparison inference result; The reasoning result processing unit is used to standardize the benchmark reasoning result and the reasoning result to be compared. The reasoning result analysis unit is used to generate reasoning accuracy and reasoning hit rate based on the standardized reasoning results to be compared and the standardized benchmark reasoning results. The reasoning consistency rate generation unit is used to calculate the reasoning consistency rate of the large language model based on the reasoning accuracy rate and the reasoning hit rate.

[0017] Furthermore, the inference server includes a first inference engine client, multiple second inference engine clients, an inference engine server, and a computing card; the inference result generation unit includes: The first benchmark inference result generation module is used to send the test cases to the first inference engine client, so that the first inference engine client can call the computing card through the inference engine server to perform benchmark inference using the target language model and obtain benchmark inference results. The first comparison inference result generation module is used to send the test cases to the plurality of second inference engine clients, so that the plurality of second inference engine clients can call the computing card through the inference engine server to perform comparison inference using the target language model and obtain a plurality of comparison inference results.

[0018] Furthermore, the inference server includes a third inference engine client, a fourth inference engine client, an inference engine server, and a computing card; the inference result generation unit includes: The second benchmark inference result generation module is used to send the test cases to the third inference engine client, so that the third inference engine client can call the computing card through the inference engine server to perform benchmark inference using the target language model and obtain benchmark inference results. The second comparison inference result generation module is used to repeatedly send the test cases to the fourth inference engine client in multiple rounds, so that the fourth inference engine client can call the computing card through the inference engine server to perform multiple rounds of comparison inference in sequence using the target language model to obtain multiple comparison inference results.

[0019] Furthermore, the inference server includes a fifth inference engine client, a sixth inference engine client, a first inference engine server, a second inference engine server, a first computing card, and a second computing card; the inference result generation unit includes: The third benchmark inference result generation module is used to send the test cases to the fifth inference engine client, so that the fifth inference engine client can call the first computing card through the first inference engine server to perform benchmark inference using the target language model and obtain benchmark inference results. The third comparison reasoning result generation module is used to send the test cases to the sixth reasoning engine client, so that the sixth reasoning engine client can call the second computing card through the second reasoning engine server to perform comparison reasoning using the target language model and obtain the comparison reasoning result.

[0020] Further, the inference server includes a first inference server and a second inference server; the first inference server includes a seventh inference engine client, a third inference engine server, and a third computing card; the second inference server includes an eighth inference engine client, a fourth inference engine server, and a fourth computing card; the inference result generation unit includes: The fourth benchmark inference result generation module is used to send the test cases to the seventh inference engine client, so that the seventh inference engine client can call the third computing card through the third inference engine server to perform benchmark inference using the target language model and obtain benchmark inference results. The fourth comparison inference result generation module is used to send the test cases to the eighth inference engine client, so that the eighth inference engine client can call the fourth computing card through the fourth inference engine server to perform comparison inference using the target language model and obtain the comparison inference result.

[0021] Furthermore, the reasoning result processing unit includes: The text segmentation module is used to perform text segmentation processing on the benchmark inference result and the inference result to be compared if the benchmark inference result and the inference result to be compared are Chinese inference results; The first case normalization module is used to perform case normalization on the baseline inference results and the inference results to be compared after text segmentation. The first symbol removal module is used to remove symbols from the baseline inference results and the inference results to be compared after case normalization. The first whitespace normalization module is used to normalize the whitespace inference results of the baseline inference results and the inference results to be compared after the symbol removal process, so as to obtain the normalized inference results to be compared and the baseline inference results.

[0022] Furthermore, the reasoning result processing unit includes: The English article removal module is used to remove English articles from the benchmark inference result and the inference result to be compared if they are English inference results. The second case normalization module is used to perform case normalization on the baseline inference results and the inference results to be compared after the removal of English articles; The second symbol removal module is used to remove symbols from the baseline inference results and the inference results to be compared after case normalization. The second whitespace normalization module is used to normalize the whitespace inference results of the baseline inference results and the inference results to be compared after the symbol removal process, so as to obtain the normalized inference results to be compared and the baseline inference results.

[0023] Furthermore, the reasoning result analysis unit includes: The inference accuracy calculation module is used to calculate the inference accuracy based on the number of consistent lexical units between the standardized inference result to be compared and the standardized baseline inference result, and the total number of lexical units in the standardized inference result to be compared. The inference hit rate calculation module is used to calculate the inference hit rate based on the number of consistent lexical units between the standardized inference result to be compared and the standardized baseline inference result, and the total number of lexical units in the standardized baseline inference result.

[0024] To achieve the above objectives, according to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for testing the consistency rate of large language model inference.

[0025] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program / instructions are stored, which, when executed by a processor, implement the steps of the above-described method for testing the consistency rate of large language model inference.

[0026] To achieve the above objectives, according to another aspect of the present invention, a computer program product is also provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the above-described method for testing the consistency rate of large language model inference.

[0027] The beneficial effects of this invention are as follows: By standardizing the inference results of large language models and calculating the inference consistency rate based on inference accuracy and inference hit rate, a quantitative assessment of inference consistency is achieved. This eliminates the need for comprehensive investigation and repeated verification when there are slight differences in inference results that are still within an acceptable range, thereby effectively reducing unnecessary R&D and verification costs. At the same time, the consistency rate score and its changes intuitively reflect the degree of difference in inference results under different scenarios, different links, or different computing resources. This provides R&D personnel with clear directional guidance for problem localization, helping to quickly determine whether the problem originates from the model itself, the inference engine, concurrency scheduling, or the hardware environment, significantly improving the efficiency of problem analysis and localization. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart illustrating the method for testing the inference consistency rate of a large language model provided in an embodiment of the present invention. Figure 2 This is a schematic block diagram of the structure of the testing device for the reasoning consistency rate of a large language model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the computer device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a single-machine fixed card multi-user parallel testing scenario provided in an embodiment of the present invention; Figure 5 This is a flowchart of the overall multi-user parallel testing scheme provided in the embodiments of the present invention; Figure 6 This is a flowchart of the benchmark inference result acquisition process provided in this embodiment of the invention; Figure 7 This is a flowchart of the process for obtaining the comparison reasoning result provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of a single-machine fixed card multi-round repeated testing scenario provided in an embodiment of the present invention; Figure 9 This is a flowchart of the overall scheme for multi-round repeated testing provided in the embodiments of the present invention; Figure 10 This is a schematic diagram of a single-machine multi-card consistency test scenario provided in an embodiment of the present invention; Figure 11 This is a flowchart of the overall testing solution between multiple cards on a single machine provided in this embodiment of the invention; Figure 12 This is a schematic diagram of a multi-machine consistency testing scenario provided in an embodiment of the present invention; Figure 13 This is a flowchart of the overall multi-machine testing solution provided in the embodiments of the present invention; Figure 14 This is a schematic diagram of a competitor's system consistency test scenario provided in an embodiment of the present invention; Figure 15 This is a flowchart of the overall solution for using a competitor's system as an external benchmark, provided in an embodiment of the present invention. Figure 16 This is a schematic diagram of an industry-standard output comparison test scenario provided in an embodiment of the present invention; Figure 17 This is a flowchart of the overall solution for using industry standard output as an external benchmark test provided by an embodiment of the present invention; Figure 18 This is a flowchart of the standardized processing of Chinese issues provided in the embodiments of the present invention; Figure 19 This is a flowchart of the standardized English problem handling process provided in the embodiments of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0031] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] The following describes the specific implementation process of the test method for the inference consistency rate of a large language model provided in this application embodiment, taking the test server as the execution subject as an example.

[0034] Figure 1 This is a flowchart illustrating the testing method for the reasoning consistency rate of a large language model provided in an embodiment of the present invention, as shown below. Figure 1 As shown, in one embodiment of the present invention, the method for testing the reasoning consistency rate of the large language model of the present invention includes: S101: Send the preset test cases to the inference server so that the inference server can perform the baseline inference and the inference to be compared respectively, and obtain the baseline inference result and the inference to be compared result. S102: Standardize the benchmark inference results and the inference results to be compared; S103: Generate inference accuracy and inference hit rate based on the standardized inference results to be compared and the standardized baseline inference results; S104: Calculate the reasoning consistency rate of the large language model based on the reasoning accuracy and the reasoning hit rate.

[0035] from Figure 1 As shown in the flowchart, the method for testing the inference consistency rate of a large language model provided in this application sends preset test cases to an inference server, which then executes baseline inference and inference to be compared, respectively, to obtain baseline inference results and inference results to be compared. The baseline inference results and the inference results to be compared are then standardized. Based on the standardized inference results to be compared and the standardized baseline inference results, inference precision and inference hit rate are generated. The inference consistency rate of the large language model is calculated based on the inference precision and the inference hit rate. This method achieves quantifiable, tolerant, and comparable evaluation of inference consistency, significantly improving the accuracy, interpretability, and adaptability of inference consistency verification.

[0036] Each step is explained in detail below.

[0037] S101: Send the preset test cases to the inference server so that the inference server can perform the baseline inference and the inference to be compared respectively, and obtain the baseline inference result and the inference to be compared result. Specifically, the test server initiates a large language model inference consistency test task and is responsible for the unified scheduling of the test process.

[0038] The test server sends test cases to the inference server, triggering the target language model to perform inference operations under the same model configuration conditions.

[0039] During the test, the test server triggers at least one inference request to obtain the baseline inference result, and triggers at least one additional inference request under the same test case conditions to obtain the inference result to be compared.

[0040] The baseline inference results are used to reflect the inference output performance of the target language model under stable execution conditions, while the inference results to be compared are used to reflect the changes in the model output under different inference execution instances. The test server collects and manages the obtained inference results in a unified manner to provide a data foundation for subsequent consistency rate calculation.

[0041] In one embodiment of the present invention, the inference server includes a first inference engine client, a plurality of second inference engine clients, an inference engine server, and a computing card; S101 includes: S201: The test case is sent to the first inference engine client, so that the first inference engine client calls the computing card through the inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result; Specifically, the test server organizes multi-user concurrent inference consistency tests and is responsible for distinguishing between the baseline inference process and the concurrent inference process.

[0042] At the start of the test, the test server sends the preset test cases to the first inference engine client. The first inference engine client then calls the computing card through the inference engine server and performs inference operations using the target language model to generate benchmark inference results.

[0043] The benchmark inference process is completed under a single inference request condition and is used to obtain the inference output of the target language model in the absence of concurrent interference. The test server receives and stores the generated benchmark inference results as a reference result for subsequent consistency evaluation.

[0044] S202: The test cases are sent to the plurality of second inference engine clients, so that the plurality of second inference engine clients call the computing card through the inference engine server to perform comparative inference using the target language model and obtain multiple comparative inference results.

[0045] Specifically, the test server sends the same test case to multiple second inference engine clients to trigger multiple inference requests to be sent to the inference engine server simultaneously.

[0046] Under multi-user concurrency conditions, the inference engine server calls the computing card and uses the target language model to execute multiple inference tasks, generating multiple inference output results respectively.

[0047] The test server receives inference outputs from multiple second inference engine clients and manages these inference outputs as inference results to be compared, so as to reflect the consistency of the target language model output under concurrent inference conditions.

[0048] The test server will organize the obtained baseline inference results with multiple inference results to be compared, forming a result set for inference consistency evaluation.

[0049] By using single-user inference results as a benchmark and inference results obtained under multi-user concurrent inference conditions as a comparison object, the test server can build a consistency evaluation data foundation suitable for multi-user concurrent scenarios, providing reliable input for subsequent consistency rate calculation based on inference accuracy and inference hit rate.

[0050] In one embodiment, the computing card can be an AI computing acceleration card, an inference computing card, etc., and the server where the computing card is located is an x86 server.

[0051] In one embodiment, the present invention verifies the inference consistency and performance stability of the system under high concurrency load by simulating multiple users concurrently accessing the same inference engine server. The test scenario is as follows: Figure 4 As shown, its overall scheme is as follows Figure 5 As shown.

[0052] In this embodiment, based on preset test cases, inference operations are performed on a target large language model to evaluate the consistency level of model inference output under multi-user parallel access scenarios. During the test, single-user inference is performed to generate a baseline inference result, while multi-user parallel inference is performed to generate inference results to be compared for each user. Subsequently, the baseline inference result and the inference results to be compared for each user are standardized to eliminate non-semantic effects caused by differences in text format, symbols, or encoding. After standardization, the test server calculates the inference consistency rate of each user's inference result relative to the baseline inference result based on inference accuracy and inference hit rate, which is used to quantify the consistency level of model inference output under different user conditions.

[0053] The process for obtaining the baseline inference results is as follows: Figure 6 As shown. After receiving the test case, the inference engine client sends an inference request to the inference engine server. The inference engine server, according to a predetermined scheduling strategy, allocates the inference computation task to the model inference unit to perform core computations, generates the corresponding model output result, and returns this output result as the baseline inference result to the inference engine client for display and further processing.

[0054] The process for obtaining the comparison and inference results is as follows: Figure 7 As shown, multiple test cases are used to simulate different user terminals simultaneously sending inference commands to the inference engine client. The inference engine client uniformly receives inference commands from different users and forwards these commands to the inference engine server. Under multi-user concurrency conditions, the inference engine server schedules multiple model inference units to execute inference computation tasks in parallel, generates corresponding inference output results, and returns each inference result to be compared to the corresponding client for display.

[0055] Using the above method, under the condition of a single-machine fixed computing card, it is possible to simultaneously obtain the baseline inference results and the inference results to be compared with multiple users, and to quantitatively evaluate the consistency of the inference output of the target large language model in a multi-user parallel scenario based on the inference consistency rate, thereby verifying the stability and deterministic performance of the model and inference engine in a multi-user parallel access scenario.

[0056] In one embodiment of the present invention, the inference server includes a third inference engine client, a fourth inference engine client, an inference engine server, and a computing card; S101 includes: S301: The test case is sent to the third inference engine client, so that the third inference engine client calls the computing card through the inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result; Specifically, the test server organizes the inference consistency test of a large language model in a single-machine, multi-round repeated inference scenario, and is responsible for the control of the test process and the management of the results.

[0057] At the start of the test, the test server sends the preset test cases to the third inference engine client. The third inference engine client then calls the computing card through the inference engine server and uses the target language model to perform the first round of inference operations, thereby generating the benchmark inference results.

[0058] The benchmark inference results are used to reflect the output performance of the target language model in the initial inference rounds. The test server receives and stores the benchmark inference results, which are used as reference results for subsequent multi-round inference consistency evaluation.

[0059] S302: The test cases are repeatedly sent to the fourth inference engine client in multiple rounds, so that the fourth inference engine client calls the computing card through the inference engine server to perform multiple rounds of comparison inference using the target language model in sequence, and obtain multiple comparison inference results.

[0060] Specifically, after obtaining the baseline inference results, the test server sends the same test case to the fourth inference engine client multiple times according to a preset number of rounds, so as to trigger the inference engine server to execute multiple rounds of inference operations sequentially under the same computing card conditions.

[0061] In each round of inference, the inference engine server calls the target language model to generate the corresponding inference output. The test server receives the inference output of each round and records these inference outputs as inference results to be compared.

[0062] Through multiple rounds of repeated inference, the test server can collect the inference results of the target language model under the same hardware environment and inference configuration, which can be used to reflect the stability of the model output over time.

[0063] The test server organizes and aggregates the obtained baseline inference results and the inference results to be compared in multiple rounds, forming a result set for evaluating the inference consistency rate in a single-machine multi-round repeated inference scenario.

[0064] By using the results of the first round of inference as a benchmark and the results of subsequent rounds of inference as comparison objects, the test server can build an evaluation data foundation for measuring the consistency of the target language model's multi-round inference outputs, providing reliable input for subsequent consistency rate calculations based on inference accuracy and inference hit rate.

[0065] In one embodiment, the stability and reproducibility of the inference results in multiple rounds of execution are verified by repeatedly executing the same test cases on the same computing device and a fixed computing card. The test scenario is as follows: Figure 8 As shown, its overall scheme is as follows Figure 9 As shown.

[0066] In this embodiment, based on preset test cases, the target large language model is subjected to multiple rounds of repeated inference on the same computing card to verify the consistency of the model's inference output during multiple consecutive executions. At the start of the test, test cases are first input, and it is determined whether the current execution round is the first inference round. If the current execution round is the first round, model inference is triggered to generate a baseline inference result, and the generated baseline inference result is standardized. If the current execution round is a subsequent round, model inference is performed to obtain the inference result to be compared for the corresponding round, and the same standardization process as the baseline inference result is applied to the inference result to be compared.

[0067] After each round of inference, the round parameter is incremented, and it is continued to be checked whether the preset number of inference rounds has been reached. If the preset number of rounds has not been reached, the model inference and result standardization process is repeated. After all rounds of inference have been executed, the benchmark inference result generated in the first round is used as a reference benchmark to calculate the inference consistency rate between the inference results to be compared in each round and the benchmark inference result, thereby quantitatively evaluating the inference consistency level of the target large language model under the condition of multiple rounds of repeated execution.

[0068] The process for obtaining the baseline inference results and the inference results to be compared in multiple rounds is the same as that in Example 1. In both cases, the inference engine client initiates an inference request to the inference engine server, and the server schedules the model inference unit to execute computational tasks and generate the inference output results. The standardization process for the inference results and the calculation method for the inference consistency rate are also consistent with Example 1 and will not be described again here.

[0069] By using the aforementioned consistency test method involving multiple rounds of repeated inference, we can quantitatively evaluate the inference stability and reproducibility of a target large language model over time under the condition of a single-machine fixed computing card. This provides a reliable basis for model validation, inference engine stability analysis, and subsequent problem localization.

[0070] In one embodiment of the present invention, the inference server includes a fifth inference engine client, a sixth inference engine client, a first inference engine server, a second inference engine server, a first computing card, and a second computing card; S101 includes: S401: The test case is sent to the fifth inference engine client, so that the fifth inference engine client calls the first computing card through the first inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result; Specifically, the test server organizes large language model inference consistency tests in a single-machine, multi-computing-card scenario and is responsible for scheduling the test process and managing the results.

[0071] At the start of the test, the test server sends the preset test cases to the fifth inference engine client. The fifth inference engine client then calls the first computing card through the first inference engine server and performs inference operations using the target language model to generate benchmark inference results.

[0072] The benchmark inference results are used to reflect the inference output performance of the target language model under the execution conditions of the first computing card. The test server receives and stores the generated benchmark inference results, which are used as reference results for cross-computing card consistency evaluation.

[0073] S402: The test case is sent to the sixth inference engine client, so that the sixth inference engine client calls the second computing card through the second inference engine server to perform the comparison inference using the target language model and obtain the comparison inference result.

[0074] Specifically, the test server sends the same test case to the sixth inference engine client to trigger the second inference engine server to call the second computing card and perform inference operations using the target language model, thereby generating the inference results to be compared.

[0075] By executing the same test case inference operations on different computing cards, the test server can obtain the inference output of the target language model under different computing resource conditions, which can be used to reflect the impact of different computing cards on the inference results.

[0076] The test server receives and records the results of the inference to be compared, providing comparative data for subsequent consistency rate evaluation.

[0077] The test server organizes and aggregates the obtained baseline inference results and the inference results to be compared, forming a set of inference results suitable for single-machine multi-computing-card scenarios.

[0078] By using the inference results generated by the first computing card as a reference benchmark and the inference results generated by the second computing card as a comparison object, the test server can build an evaluation data foundation for measuring the consistency of the inference output of the target language model under different computing card execution conditions, providing reliable input for subsequent consistency rate calculation based on inference accuracy and inference hit rate.

[0079] In one embodiment, the consistency of results from different computing cards or different computing card groups under the same model and inference configuration is detected by executing the same test cases on different computing cards or different computing card groups on the same device. The test scenario is as follows: Figure 10 As shown, its overall scheme is as follows Figure 11 As shown.

[0080] In this embodiment, at least two computing cards are configured on the same device, and based on preset test cases, inference operations of the target large language model are executed on different computing cards to verify the consistency of the model's inference output under different computing card conditions. During the test, model inference operations are executed on different computing cards to obtain the corresponding inference output results.

[0081] The process for obtaining the baseline inference results and the inference results to be compared is the same as that for obtaining the baseline inference results. Both involve the inference engine client initiating an inference request to the inference engine server. The inference engine server then allocates the inference computation task to the designated model inference unit according to the scheduling strategy, and executes the computation task on the corresponding computing card to generate the model inference output results. By executing the same test case inference operations on different computing cards, the baseline inference results and the inference results to be compared are obtained respectively.

[0082] After obtaining the inference results, the standardization process of the inference results and the calculation method of the inference consistency rate are consistent with those in Example 1. That is, the benchmark inference results and the inference results to be compared are uniformly standardized, and the inference consistency rate is calculated based on the inference accuracy and the inference hit rate. This will not be described again here.

[0083] By using the above-mentioned consistency test method with multiple computing cards on a single machine, we can quantitatively evaluate the consistency of the inference output of the target large language model under different computing card execution conditions, thereby verifying the stability and deterministic performance of the model and inference engine in a multi-computing card environment.

[0084] In one embodiment of the present invention, the inference server includes a first inference server and a second inference server; the first inference server includes a seventh inference engine client, a third inference engine server, and a third computing card; the second inference server includes an eighth inference engine client, a fourth inference engine server, and a fourth computing card; S101 includes: S501: The test case is sent to the seventh inference engine client, so that the seventh inference engine client calls the third computing card through the third inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result; Specifically, the test server organizes large language model inference consistency tests in a multi-machine environment and is responsible for the scheduling and result management of the overall test process.

[0085] At the start of the test, the test server sends the preset test cases to the seventh inference engine client in the first inference server. The inference engine client then calls the third computing card through the third inference engine server and performs inference operations using the target language model to generate benchmark inference results.

[0086] The benchmark inference results are used to reflect the inference output performance of the target language model in the first inference server operating environment. The test server receives and stores the benchmark inference results and uses them as reference results for multi-machine consistency evaluation.

[0087] S502: The test case is sent to the eighth inference engine client, so that the eighth inference engine client calls the fourth computing card through the fourth inference engine server to perform the comparison inference using the target language model and obtain the comparison inference result.

[0088] Specifically, the test server sends the same test cases to the eighth inference engine client in the second inference server to trigger the fourth inference engine server to call the fourth computing card and perform inference operations using the target language model, thereby generating the inference results to be compared.

[0089] By executing the same test case inference operations on different inference servers, the test server can obtain the inference output of the target language model under different hardware environments, system configurations, or deployment conditions, which can be used to reflect the consistency of inference results under cross-server conditions.

[0090] The test server receives and records the results of the inference to be compared, providing a basis for comparative data for subsequent consistency rate evaluation.

[0091] The test server organizes and aggregates the obtained baseline inference results and the inference results to be compared, forming a set of inference results suitable for multi-machine inference scenarios.

[0092] By using the inference results generated by the first inference server as a reference benchmark and the inference results generated by the second inference server as a comparison object, the test server can build an evaluation data foundation for measuring the consistency of the inference output of the target language model under different server environments, providing reliable input for subsequent consistency rate calculation based on inference accuracy and inference hit rate.

[0093] In one embodiment, the consistency between the inference engine and model output is verified by executing the same inference task on multiple independent computing devices or in different deployment environments. The test scenario is as follows: Figure 12 As shown, the overall testing process is as follows: Figure 13 As shown.

[0094] In this embodiment, an inference runtime environment is deployed on multiple physical devices, and based on preset test cases, the inference operation of the target large language model is executed on different physical devices to verify the consistency of the model inference output under cross-device conditions. During the test, the inference tasks are distributed across multiple physical devices, and each physical device independently completes the model inference calculation and generates the corresponding inference output results.

[0095] The overall testing process is basically the same as the single-machine multi-compute card consistency testing process described above. The difference is that the inference tasks are no longer executed on different compute cards on the same physical device, but rather on multiple independent physical devices. By running the same test case inference operations on different physical devices, the corresponding inference results are obtained for cross-device consistency evaluation.

[0096] After obtaining the inference results, the standardization process of the inference results and the calculation method of the inference consistency rate are consistent with those in Example 1. That is, the inference results generated by different devices are uniformly standardized, and the inference consistency rate is calculated based on the inference accuracy and inference hit rate. This will not be described again here.

[0097] By using the above multi-machine consistency test method, we can quantitatively evaluate the consistency of the inference output of the target large language model under different physical device deployment conditions, thereby verifying the stability, determinism and reproducibility of the model and inference engine in cross-device environments.

[0098] In one embodiment, a competitor's system is used as an external benchmark to compare the consistency rate of the target model's inference results, thereby evaluating the system's external compatibility and industry consistency. The test scenario is as follows: Figure 14 As shown, the overall testing process is as follows: Figure 15 As shown.

[0099] In this embodiment, a competitor's system is used as an external reference benchmark. Based on preset test cases, inference operations on the target language model are performed on both the competitor's system and the local system to achieve a consistent evaluation of cross-system inference output. At the start of the test, inference operations on the target language model are first performed on the competitor's system to generate an external benchmark inference result, which is then standardized. Simultaneously, the same test cases are used to perform inference operations on the target language model on the local system, generating a corresponding inference result to be compared. This result undergoes the same standardization process as the external benchmark inference result.

[0100] The process for obtaining the inference results to be compared is the same as that for obtaining the baseline inference results in Example 1. Both involve the inference engine client initiating an inference request to the inference engine server, and the server scheduling the model inference unit to execute computational tasks and generate the inference output results. The standardization process for the inference results and the calculation method for the inference consistency rate are also consistent with Example 1 and will not be repeated here.

[0101] After standardizing the external benchmark inference results and the inference results to be compared, the inference consistency rate between the two is calculated based on the inference accuracy and inference hit rate, thereby realizing the quantitative comparison and quality assessment of the output results of different systems.

[0102] By using the aforementioned consistency rate test method with competitor systems as external benchmarks, we can quantitatively evaluate the consistency and compatibility of the target language model's inference output in a cross-system environment, providing an effective technical means for external benchmarking analysis, product quality assessment, and cross-platform consistency verification.

[0103] In one embodiment, industry standard output is used as an external benchmark to compare the inference results of the target model and evaluate the degree of conformity between the system and the industry standard. The test scenario is as follows: Figure 16 As shown, the overall testing process is as follows: Figure 17 As shown.

[0104] In this embodiment, industry-standard output is used as an external reference benchmark. Based on preset test cases, the consistency of the inference results generated by the target language model is evaluated. At the start of the test, the external benchmark inference result corresponding to the test case is obtained from the industry-standard output and standardized. Simultaneously, the target language model inference operation is performed on the local system using the same test cases to generate the corresponding inference result to be compared. The same standardization process as the external benchmark inference result is then applied to the inference result to be compared.

[0105] After standardizing the external benchmark inference results and the inference results to be compared, the inference consistency rate between the two is calculated based on the inference accuracy and inference hit rate, thereby achieving a quantitative evaluation of the consistency between the target language model's inference output and the industry standard output.

[0106] The overall testing process is similar to the consistency test process using a competitor's system as an external benchmark. The difference lies in that the external benchmark inference results no longer originate from the competitor's system's inference output, but are directly derived from industry standard output. The standardization process for inference results and the calculation method for inference consistency rate remain consistent with Example 1, and will not be repeated here.

[0107] By using the above-mentioned consistency test method with industry standard output as an external benchmark, the consistency level between the target language model inference output and the industry-recognized standard can be quantitatively evaluated, thus providing an effective technical means for model quality verification, system performance evaluation, and industry consistency benchmarking.

[0108] S102: Standardize the benchmark inference results and the inference results to be compared; Specifically, after receiving the baseline inference results and the inference results to be compared, the test server performs standardization processing on each inference result to eliminate the impact of non-semantic differences on the consistency evaluation.

[0109] The standardization process includes text segmentation of the inference results, case unification, symbol standardization, and whitespace standardization, thereby converting different inference results into standardized text forms with consistent format and comparable structure.

[0110] This step ensures that subsequent consistency calculations focus on differences in the inference content itself, rather than interference factors introduced by format differences, symbol differences, or encoding differences.

[0111] In one embodiment of the present invention, S102 includes: S601: If the benchmark inference result and the inference result to be compared are Chinese inference results, perform text segmentation on the benchmark inference result and the inference result to be compared. Specifically, a differentiated standardization process for inference results is employed for test questions in different language types. Before standardizing the inference results, they are first differentiated based on the language of the test question, determining whether the corresponding test question is in Chinese or English. For each language type, a standardization process tailored to its characteristics is used to eliminate the impact of language differences on the consistency assessment of inference results.

[0112] When the test question involves Chinese characters, the test server follows... Figure 18 The process shown standardizes the inference results. The test server first performs text segmentation on the inference results. Through text segmentation, the test server divides continuous Chinese text into multiple discrete text units, transforming the inference results from the original string format into a structured sequence of words, thus providing a uniform processing granularity for subsequent consistency analysis.

[0113] S602: Perform case normalization on the baseline inference results and the inference results to be compared after text segmentation; Specifically, after completing the text segmentation process, the test server performs case normalization on the segmented inference results. Case normalization converts all letter characters in the text into a preset form to avoid affecting the consistency comparison of inference results due to differences in letter case, thereby ensuring that subsequent consistency calculations focus on the differences in the inference content itself.

[0114] S603: Perform symbol removal processing on the baseline inference results and the inference results to be compared after case normalization; Specifically, after completing the case normalization process, the test server performs symbol removal processing on the inference results. Symbol removal processing is used to remove punctuation marks and non-semantic symbols (control characters, newline characters, tab characters, and emojis, etc.) from the inference results to reduce the interference of symbol differences on the consistency evaluation results and make the inference results more focused on reflecting the semantic content of the text.

[0115] S604: Perform blank symbol normalization on the baseline inference result and the inference result to be compared after symbol removal processing to obtain the standardized inference result to be compared and the baseline inference result.

[0116] Specifically, after completing the symbol removal process, the test server performs whitespace normalization on the inference results. Whitespace normalization is used to uniformly handle spaces, newlines, or tabs in the text, ensuring that whitespace characters in the inference results maintain a consistent expression, thereby obtaining standardized inference results with uniform structure and comparable content.

[0117] In one embodiment of the present invention, S102 includes: S701: If the benchmark inference result and the inference result to be compared are English inference results, perform English article removal processing on the benchmark inference result and the inference result to be compared; Specifically, when the test question is in English, the test server follows... Figure 19 The process shown standardizes the inference results. The test server first performs English article removal on the inference results.

[0118] The English article removal process is used to remove functional words from the inference results that do not affect the core semantic expression, so as to reduce the interference of differences in article usage on the consistency assessment of inference results, thereby making subsequent comparisons more focused on the semantic content itself.

[0119] S702: Perform case normalization on the baseline inference results and the inference results to be compared after removing English articles; Specifically, after removing English articles, the test server performs case normalization on the inference results. By uniformly converting the letters in the text to a preset form, the test server can eliminate non-semantic differences introduced by differences in capitalization, ensuring that different inference results have a consistent basis for comparison at the character level.

[0120] S703: Perform symbol removal processing on the baseline inference results and the inference results to be compared after case normalization; Specifically, after completing the case normalization process, the test server performs symbol removal processing on the inference results. Symbol removal processing is used to remove punctuation marks and other non-semantic symbols (control characters, newline characters, tabs, and emojis, etc.) from the text, reducing the impact of symbol usage differences on the consistency calculation results, and making the inference results more focused on reflecting the semantic information generated by the language model.

[0121] S704: Perform blank symbol normalization on the baseline inference result and the inference result to be compared after symbol removal processing to obtain the standardized inference result to be compared and the baseline inference result.

[0122] Specifically, after completing the symbol removal process, the test server performs whitespace normalization on the inference results. Whitespace normalization is used to uniformly handle spaces, newlines, or tabs in the text, ensuring consistent text structure and thus obtaining standardized English inference results with uniform format and comparable content.

[0123] By adopting differentiated standardized processing procedures for Chinese and English test questions, the impact of differences in grammatical structure, function word usage, and text expression in different languages ​​on the consistency assessment of inference results can be eliminated more effectively, thereby improving the accuracy and reliability of the calculation results of inference precision, inference hit rate, and inference consistency rate.

[0124] The standardization process in this application serves the testing objective of quantifying the consistency rate of reasoning. By setting differentiated standardization processes for different language types, general vocabulary, symbols, and format differences that do not affect the core semantic expression are filtered out, and only output units that are discriminative in terms of the consistency of reasoning content are retained, making the reasoning results comparable at the semantic level.

[0125] Building upon this foundation, standardization provides a stable and reliable input for subsequent consistency score calculations, enabling the consistency score to accurately reflect the matching relationships between different inference results on key output content. This avoids distortions in consistency assessment caused by differences in articles, word segmentation, or formatting. By integrating standardization with consistency score calculation as an organic whole, this application not only achieves continuous quantitative evaluation of the degree of inconsistency in large language model inference but also significantly improves the accuracy and interpretability of consistency test results, providing reliable technical support for multi-scenario inference stability assessment and problem localization.

[0126] S103: Generate inference accuracy and inference hit rate based on the standardized inference results to be compared and the standardized baseline inference results; Specifically, the test server generates inference accuracy and inference hit rate based on the inference results after standardization, which are used to characterize the degree of consistency between the inference results to be compared and the benchmark inference results.

[0127] Inference precision reflects the proportion of output content that is consistent with the benchmark inference result in the inference result to be compared, while inference hit rate reflects the coverage of key output content in the benchmark inference result in the inference result to be compared.

[0128] By simultaneously introducing inference accuracy and inference hit rate, the test server can comprehensively measure the consistency of inference results from two dimensions: output accuracy and content coverage.

[0129] In one embodiment of the present invention, S103 includes: S801: Calculate the inference accuracy based on the number of consistent lexical units between the standardized inference result to be compared and the standardized baseline inference result, and the total number of lexical units in the standardized inference result to be compared. Specifically, after standardizing the inference results, the test server performs a word-by-word comparison between the inference results to be compared and the benchmark inference results.

[0130] During the comparison process, the test server aligns the lexical units in the two inference results based on a unified word segmentation or output unit partitioning rule, and identifies lexical units that are consistent in content and position.

[0131] In this way, the test server counts the number of consistent lexical units between the inference result to be compared and the benchmark inference result, which is used to reflect the degree of matching between the two inference results at the fine-grained output level.

[0132] The test server counts the lexical units contained in the comparison inference results after the standardization process is completed, in order to obtain the total number of lexical units in the comparison inference results.

[0133] The total number of lexical units is used to characterize the overall output scale of the reasoning results to be compared, providing a normalized benchmark for the calculation of subsequent reasoning accuracy.

[0134] By introducing the total number of tokens, the test server can avoid bias in consistency assessment caused by differences in the length of inference results.

[0135] The test server calculates the inference accuracy based on the number of consistent tokens and the total number of tokens in the inference results to be compared.

[0136] Inference accuracy is used to reflect the proportion of output content in the inference results to be compared that is consistent with the benchmark inference results in the overall output, thereby measuring the output accuracy of the inference results to be compared.

[0137] This metric allows the test server to characterize the fine-grained matching degree between the inference result to be compared and the benchmark inference result.

[0138] S802: Calculate the inference hit rate based on the number of consistent lexical units between the standardized inference result to be compared and the standardized baseline inference result, and the total number of lexical units in the standardized baseline inference result.

[0139] Specifically, the test server performs lexical statistics on the benchmark inference results that have undergone standardization to obtain the total number of lexical units in the benchmark inference results.

[0140] The total number of morphemes is used to characterize the scale of the complete output content contained in the benchmark inference result, providing a reference benchmark for calculating the inference hit rate.

[0141] By incorporating the total number of lexical units in the baseline inference result, the test server can evaluate the matching of the inference result to be compared with the baseline inference result from a coverage perspective.

[0142] The test server calculates the inference hit rate based on the number of consistent tokens and the total number of tokens in the baseline inference result.

[0143] Inference hit rate is used to reflect the extent to which the output content in the benchmark inference result is covered in the inference result to be compared, thereby measuring the hit rate of the inference result to be compared with the benchmark inference result.

[0144] By combining inference accuracy and inference hit rate, the test server can comprehensively characterize the consistency of inference results from two dimensions: output accuracy and content coverage.

[0145] S104: Calculate the reasoning consistency rate of the large language model based on the reasoning accuracy and the reasoning hit rate.

[0146] Specifically, the test server calculates the reasoning consistency rate of the large language model based on the generated reasoning accuracy and reasoning hit rate, which is used to quantitatively evaluate the reasoning consistency level of the target language model under the same input conditions.

[0147] Inference consistency reflects the degree of similarity between the output results of different inference execution instances. It can be used to determine whether the model's inference output remains stable and consistent, or whether there are differences that exceed the acceptable range.

[0148] By quantifying the inference consistency rate, the test server can assess inference stability without relying on binary consistency judgment, providing an objective basis for system verification, model evaluation, and subsequent problem localization.

[0149] In one embodiment, after the Chinese or English reasoning results are processed according to the corresponding standardized processing flow, the test server performs a consistency evaluation on the standardized reasoning results and calculates the reasoning consistency score.

[0150] For each user-generated inference result to be compared, the test server compares that result with the baseline inference result and calculates the corresponding inference consistency score. The inference consistency score is calculated as follows: (1) in: (2) (3) In a single-machine, fixed-card, multi-user parallel testing scenario, after completing the above calculations, the test server can further perform statistical analysis on the inference consistency scores of multiple users in the same test scenario. When the average inference consistency score for each user is equal to 1.0, it indicates that the inference results generated by each user under the same test case conditions are completely consistent; when the average inference consistency score for each user is less than 1.0, it indicates that there are inconsistencies in the inference results between different users, and the smaller the average inference consistency score, the higher the degree of inconsistency between the inference results.

[0151] By adopting the above-mentioned reasoning consistency score calculation method, we can continuously quantify the consistency of reasoning results while maintaining the simplicity of the calculation logic. This avoids the misjudgment problem caused by relying solely on binary judgments of whether or not the reasoning is consistent, and provides a more refined and reliable quantitative basis for consistency analysis of multi-user, multi-round, or cross-environment reasoning.

[0152] This application provides a method for testing the inference consistency rate of a large language model. It involves sending preset test cases to an inference server, which then executes baseline inference and comparison inference to obtain baseline and comparison results. The baseline and comparison results are then standardized. Inference precision and inference hit rate are generated based on the standardized comparison and baseline inference results. Finally, the inference consistency rate of the large language model is calculated based on the inference precision and inference hit rate. This method achieves quantifiable, tolerant, and comparable evaluation of inference consistency, significantly improving the accuracy, interpretability, and adaptability of inference consistency verification.

[0153] This invention sends pre-defined test cases to the inference engine server via the inference engine client. Upon receiving the test request, the server schedules model inference resources and invokes the model inference unit to execute computational tasks, generating corresponding model inference output results. Under the same inference process and configuration conditions, benchmark inference results and comparison results are obtained. Subsequently, the obtained inference results undergo unified standardization processing, including text segmentation, case normalization, symbol removal, and whitespace normalization, to eliminate the impact of format differences on result comparison. Finally, based on the standardized inference results, the inference precision and inference hit rate between the benchmark and comparison results are calculated, and an inference consistency score is generated based on the inference precision and inference hit rate.

[0154] By using inference consistency scores, the consistency level of a model's output under the same input and operating environment conditions can be quantitatively evaluated. This allows for the assessment of the deterministic and stability performance of inference engines and large language models in scenarios involving multiple rounds of execution, multiple user accesses, multiple computing cards, or multiple device deployments. Furthermore, this method supports using competitor system outputs or industry standard outputs as external reference benchmarks to evaluate the external compatibility and industry consistency of the target system. Quantitatively evaluating the inference consistency rate of large language models not only meets the tolerance requirements for result consistency in different R&D and testing scenarios, effectively reducing unnecessary R&D and verification costs when inference discrepancies are within acceptable limits, but also serves as an analytical tool for problem localization. Changes in consistency rates provide directional guidance for discrepancy analysis, assisting in identifying and pinpointing root causes of problems, thereby significantly improving the efficiency of problem localization and analysis.

[0155] In summary, this invention addresses the technical problem that existing large language models can only make qualitative judgments on reasoning consistency and are unable to quantitatively assess the degree of difference in reasoning results. It proposes a method for calculating reasoning consistency rate based on reasoning accuracy and reasoning hit rate. By continuously and quantitatively evaluating the consistency of reasoning results, unnecessary repeated verification and problem investigation can be avoided when the reasoning differences are within an acceptable range, thereby effectively reducing R&D and verification costs. Simultaneously, by utilizing the reasoning consistency rate score and its changes, clear directional guidance is provided for analyzing differences in reasoning results, assisting in quickly locating and identifying the root causes of problems, and significantly improving the efficiency of problem location and analysis.

[0156] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0157] Based on the same inventive concept, embodiments of the present invention also provide a testing apparatus for the consistency rate of large language model reasoning, which can be used to implement the testing method for the consistency rate of large language model reasoning described in the above embodiments, as described in the following embodiments. Since the principle of the testing apparatus for the consistency rate of large language model reasoning is similar to that of the testing method for the consistency rate of large language model reasoning, embodiments of the testing apparatus for the consistency rate of large language model reasoning can refer to embodiments of the testing method for the consistency rate of large language model reasoning, and repeated details will not be elaborated upon. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0158] Figure 2This is a schematic block diagram of the structure of the testing device for the reasoning consistency rate of a large language model provided in an embodiment of the present invention, as shown below. Figure 2 As shown, in one embodiment of the present invention, the testing device for the reasoning consistency rate of the large language model of the present invention includes: The inference result generation unit 201 is used to send preset test cases to the inference server, so that the inference server performs benchmark inference and comparison inference respectively, and obtains benchmark inference result and comparison inference result; The reasoning result processing unit 202 is used to standardize the benchmark reasoning result and the reasoning result to be compared. The reasoning result analysis unit 203 is used to generate reasoning accuracy and reasoning hit rate based on the standardized reasoning results to be compared and the standardized benchmark reasoning results. The reasoning consistency rate generation unit 204 is used to calculate the reasoning consistency rate of the large language model based on the reasoning accuracy rate and the reasoning hit rate.

[0159] In one embodiment of the present invention, the inference server includes a first inference engine client, multiple second inference engine clients, an inference engine server, and a computing card; the inference result generation unit 201 includes: The first benchmark inference result generation module 1001 is used to send the test cases to the first inference engine client, so that the first inference engine client can call the computing card through the inference engine server to perform benchmark inference using the target language model and obtain benchmark inference results. The first comparison inference result generation module 1002 is used to send the test cases to the plurality of second inference engine clients, so that the plurality of second inference engine clients can call the computing card through the inference engine server to perform comparison inference using the target language model and obtain a plurality of comparison inference results.

[0160] In one embodiment of the present invention, the inference server includes a third inference engine client, a fourth inference engine client, an inference engine server, and a computing card; the inference result generation unit 201 includes: The second benchmark inference result generation module 1101 is used to send the test cases to the third inference engine client, so that the third inference engine client can call the computing card through the inference engine server to perform benchmark inference using the target language model and obtain benchmark inference results. The second comparison reasoning result generation module 1102 is used to repeatedly send the test cases to the fourth reasoning engine client in multiple rounds, so that the fourth reasoning engine client can call the computing card through the reasoning engine server to execute multiple rounds of comparison reasoning using the target language model in sequence, and obtain multiple comparison reasoning results.

[0161] In one embodiment of the present invention, the inference server includes a fifth inference engine client, a sixth inference engine client, a first inference engine server, a second inference engine server, a first computing card, and a second computing card; the inference result generation unit 201 includes: The third benchmark inference result generation module 1201 is used to send the test cases to the fifth inference engine client, so that the fifth inference engine client can call the first computing card through the first inference engine server to perform benchmark inference using the target language model and obtain benchmark inference results. The third comparison reasoning result generation module 1202 is used to send the test cases to the sixth reasoning engine client, so that the sixth reasoning engine client calls the second computing card through the second reasoning engine server to perform comparison reasoning using the target language model and obtain the comparison reasoning result.

[0162] In one embodiment of the present invention, the inference server includes a first inference server and a second inference server; the first inference server includes a seventh inference engine client, a third inference engine server, and a third computing card; the second inference server includes an eighth inference engine client, a fourth inference engine server, and a fourth computing card; the inference result generation unit 201 includes: The fourth benchmark inference result generation module 1301 is used to send the test cases to the seventh inference engine client, so that the seventh inference engine client calls the third computing card through the third inference engine server to perform benchmark inference using the target language model and obtain benchmark inference results; The fourth comparison reasoning result generation module 1302 is used to send the test cases to the eighth reasoning engine client, so that the eighth reasoning engine client calls the fourth computing card through the fourth reasoning engine server to perform comparison reasoning using the target language model and obtain the comparison reasoning result.

[0163] In one embodiment of the present invention, the reasoning result processing unit 202 includes: The text segmentation module 1401 is used to perform text segmentation processing on the benchmark inference result and the inference result to be compared if the benchmark inference result and the inference result to be compared are Chinese inference results; The first case normalization module 1402 is used to perform case normalization on the baseline inference result and the inference result to be compared after text segmentation. The first symbol removal module 1403 is used to perform symbol removal processing on the baseline inference result and the inference result to be compared after case normalization. The first blanking symbol normalization module 1404 is used to normalize the blanking symbol of the baseline inference result and the inference result to be compared after symbol removal processing, so as to obtain the normalized inference result to be compared and the baseline inference result.

[0164] In one embodiment of the present invention, the reasoning result processing unit 202 includes: The English article removal module 1501 is used to remove English articles from the benchmark inference result and the inference result to be compared if they are English inference results. The second case normalization module 1502 is used to perform case normalization processing on the baseline inference result and the inference result to be compared after the removal of English articles; The second symbol removal module 1503 is used to perform symbol removal processing on the baseline inference result and the inference result to be compared after case normalization. The second whitespace normalization module 1504 is used to perform whitespace normalization on the baseline inference result and the inference result to be compared after symbol removal processing, so as to obtain the standardized inference result to be compared and the baseline inference result.

[0165] In one embodiment of the present invention, the reasoning result analysis unit 203 includes: The inference accuracy calculation module 1601 is used to calculate the inference accuracy based on the number of consistent lexical units between the standardized inference result to be compared and the standardized baseline inference result, and the total number of lexical units in the standardized inference result to be compared. The inference hit rate calculation module 1602 is used to calculate the inference hit rate based on the number of consistent lexical units between the standardized inference result to be compared and the standardized baseline inference result, and the total number of lexical units in the standardized baseline inference result.

[0166] This application provides a method and apparatus for testing the inference consistency rate of a large language model. By sending preset test cases to an inference server, the inference server executes baseline inference and comparison inference respectively, obtaining baseline inference results and comparison results. The baseline and comparison results are then standardized. Inference precision and inference hit rate are generated based on the standardized comparison results and the standardized baseline inference results. The inference consistency rate of the large language model is calculated based on the inference precision and inference hit rate. This method achieves quantifiable, tolerant, and comparable evaluation of inference consistency, significantly improving the accuracy, interpretability, and adaptability of inference consistency verification.

[0167] Figure 3This is a schematic diagram of the structure of the computer device provided in an embodiment of the present invention, such as... Figure 3 As shown, the electronic device may include a processor 301, a communication interface 302, a memory 303, and a communication bus 304. The processor 301, communication interface 302, and memory 303 communicate with each other via the communication bus 304. The processor 301 can call logical instructions in the memory 303 to execute the following methods: sending preset test cases to an inference server, causing the inference server to execute baseline inference and comparison inference respectively, obtaining baseline inference results and comparison inference results; standardizing the baseline inference results and comparison inference results; generating inference precision and inference hit rate based on the standardized comparison inference results and standardized baseline inference results; and calculating the inference consistency rate of the large language model based on the inference precision and inference hit rate.

[0168] Furthermore, the logical instructions in the aforementioned memory 303 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a top-drive control center server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] This embodiment discloses a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the methods provided in the above-described method embodiments, such as: sending preset test cases to an inference server, so that the inference server performs baseline inference and comparison inference respectively, to obtain baseline inference results and comparison inference results; standardizing the baseline inference results and comparison inference results; generating inference precision and inference hit rate based on the standardized comparison inference results and standardized baseline inference results; and calculating the inference consistency rate of a large language model based on the inference precision and inference hit rate.

[0170] This embodiment provides a computer-readable storage medium storing a computer program that causes the computer to execute the methods provided in the above-described method embodiments. For example, the methods include: sending preset test cases to an inference server, causing the inference server to perform baseline inference and comparison inference respectively, obtaining baseline inference results and comparison inference results; standardizing the baseline inference results and comparison inference results; generating inference precision and inference hit rate based on the standardized comparison inference results and standardized baseline inference results; and calculating the inference consistency rate of a large language model based on the inference precision and inference hit rate.

[0171] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0173] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0175] In the description of this specification, the references to terms such as "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0176] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for testing inference consistency of a large language model, characterized in that, include: The preset test cases are sent to the inference server, so that the inference server performs the baseline inference and the inference to be compared respectively, and obtains the baseline inference result and the inference to be compared result. The baseline inference results and the inference results to be compared are standardized. The inference accuracy and inference hit rate are generated based on the standardized inference results to be compared and the standardized baseline inference results. The reasoning consistency rate of the large language model is calculated based on the reasoning accuracy and the reasoning hit rate.

2. The method of claim 1, wherein the method is performed by a computer system. The inference server includes a first inference engine client, multiple second inference engine clients, an inference engine server, and a computing card; the step of sending preset test cases to the inference server, so that the inference server executes benchmark inference and comparison inference respectively, to obtain benchmark inference results and comparison inference results, includes: The test cases are sent to the first inference engine client, so that the first inference engine client calls the computing card through the inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result. The test cases are sent to the multiple second inference engine clients, so that the multiple second inference engine clients can call the computing card through the inference engine server to perform comparative inference using the target language model and obtain multiple comparative inference results. 3.The method of claim 1, wherein, The inference server includes a third inference engine client, a fourth inference engine client, an inference engine server, and a computing card; the step of sending preset test cases to the inference server, so that the inference server executes baseline inference and comparison inference respectively, to obtain baseline inference results and comparison inference results, includes: The test cases are sent to the third inference engine client, so that the third inference engine client calls the computing card through the inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result. The test cases are repeatedly sent to the fourth inference engine client in multiple rounds, so that the fourth inference engine client calls the computing card through the inference engine server to perform multiple rounds of comparison inference using the target language model in sequence, and obtain multiple comparison inference results. 4.The method of claim 1, wherein, The inference server includes a fifth inference engine client, a sixth inference engine client, a first inference engine server, a second inference engine server, a first computing card, and a second computing card; the step of sending preset test cases to the inference server, so that the inference server performs baseline inference and comparison inference respectively, to obtain baseline inference results and comparison inference results, includes: The test cases are sent to the fifth inference engine client, so that the fifth inference engine client calls the first computing card through the first inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result. The test cases are sent to the sixth inference engine client, so that the sixth inference engine client calls the second computing card through the second inference engine server to perform the comparison inference using the target language model and obtain the comparison inference result. 5.The method of claim 1, wherein, The inference server includes a first inference server and a second inference server; the first inference server includes a seventh inference engine client, a third inference engine server, and a third computing card; the second inference server includes an eighth inference engine client, a fourth inference engine server, and a fourth computing card; the step of sending preset test cases to the inference server, so that the inference server performs benchmark inference and comparison inference respectively, to obtain benchmark inference results and comparison inference results, includes: The test cases are sent to the seventh inference engine client, so that the seventh inference engine client calls the third computing card through the third inference engine server to perform benchmark inference using the target language model and obtain the benchmark inference result. The test cases are sent to the eighth inference engine client, so that the eighth inference engine client calls the fourth computing card through the fourth inference engine server to perform the comparison inference using the target language model and obtain the comparison inference result. 6.The method of claim 1, wherein, The standardization process for the baseline inference result and the inference result to be compared includes: If the benchmark inference result and the inference result to be compared are Chinese inference results, then the benchmark inference result and the inference result to be compared are subjected to text segmentation. The baseline inference results and the inference results to be compared after text segmentation are subjected to case normalization. Sign removal is performed on the baseline inference results and the inference results to be compared after case normalization. The baseline inference result and the inference result to be compared are normalized by whitespace symbol normalization after symbol removal, so as to obtain the normalized inference result to be compared and the baseline inference result. 7.The method of claim 1, wherein, The standardization process for the baseline inference result and the inference result to be compared includes: If the baseline reasoning result and the reasoning result to be compared are in English, the English articles are removed from the baseline reasoning result and the reasoning result to be compared. The baseline inference results and the inference results to be compared after removing English articles are case-normalized. Sign removal is performed on the baseline inference results and the inference results to be compared after case normalization. The baseline inference result and the inference result to be compared are normalized by whitespace symbol normalization after symbol removal, so as to obtain the normalized inference result to be compared and the baseline inference result. 8.The method of claim 1, wherein, The generation of inference accuracy and inference hit rate based on the standardized inference results to be compared and the standardized baseline inference results includes: The inference accuracy is calculated based on the number of consistent lexical units between the standardized inference result to be compared and the standardized baseline inference result, and the total number of lexical units in the standardized inference result to be compared. The inference hit rate is calculated based on the number of consistent lexical units between the standardized inference result to be compared and the standardized baseline inference result, and the total number of lexical units in the standardized baseline inference result. 9.A device for testing consistency of a large language model inference, characterized in that, include: The inference result generation unit is used to send preset test cases to the inference server, so that the inference server can perform benchmark inference and comparison inference respectively to obtain benchmark inference result and comparison inference result; The reasoning result processing unit is used to standardize the benchmark reasoning result and the reasoning result to be compared. The reasoning result analysis unit is used to generate reasoning accuracy and reasoning hit rate based on the standardized reasoning results to be compared and the standardized benchmark reasoning results. The reasoning consistency rate generation unit is used to calculate the reasoning consistency rate of the large language model based on the reasoning accuracy rate and the reasoning hit rate.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product comprising computer programs / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 8.

Images