Large model reasoning evaluation method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a large model reasoning evaluation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining an answer output by a to-be-tested model according to a target test question; the answers comprise thinking chains and reasoning answers; performing multi-dimensional semantic evaluation on the thinking chain to obtain a first evaluation score; performing rule verification on the thinking chain and the reasoning answer to obtain a second evaluation score; and based on the first evaluation score and the second evaluation score, obtaining an evaluation result of the answer. By adopting the method, the accuracy and comprehensiveness of an evaluation result of large model reasoning can be improved.

Description

Technical Field

[0001] The present application relates to the field of testing and evaluation technology, and in particular to a large-model reasoning evaluation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid rise of GPUs in the field of AI computing, large language models with thought chains have emerged. These models feature a specialized output format, differing from standard large language models in that they first output a thought process before producing a formal answer. Therefore, the logical rationality of the thought process determines the accuracy and logic of the model's output. In a chain-of-thought scenario, the model's intermediate reasoning process, or thought chain, directly influences the final output answer.

[0003] In traditional technology, the output results of large models are often compared and verified to evaluate the accuracy of large model reasoning. However, the evaluation results of large model reasoning are not accurate enough. Summary of the Invention

[0004] Based on this, it is necessary to provide a large-model reasoning evaluation method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy and comprehensiveness of reasoning evaluation results in response to the above technical problems.

[0005] In a first aspect, the present application provides a large model reasoning evaluation method, the method comprising:

[0006] Obtaining the answer output by the model to be tested according to the target test question; the answer includes a thought chain and a reasoning answer;

[0007] Performing a multi-dimensional semantic evaluation on the thought chain to obtain a first evaluation score;

[0008] Performing rule verification on the thought chain and the reasoning answer to obtain a second evaluation score;

[0009] An evaluation result of the answer is obtained based on the first evaluation score and the second evaluation score.

[0010] In one embodiment, obtaining the answer output by the model to be tested according to the target test question includes:

[0011] Obtaining an initial test question for inputting the model to be tested, wherein the initial test question carries a domain label;

[0012] Initiate a data loading request to the test case library based on the domain label;

[0013] receiving key data of a target domain returned by the test case library in response to the data loading request; wherein the domain label corresponds to the target domain in a one-to-one manner;

[0014] Determining the target test questions based on the initial test questions and key data of the target field;

[0015] The target test question is input into the model to be tested, and the answer output by the model to be tested is received.

[0016] In one embodiment, performing a multi-dimensional semantic evaluation on the thought chain to obtain a first evaluation score includes:

[0017] The first evaluation score is obtained by performing a multi-dimensional semantic evaluation on the thought chain through a multi-expert model, wherein the multi-expert model is a large model with a parameter amount greater than or equal to the parameter amount of the model to be tested.

[0018] In one embodiment, before performing multi-dimensional semantic evaluation on the thought chain using a multi-expert model, the method further includes:

[0019] Determining a target prompt word through a test processing program based on the tags in the thought chain and the relationship between the tags and prompt words in the thought chain established in advance; the target prompt word is used to activate the multi-expert model; the tags are used to identify the domain of the test case library from which the thought chain originates;

[0020] The thought chain is input into the multi-expert model in the text format of the target prompt word through the test processing program.

[0021] In one embodiment, performing a multi-dimensional semantic evaluation on the thought chain to obtain the first evaluation score includes:

[0022] Conducting integrity assessment on the thought chain to obtain a step integrity assessment score;

[0023] When the thought chain meets the preset integrity condition, the logical coherence of the thought chain is evaluated to obtain a logical coherence evaluation score;

[0024] When the thought chain satisfies a preset completeness condition and a preset logical coherence condition, evaluating the terminology accuracy of the thought chain to obtain a terminology accuracy evaluation score;

[0025] The first evaluation score is determined according to the step completeness evaluation score, the logical coherence evaluation score, and the terminology accuracy evaluation score.

[0026] In one embodiment, the performing integrity assessment on the thought chain to obtain a step integrity assessment score includes:

[0027] Obtaining the number of tokens in the thought chain and a threshold value of the number of tokens corresponding to the target prompt word; the tokens correspond to steps in the reasoning answer;

[0028] When the number of tokens in the thought chain is greater than or equal to the token number threshold, a step completeness evaluation score is determined according to the number of tokens in the thought chain.

[0029] In one embodiment, the evaluating the logical coherence of the thought chain to obtain a logical coherence evaluation score includes:

[0030] Obtaining each target paragraph in the thought chain and the previous paragraph corresponding to each target paragraph;

[0031] Calculating a similarity change rate of each paragraph based on each target paragraph and each corresponding previous paragraph;

[0032] The logical coherence evaluation score of the thought chain is determined according to the similarity change rate of each paragraph.

[0033] In one embodiment, determining the first evaluation score according to the step completeness evaluation score, the logical coherence evaluation score, and the terminology accuracy evaluation score includes:

[0034] The first evaluation score is obtained based on the step completeness evaluation score and the corresponding first weight, the logical coherence evaluation score and the corresponding second weight, and the terminology accuracy evaluation score and the corresponding third weight.

[0035] In one embodiment, the first weight, the second weight, and the third weight are determined by:

[0036] The first weight, the second weight and the third weight are determined according to the acquired parameter quantity and mapping relationship of the model to be tested; the mapping relationship includes the relationship between the first weight, the second weight, the third weight and the parameter quantity.

[0037] In one embodiment, performing rule verification on the thought chain and the reasoning answer to obtain a second evaluation score includes:

[0038] Obtaining the number of tags in the thought chain, and determining a first rule score according to the number of tags;

[0039] Performing term distance detection on the thought chain and the reasoning answer; determining a second rule score;

[0040] Performing keyword coverage detection on the thought chain and the reasoning answer to determine a third rule score;

[0041] performing a banned word match on the thought chain and the reasoning answer to determine a fourth rule score;

[0042] The second evaluation score is obtained based on the first rule score, the second rule score, the third rule score, and the fourth rule score.

[0043] In a second aspect, the present application further provides a large model reasoning and evaluation device, the device comprising:

[0044] An acquisition module is used to obtain the answer output by the model to be tested according to the target test question; the answer includes a thought chain and a reasoning answer;

[0045] A first evaluation module is used to perform a multi-dimensional semantic evaluation on the thought chain to obtain a first evaluation score;

[0046] A second evaluation module is used to perform rule verification on the thought chain and the reasoning answer to obtain a second evaluation score;

[0047] An evaluation result determination module is configured to obtain an evaluation result of the answer based on the first evaluation score and the second evaluation score.

[0048] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0049] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0050] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0051] The above-mentioned large-model reasoning evaluation method, device, computer equipment, computer-readable storage medium and computer program product, by considering the multi-dimensional semantic evaluation of the thinking chain and the rule evaluation of the reasoning answer and the thinking chain in the evaluation process, evaluates the output answer of the target question with more comprehensive evaluation content, such as "thinking chain + reasoning answer". On the one hand, compared with the traditional technology that only focuses on the correctness of the result and ignores the rationality of the thinking chain process, the accuracy of the evaluation result of the answer output by the model to be tested is improved. On the other hand, on the basis of the evaluation of the reasoning result, the evaluation of the thinking chain is also considered, thereby improving the comprehensiveness of the evaluation of the reasoning process of the model to be tested. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 This is a diagram of an application environment of a large model reasoning and evaluation method in one embodiment;

[0054] Figure 2 1 is a flow chart of a large model reasoning and evaluation method according to an embodiment;

[0055] Figure 3 A schematic diagram of a process for obtaining an answer output by a model to be tested according to a target test question in one embodiment;

[0056] Figure 4 A schematic diagram of a process for a multi-expert evaluation model to perform multi-dimensional semantic evaluation on a thought chain in one embodiment;

[0057] Figure 5 A schematic diagram of a process for evaluating the logical coherence of a thought chain in one embodiment;

[0058] Figure 6 A schematic diagram of a process for verifying rules for thought chains and reasoning answers in one embodiment;

[0059] Figure 7 A schematic flow chart of a large model reasoning and evaluation method in another embodiment;

[0060] Figure 8 It is a structural block diagram of a large model reasoning and evaluation device in one embodiment;

[0061] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] In the related art, the large model reasoning evaluation method provided by the embodiment of the present application can be applied to Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104, or located in the cloud or on another network server. Server 104 obtains the answer output by the model under test based on the target test question; the answer includes a thought chain and a reasoning answer; a multi-dimensional semantic evaluation is performed on the thought chain to obtain a first evaluation score; rule verification is performed on the thought chain and the reasoning answer to obtain a second evaluation score; and an evaluation result of the answer is obtained based on the first and second evaluation scores. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart car devices, projectors, etc. Portable wearable devices can include smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0064] The large model reasoning and evaluation method provided in the embodiment of the present application can also be applied to terminals.

[0065] In an exemplary embodiment, Figure 2 As shown, a large model reasoning evaluation method is provided, which is applied to Figure 1 The server in the example is used to illustrate the method, which includes the following steps S202 to S208.

[0066] Step S202: Obtain the answer output by the model to be tested according to the target test question; the answer includes a thought chain and a reasoning answer.

[0067] The answer corresponds to the target test question; each target test question has one answer. Each answer includes a thought chain and a non-thought chain. The thought chain refers to the process by which the model under test analyzes and thinks about the question. The non-thought chain is the inferred answer. The model under test is a large language model, also called a large model.

[0068] Optionally, the server obtains the corresponding answer output by the model under test based on the target test question through the test system. The answer includes a thought chain and a reasoning answer, with the reasoning answer generated based on the thought chain. The target test question can be directly obtained by the server from the test case library through the test system, or it can be obtained by obtaining an initial test question from the test case library and normalizing the initial test question to obtain the target test question.

[0069] Step S204: perform a multi-dimensional semantic evaluation on the thought chain to obtain a first evaluation score.

[0070] Optionally, the server performs a multi-dimensional semantic evaluation on the thought chain in the answer through a testing system. The multi-dimensional semantic evaluation includes evaluating the completeness, logical coherence, and terminology accuracy of the thought chain, etc.; using multi-dimensional evaluation indicators, the server evaluates from dimensions such as reasoning completeness, terminology accuracy, and logical coherence to obtain a first evaluation score.

[0071] Step S206: perform rule verification on the thought chain and the reasoning answer to obtain a second evaluation score.

[0072] Optionally, the server performs rule verification on the answer, i.e., the thought chain and reasoning answer, through a testing system to obtain a second evaluation score. The rules are pre-configured, such as missing steps, missing required terms, presence of prohibited terms, keyword coverage, etc.

[0073] Step S208: Obtain an evaluation result of the answer based on the first evaluation score and the second evaluation score.

[0074] Optionally, the server obtains an evaluation result of the answer corresponding to the target test question based on weighting the first evaluation score and the second evaluation score through the testing system, such as the weight of the first evaluation score is 60% and the weight of the second evaluation score is 40%.

[0075] In another embodiment, the testing system includes a standardized thought chain generation module, a multi-expert model evaluation module (also known as a multi-expert model), a rule engine module, and a dynamic scoring module. The server obtains the answer output by the model under test based on the target test question through the standardized thought chain generation module of the testing system; the answer includes the thought chain and the reasoned answer. The server performs a multi-dimensional semantic evaluation of the thought chain using the multi-expert model of the testing system to obtain a first evaluation score. The server performs rule verification on the thought chain and the reasoned answer using the rule engine module of the testing system to obtain a second evaluation score. The server obtains an evaluation result of the answer based on the first and second evaluation scores through the dynamic scoring module of the testing system.

[0076] In the above-mentioned large-model reasoning evaluation method, by considering the multi-dimensional semantic evaluation of the thinking chain and the rule evaluation of the reasoning answer and the thinking chain in the evaluation process, the output answer of the target question is evaluated with more comprehensive evaluation content, such as "thinking chain + reasoning answer". On the one hand, compared with traditional technologies that only focus on the correctness of the results and ignore the rationality of the thinking chain process, the accuracy of the evaluation results is improved. On the other hand, the evaluation of the thinking chain is also considered on the basis of the evaluation of the reasoning results, thereby improving the comprehensiveness of the evaluation of the reasoning process of the model to be tested.

[0077] In an exemplary embodiment, Figure 3 As shown, obtaining the answer output by the model to be tested according to the target test question includes steps S302 to S310.

[0078] Step S302: obtaining an initial test question for inputting the model to be tested, where the initial test question carries a domain label.

[0079] In practical applications, large language models capable of outputting thought chains may generate results with varying degrees of randomness. For example, when solving a math problem, one thought chain might first decompose the complex problem into multiple subproblems and then gradually solve them. Another thought chain might directly attempt to find a solution from a holistic perspective, skipping the decomposition step and first applying specific mathematical formulas or theorems for derivation. For another example, when determining the sentiment of a sentence, one thought chain might first analyze key words, such as positive or negative, and then determine sentiment based on their weight and context. Another thought chain might focus more on the overall structure and semantic coherence of the sentence, inferring sentiment by analyzing the sentence's grammar and context. This issue can mask the precision loss of the reasoning framework, necessitating the standardization of the generated thought chains.

[0080] Optionally, the user interacts with the model under test through a terminal, which inputs initial test questions to the model under test. These initial test questions carry domain tags. The domain tags are included to retrieve domain terminology and event library information corresponding to the domain tags from the test case library. The server obtains the initial test questions with domain tags input to the model under test through the test system or a standardized thought chain generation module within the test system. For example, domain tags can include history, humanities, science, etc.

[0081] Step S304: Initiate a data loading request to the test case library based on the domain tag.

[0082] Among them, the test case library covers multiple target fields, such as history, humanities, science, etc., and each target field corresponds to a field label one by one.

[0083] Optionally, after receiving the initial question input by the user, the server, through the test system or its standardized thought chain generation module, initiates a data loading request to the test case library based on the domain tag, aiming to obtain domain knowledge related to the initial test question. Upon receiving the data loading request, the test case library retrieves key data corresponding to the target domain based on the domain tag, including domain terminology and event library information.

[0084] Step S306: receiving key data of the target domain returned by the test case library in response to the data loading request; the domain label corresponds to the target domain in a one-to-one manner.

[0085] Optionally, the server receives key data of the target domain returned by the test case library in response to the data loading request through the test system or the standardized thinking chain generation module in the test system, where the key data includes domain terms and event library information.

[0086] Step S308: Determine target test questions based on the initial test questions and key data of the target domain.

[0087] Optionally, the server merges the key data of the target field with the initial test questions through the test system or a standardized thinking chain generation module in the test system to obtain target test questions.

[0088] Step S310: input the target test question into the model to be tested, and receive the answer output by the model to be tested.

[0089] Optionally, the server inputs the target test question into the model to be tested through the test system or the standardized thinking chain generation module in the test system. After the model to be tested successfully obtains the key data of the target domain from the test case library, such as domain terms and event library information, the model to be tested performs reasoning analysis on the target test question based on the key data and generates an answer corresponding to the target test question. The answer includes a thinking chain and a reasoning answer; the server receives the answer output by the model to be tested through the test system or the standardized thinking chain generation module in the test system.

[0090] In this embodiment, by standardizing the thought chains to be generated, on the one hand, the standards for the generated thought chains are unified. Standardized thought chains can establish a unified reasoning standard, so that the model can reason according to the same logic and rules in different situations, thereby improving the consistency and reliability of the reasoning results; on the other hand, it reduces the randomness of the results generated by the large language model.

[0091] In an exemplary embodiment, performing a multi-dimensional semantic evaluation on the thought chain to obtain a first evaluation score includes: performing a multi-dimensional semantic evaluation on the thought chain through a multi-expert model to obtain the first evaluation score.

[0092] A multi-expert model is a large model with a parameter count equal to or greater than that of the model under test, while maintaining a high-precision data type for the model weights. When evaluating large language models with thought chains, the multi-expert model offers high tolerance and is closer to the actual user experience. This tolerance stems from, firstly, the multi-expert model's unique understanding and perspective on the large model's output. When evaluating the thought chain of the model under test's output, using a multi-expert model avoids the rigidity of a single standard and allows for minor flaws in the model's output that do not affect the core content. This approach complements multiple perspectives and demonstrates high tolerance. Furthermore, the output of a large language model is subject to uncertainty due to the influence of multiple factors. The multi-expert model evaluation system can utilize its comprehensive judgment to address this uncertainty. For example, when the model under test has multiple plausible outputs in different contexts, the multi-expert model can evaluate from its respective areas of expertise, encompassing multiple possibilities. It will not reject the model's output simply because it deviates from a fixed standard, thus providing greater tolerance for the output.

[0093] To ensure real-world user experience, different expert models can simulate the feedback different types of users might have on the output of the large model. Real-world users have diverse knowledge backgrounds, language habits, and usage needs, and multi-expert models can reflect this diversity from different perspectives. Comprehensive evaluation results can better align with the actual needs of different user groups, making the evaluation more closely aligned with the actual user experience. Furthermore, the training data for the multi-expert model includes numerous real-world application scenario examples. When evaluating the output of the large model, these real-world scenarios can be used to assess whether the output meets user expectations and usage habits, based on the perspective of real-world users using the large model.

[0094] In the context of large language models, quantization is the process of converting model weights from higher-precision data types to lower-precision data types to reduce model memory usage. Unquantized models retain higher-precision data types without undergoing this conversion, such as converting from float64 or float16 to lower-precision data types like 8-bit. Unquantized model weights maintain high precision, enabling more accurate calculation and representation of semantic information, reducing the potential loss of precision associated with quantization and providing more reliable results when handling complex semantic evaluation tasks. For example, when handling semantically ambiguous or ambiguous text, they can more precisely grasp meaning, achieving higher accuracy and stability. Furthermore, because the test case library covers multiple domains, unquantized models possess stronger representational capabilities and can better handle the complex and diverse knowledge and semantics of different domains. Compared to models that suffer a certain degree of precision loss after quantization, they can more comprehensively and accurately analyze content from multiple domains. For example, when faced with specialized knowledge in diverse fields such as history and science, unquantized models can more accurately correlate and invoke knowledge for semantic evaluation, making them more adaptable to multi-domain tasks.

[0095] Optionally, the server uses a multi-dimensional evaluation index to evaluate the thinking chain from the dimensions of reasoning completeness, terminology accuracy, logical coherence, etc. through the test system or a multi-expert model in the test system to obtain a first evaluation score.

[0096] In this embodiment, by using a multi-expert model, it is possible to have the advantages of high tolerance and closeness to the actual user experience when evaluating the language model to be tested with a thought chain, and it has high accuracy and stability.

[0097] In an exemplary embodiment, before performing a multi-dimensional semantic evaluation on a thinking chain through a multi-expert model, it also includes: determining the target prompt word through a test processing program based on the tags in the thinking chain and the relationship between the tags and prompt words in the pre-established thinking chain; the target prompt word is used to activate the multi-expert model; the tag is used to identify the field of the test case library of the source of the thinking chain; and the thinking chain is input into the multi-expert model through the test processing program in the text format of the target prompt word.

[0098] Among them, the test system includes not only the standardized thinking chain generation module, the multi-expert model evaluation module (also called multi-expert model), the rule engine module, and the dynamic scoring module, but also a test processing program.

[0099] Optionally, before using the multi-expert model to perform a multi-dimensional semantic evaluation of a thought chain, the server first establishes the relationship between the markers and prompt words in the thought chain through a testing system. For example, if a thought chain originates from the history field, the corresponding target prompt word for the history field is aaa. Alternatively, if a thought chain originates from the science field, the corresponding target prompt word for the science field is bbb. In other words, prompt words containing thought chain markers are compiled for different fields of content.

[0100] The server, through the test system or the test processing program of the test system, determines the target prompt word based on the tokens in the thought chain and the relationship between the tokens and prompt words in the pre-established thought chain. The target prompt word is used to activate the multi-expert model. The token is used to identify the domain of the test case library from which the thought chain originated. The test processing program inputs the thought chain into the multi-expert model in the text format of the target prompt word. For example, the text format of the target prompt word is: Please follow the {xxxx} steps based on the {xxxxx} content and think about and answer the {xxxx} question.

[0101] In this embodiment, the target prompt word is determined by constructing and using the relationship between the markers and prompt words in the thought chain. The target prompt word is used to activate the multi-expert model, thereby avoiding the inability of a single large language model to accurately analyze content in multiple fields.

[0102] In an exemplary embodiment, Figure 4As shown, a multi-dimensional semantic evaluation is performed based on the thought chain to obtain a first evaluation score, including steps S402 to S408.

[0103] Step S402: perform integrity assessment on the thought chain to obtain a step integrity assessment score.

[0104] Optionally, the server performs an integrity assessment on the thought chain through a test system or a multi-expert model of the test system to determine whether the steps involved in the thought chain content are complete, which helps to ensure the comprehensiveness of the entire process or task, and obtains a step integrity assessment score based on the assessment of the integrity of the thought chain.

[0105] Step S404: When the thought chain meets the preset integrity condition, the logical coherence of the thought chain is evaluated to obtain a logical coherence evaluation score.

[0106] The preset integrity condition includes that the integrity of the thought chain is greater than a first threshold, such as 20%.

[0107] In practical applications, the coherence and rationality of the content are considered from a logical perspective, and it is judged whether the logical connection between each part is tight and whether there are logical loopholes or contradictions, so as to evaluate the quality of the content in logical expression.

[0108] Optionally, when the thought chain meets the preset integrity conditions, if the integrity of the thought chain is greater than or equal to 20%, the server evaluates the logical coherence of the thought chain through the test system or the multi-expert model of the test system to obtain a logical coherence evaluation score; if the integrity of the thought chain is less than 20%, no subsequent logical coherence evaluation is performed.

[0109] Step S406: When the thought chain satisfies the preset completeness condition and the preset logical coherence condition, the terminology accuracy of the thought chain is evaluated to obtain a terminology accuracy evaluation score.

[0110] The preset logical coherence condition includes the logical coherence of the thought chain being greater than a second threshold, such as 30%. The first threshold and the second threshold may be the same or different.

[0111] In practical applications, terminology context verification is carried out to check whether the terms are used appropriately and accurately in the context in which they are used, to ensure that the meaning of the terms is consistent with the context, and to avoid misunderstandings caused by improper use of terms.

[0112] Optionally, when the thought chain meets the preset completeness conditions and the preset logical coherence conditions, if the completeness of the thought chain is greater than or equal to 20% and the logical coherence of the thought chain is greater than or equal to 30%, the server evaluates the terminology accuracy of the thought chain through the test system or the multi-expert model of the test system to obtain a terminology accuracy evaluation score.

[0113] Step S408 : determining a first evaluation score according to the step completeness evaluation score, the logical coherence evaluation score, and the terminology accuracy evaluation score.

[0114] Optionally, the server determines the first evaluation score according to the step completeness evaluation score, the logical coherence evaluation score, and the terminology accuracy evaluation score by using a multi-expert model of the test system.

[0115] In this embodiment, the thought chain is evaluated through multiple dimensions such as step completeness, logical coherence, and terminology accuracy, which can improve the comprehensiveness of the evaluation of the answer output by the model to be tested.

[0116] In an exemplary embodiment, a completeness evaluation is performed on a thought chain to obtain a step completeness evaluation score, including: obtaining the number of tags in the thought chain and a tag number threshold corresponding to the target prompt word; the tags correspond to the steps in the reasoning answer; when the number of tags in the thought chain is greater than or equal to the tag number threshold, determining the step completeness evaluation score based on the number of tags in the thought chain.

[0117] Optionally, the answer output by the model to be tested includes a thought chain and a reasoning answer, wherein the thought chain carries a mark, and each mark corresponds to a step in the reasoning answer. If there are 5 marks in the thought chain, there are 5 steps in the reasoning answer. Before performing a multi-dimensional semantic evaluation of the thought chain through a multi-expert model, the server first establishes the relationship between the marks and prompt words in the thought chain through the test system. For content in different fields, prompt words containing thought chain marks for different fields are written. Each prompt word corresponds to a threshold for the number of marks, and the threshold for the number of marks is the node with the least number of marks in the thought chain. For example, if the thought chain comes from the field of science, the target prompt word corresponding to the scientific field is bbb, and the corresponding threshold for the number of marks is N, where N is a positive integer.

[0118] The server uses the multi-expert model of the test system to obtain the number of tokens in the thought chain and compares it with the token number threshold corresponding to the target prompt word determined by the token. If the number of tokens in the thought chain is greater than or equal to the token number threshold, the step completeness assessment score is determined based on the number of tokens in the thought chain. For example, if the number of tokens in the thought chain is 5 and the token number threshold is 2, then the step completeness assessment score is determined based on the 5 tokens in the thought chain. For another example, if the number of tokens in the thought chain is 7 and the token number threshold is 2, then the step completeness assessment score is determined based on the 7 tokens in the thought chain. The step completeness assessment score of 2 is greater than the step completeness assessment score of 1.

[0119] In this embodiment, when the number of tags in the thought chain part output by the tested model meets the threshold of the number of tags corresponding to the prompt word, it is considered that the precision error generated by GPU reasoning is within an acceptable range, and a step integrity assessment score is given; if it does not meet the threshold of the number of tags corresponding to the prompt word, the precision error is unacceptable.

[0120] In an exemplary embodiment, Figure 5 As shown, evaluating the logical coherence of the thought chain to obtain a logical coherence evaluation score includes steps S502 to S506.

[0121] Step S502: Obtain each target paragraph in the thought chain and the previous paragraph corresponding to each target paragraph.

[0122] Optionally, the server uses the multi-expert model of the test system to obtain any paragraph in the thought chain as a target paragraph. For example, the nth paragraph is the target paragraph, which can be denoted as Pn. The target paragraph is the unit of text content being analyzed and is a paragraph sample used to measure logical coherence. The server also obtains the previous paragraph corresponding to the target paragraph. If the target paragraph is Pn, the previous paragraph is Pn-1, which is used to compare with the target paragraph Pn. The similarity change rate between adjacent paragraphs is calculated to assess logical coherence, where n is (1-m), where m is a positive integer greater than or equal to n.

[0123] Step S504 : Calculate the similarity change rate of each paragraph based on each target paragraph and each corresponding previous paragraph.

[0124] Optionally, the server calculates the paragraph similarity change rate for each target paragraph using the multi-expert model of the test system according to the target paragraph and the corresponding previous paragraph using formula (1).

[0125] Formula (1)

[0126] Where ΔS is the rate of change of paragraph similarity, E is the text encoder, Pn is the target paragraph of the text encoder, and Pn-1 is the previous paragraph corresponding to the target paragraph. ||2 represents the 2-norm (also called the Euclidean norm) of the vector. In the formula, the 2-norm is calculated for the vector output by the text encoder E. Here, it is used to measure the "size" of the vector after the encoder processes the text paragraph, and thus calculate the rate of change of similarity between adjacent paragraphs.

[0127] Step S506: Determine the logical coherence evaluation score of the thought chain based on the similarity change rate of each paragraph.

[0128] Optionally, the server uses the multi-expert model of the testing system to determine the logical coherence evaluation score of the thought chain based on the similarity change rate of each paragraph. For example, if the paragraph similarity change rate is within a target range, such as (0.2, 0.6), the server determines the logical coherence evaluation score of the thought chain based on whether the paragraph similarity change rate is within the target range. For example, if there are m paragraphs in the text, there are m-1 paragraph similarity change rates, and the number of paragraph similarity change rates among the m-1 paragraph similarity change rates that meet the target range. The logical coherence evaluation score of the thought chain can also be determined based on the variance of the m-1 paragraph similarity change rates.

[0129] In this embodiment, logical coherence can be determined by using the paragraph similarity change rate.

[0130] In an exemplary embodiment, determining a first evaluation score based on the step completeness evaluation score, the logical coherence evaluation score, and the terminology accuracy evaluation score includes: obtaining the first evaluation score based on the step completeness evaluation score and the corresponding first weight, the logical coherence evaluation score and the corresponding second weight, and the terminology accuracy evaluation score and the corresponding third weight.

[0131] The first weight, the second weight and the third weight are determined according to the parameter quantities of the model to be tested.

[0132] Optionally, the server weights the step completeness evaluation score, the logical coherence evaluation score, and the terminology accuracy evaluation score through a multi-expert model of the test system, where the step completeness evaluation score*the corresponding first weight+the logical coherence evaluation score*the corresponding second weight+the terminology accuracy evaluation score*the corresponding third weight=the first evaluation score.

[0133] In this embodiment, the first weight, second weight and third weight are dynamically determined by the parameter quantities of different models to be tested, and weighted according to the step completeness evaluation score, the logical coherence evaluation score and the terminology accuracy evaluation score to obtain the first evaluation score, thereby reducing the performance impact of the evaluation result brought by the parameter quantity of the model to be tested.

[0134] In an exemplary embodiment, the method for determining the first weight, the second weight and the third weight includes: determining the first weight, the second weight and the third weight based on the obtained parameter quantity and mapping relationship of the model to be tested; the mapping relationship includes the relationship between the first weight, the second weight, the third weight and the parameter quantity.

[0135] Optionally, assume that the weight of the first evaluation score in the evaluation result is 60%, and the weight of the second evaluation score in the evaluation result is 40%. The server obtains the parameter quantity of the model to be tested through the test system, such as 1.5b, 7b, 8b, and 14b. Among them, "b" is the abbreviation of "billion", which represents one billion. A set of corresponding weights are configured for the models to be tested with different parameter quantities. As shown in Table 1. For example, when the parameter quantity of the model to be tested is 1.5b, the first weight of the step completeness evaluation score in the first evaluation score is 24%, the second weight of the logical coherence evaluation score in the first evaluation score is 10%, and the third weight of the term accuracy evaluation score in the first evaluation score is 25%.

[0136] Table 1 Relationship between the first weight, second weight, third weight and parameter quantity

[0137]

[0138] In this embodiment, the parameter performance of models with different parameter amounts is specifically evaluated to avoid the test results being affected by the performance of the model parameter amounts.

[0139] In an exemplary embodiment, Figure 6 As shown, rule verification is performed on the thought chain and the reasoning answer to obtain the second evaluation score, including steps S602 to S610.

[0140] In practice, symbolic verification is used to assess the completeness of thought chains. This verification verifies whether the output of the model under test satisfies specific rules in the test case library. The test case library has specific rules for missing steps, required terms, prohibited terms, and keyword coverage. A step completeness score is then determined based on whether the answer satisfies these specific rules.

[0141] Step S602: Obtain the number of tags in the thought chain, and determine the first rule score according to the number of tags.

[0142] Optionally, the server obtains the number of tokens in the thought chain through the rule engine module of the test system. By accurately counting, it can be determined whether the thought chain meets the specified token quantity standard. The tokens are passed to the model to be tested by inputting the target prompt word, and the model to be tested is required to reason based on the number of tokens. Therefore, when the number of tokens in the thought chain in the output of the model to be tested meets the requirement of the token quantity threshold, it can be considered that the precision error generated by the GPU when reasoning about the model to be tested is acceptable; otherwise, it is unacceptable. If it does not meet the required number, the precision error is unacceptable.

[0143] Optionally, the number of tags is compared with a threshold of the number of tags corresponding to the target prompt word, and points are deducted for each tag that is missing, such as 2 points per tag.

[0144] Step S604: perform term distance detection on the thought chain and the reasoning answer to determine the second rule score.

[0145] Optionally, the server performs term distance detection on the thought chain and reasoning answer through the rule engine module of the test system; by implementing term distance detection, the distance relationship between terms is analyzed, the degree of association between terms is judged, and the rationality and standardization of the content are further verified from the perspective of semantic distance. The number of necessary terms and common terms is determined according to the semantic distance, and the second rule score is determined.

[0146] For example, if the necessary terms in the thought chain and reasoning answer are compared with the threshold number of necessary terms corresponding to the target prompt word, a point will be deducted for each missing necessary term, such as 3 points per term. If the common terms in the thought chain and reasoning answer are compared with the threshold number of common terms corresponding to the target prompt word, a point will be deducted for each missing common term, such as 1 point per term.

[0147] Step S606: Perform keyword coverage detection on the thought chain and the reasoning answer to determine the third rule score.

[0148] Optionally, the server uses the rule engine module of the test system to test the keyword coverage of the thought chain and the reasoning answer to determine the third rule score. For example, the keyword coverage in the thought chain and the reasoning answer is compared with the keyword coverage threshold corresponding to the target prompt word. For each percentage of keyword coverage that is less, a point is deducted, such as 8 points * coverage.

[0149] Optionally, multiple keywords that are logically related to the reasoning process are annotated on the output results of the model to be tested in the Nvidia GPU inference test case library. During the test process, the correlation between the model's reasoning content and the results obtained on the Nvidia GPU is detected by looking up the keyword coverage in the output content of the same test case library when the model to be tested is reasoned on a domestic GPU.

[0150] Step S608: performing a banned word match on the thought chain and the reasoning answer to determine the fourth rule score.

[0151] Optionally, the server uses the rule engine module of the test system to match the thought chain and reasoning answer with banned words to determine the fourth rule score. The server checks whether the thought chain and reasoning answer output by the tested model contain banned words (also known as banned terms) corresponding to the target prompt word. If a banned word match is found, a score is deducted for each banned word occurrence, for example, 5 points per banned word.

[0152] Optionally, the first rule score, the second rule score, the third rule score, and the fourth rule score are all calculated using a deduction system. As shown in Table 2 below, missing steps are determined by counting the number of tags, missing common terms and missing required terms are determined by term distance detection, the presence of banned terms is determined by banned word matching, and keyword coverage is determined by keyword coverage detection.

[0153] Table 2 Deduction items and corresponding deduction values

[0154]

[0155] Step S610 : Obtain a second evaluation score based on the first rule score, the second rule score, the third rule score, and the fourth rule score.

[0156] Optionally, the server adds the first rule score, the second rule score, the third rule score and the fourth rule score through a rule engine module of the test system to obtain a second evaluation score.

[0157] In this embodiment, the second evaluation score can be obtained comprehensively and accurately through the counting of the number of tokens, the detection of term distance, the matching of banned words and the detection of keyword coverage.

[0158] In an exemplary embodiment, Figure 7As shown, the test system uses a standardized thought chain generation module to select an initial test question from the test case library and sends the initial test question to the user's terminal. The user interacts with the model under test through the terminal and inputs the initial test question into the model under test. The initial test question carries a domain tag. The test system obtains the initial test question with the domain tag input into the model under test through the standardized thought chain generation module and initiates a data loading request to the test case library based on the domain tag. After receiving the data loading request, the test case library retrieves the key data corresponding to the target domain based on the domain tag. The key data includes domain terms and event library information. The test system receives the domain terms and event library information of the target domain returned by the test case library in response to the data loading request through the standardized thought chain generation module. The domain terms and event library information of the target domain are merged with the initial test question to obtain the target test question. The target test question is then input into the model under test. After successfully obtaining the key data of the target domain from the test case library, such as domain terms and event library information, the model under test performs reasoning analysis on the target test question based on the key data and generates the answer corresponding to the target test question. The answer includes a thought chain and a reasoned answer. The above process is a method to standardize the thinking chain.

[0159] The test system first establishes the relationship between the tokens and prompt words in a thought chain through a test processing program. For example, if a thought chain originates from the history field, the target prompt word corresponding to the history field is aaa. Similarly, if a thought chain originates from the science field, the target prompt word corresponding to the science field is bbb. In other words, prompt words containing thought chain tokens are compiled for different fields of content. The test system then uses a test processing program to determine the target prompt word based on the tokens in the thought chain and the pre-established relationship between the tokens and prompt words in the thought chain. The target prompt word is used to activate the multi-expert model; the domain of the test case library used to identify the source of the thought chain is marked. The test processing program then inputs the thought chain into the multi-expert model in the text format of the target prompt word. The target prompt word is then assigned a threshold for the number of tokens, the number of common terms, the number of necessary terms, the keyword coverage threshold, and the number of banned words.

[0160] The test system uses a multi-expert model to evaluate the step completeness, logical coherence and terminology accuracy of the thinking chain to obtain the first evaluation score.

[0161] Step Completeness Assessment: The number of tokens in the thought chain is obtained and compared with the threshold number of tokens corresponding to the target prompt word identified by the tokens. If the number of tokens in the thought chain is greater than or equal to the threshold number of tokens, the step completeness assessment score is determined based on the number of tokens in the thought chain.

[0162] Logical coherence assessment: When the thought chain meets the preset integrity conditions, any paragraph in the thought chain is taken as the target paragraph. For example, the nth paragraph is taken as the target paragraph, which can be recorded as Pn. The target paragraph is the text content unit being analyzed and is a paragraph sample used to measure logical coherence. The previous paragraph corresponding to the target paragraph is obtained. When the target paragraph is Pn, the previous paragraph is Pn-1, which is used to compare with the target paragraph pn. The similarity change rate between adjacent paragraphs is calculated to evaluate logical coherence, where n is (1-m), and m is a positive integer greater than or equal to n. For each target paragraph, the paragraph similarity change rate is calculated based on the target paragraph and the corresponding previous paragraph using formula (1). Based on the similarity change rate of each paragraph, the logical coherence assessment score of the thought chain is determined.

[0163] Formula (1)

[0164] Terminology accuracy assessment: When the thought chain meets the preset completeness conditions and the preset logical coherence conditions, the terminology accuracy of the thought chain is assessed to obtain a terminology accuracy assessment score.

[0165] The test system uses a multi-expert model to weight the step completeness assessment score, logical coherence assessment score, and terminology accuracy assessment score. The first assessment score is calculated by multiplying the step completeness assessment score by the first weight + the logical coherence assessment score by the second weight + the terminology accuracy assessment score by the third weight. The first, second, and third weights are determined based on the acquired parameter quantities and mapping relationships of the model under test. The mapping relationships include the relationships between the first, second, and third weights and the parameter quantities. This is shown in Table 1.

[0166] The test system uses a rule matching engine to detect missing steps, required terms, prohibited terms, and keyword coverage in the thought chain and reasoning answers to obtain a second evaluation score.

[0167] Missing step: The test system uses the rule matching engine to compare the number of tokens with the threshold number of tokens corresponding to the target prompt word. For each missing token, the test system deducts points, such as 2 points per token, to obtain the first rule score.

[0168] Missing required and prohibited terms: The test system uses a rule matching engine to compare the required terms in the thought chain and reasoning answer with the required term threshold corresponding to the target prompt. For each missing required term, a point is deducted, such as 3 points. The system also compares the common terms in the thought chain and reasoning answer with the common term threshold corresponding to the target prompt. For each missing common term, a point is deducted, such as 1 point, to determine the second rule score.

[0169] Keyword coverage detection: The test system uses the rule matching engine to compare the keyword coverage in the thought chain and reasoning answer with the keyword coverage threshold corresponding to the target prompt word. For every percentage of keyword coverage that is less, points are deducted, such as 8 points * coverage, to obtain the third rule score.

[0170] Banned Terms: The test system uses a rule matching engine to search for banned terms (also known as banned terms) corresponding to the target prompt word in the thought chains and reasoning answers output by the tested model. Once a banned term match is found, a point is deducted for each banned term, for example, 5 points per occurrence, to obtain the score for the fourth rule. The deduction items and corresponding deduction values ​​are shown in Table 2.

[0171] The test system adds the first rule score, the second rule score, the third rule score and the fourth rule score through the rule matching engine to obtain a second evaluation score.

[0172] The test system uses a dynamic scoring module to weight the first evaluation score and the second evaluation score, such as the weight of the first evaluation score is 60% and the weight of the second evaluation score is 40%, to obtain the evaluation result of the answer corresponding to the target test question of the model to be tested.

[0173] Until all the initial test questions in the test case library have been evaluated by the above method, the evaluation of the reasoning accuracy of the model to be tested is completed.

[0174] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0175] Based on the same inventive concept, the present application also provides a large-model reasoning and evaluation device for implementing the large-model reasoning and evaluation method involved above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more large-model reasoning and evaluation device embodiments provided below can be found in the above-mentioned limitations of the large-model reasoning and evaluation method, and will not be repeated here.

[0176] In an exemplary embodiment, Figure 8 As shown, a large model reasoning evaluation device is provided, including: an acquisition module 801, a first evaluation module 802, a second evaluation module 803 and an evaluation result determination module 804, wherein:

[0177] The acquisition module 801 is used to obtain the answer output by the model to be tested according to the target test question; the answer includes a thought chain and a reasoning answer.

[0178] The first evaluation module 802 is used to perform a multi-dimensional semantic evaluation on the thought chain to obtain a first evaluation score.

[0179] The second evaluation module 803 is used to perform rule verification on the thought chain and the reasoning answer to obtain a second evaluation score.

[0180] The evaluation result determination module 804 is configured to obtain an evaluation result of the answer based on the first evaluation score and the second evaluation score.

[0181] In an exemplary embodiment, the acquisition module 801 is also used to obtain an initial test question input into the model to be tested, the initial test question carries a domain label; based on the domain label, a data loading request is initiated to the test case library; key data of the target domain returned by the test case library in response to the data loading request is received; the domain label corresponds one-to-one to the target domain; based on the initial test question and the key data of the target domain, a target test question is determined; the target test question is input into the model to be tested, and the answer output by the model to be tested is received.

[0182] In an exemplary embodiment, the first evaluation module 802 is also used to perform a multi-dimensional semantic evaluation of the thought chain through a multi-expert model to obtain a first evaluation score, wherein the multi-expert model is a large model with a parameter amount greater than or equal to the parameter amount of the model to be tested.

[0183] In an exemplary embodiment, a large model reasoning and evaluation device includes: a hybrid processing module for determining a target prompt word through a test processing program based on tags in a thinking chain and the relationship between tags and prompt words in a pre-established thinking chain; the target prompt word is used to activate a multi-expert model; tags are used to identify the domain of a test case library from which the thinking chain originates; and the thinking chain is input into the multi-expert model through the test processing program in the text format of the target prompt word.

[0184] In an exemplary embodiment, the first evaluation module 802 is also used to evaluate the integrity of the thought chain to obtain a step integrity evaluation score; when the thought chain meets the preset integrity condition, the logical coherence of the thought chain is evaluated to obtain a logical coherence evaluation score; when the thought chain meets the preset integrity condition and the preset logical coherence condition, the terminology accuracy of the thought chain is evaluated to obtain a terminology accuracy evaluation score; and the first evaluation score is determined based on the step integrity evaluation score, the logical coherence evaluation score and the terminology accuracy evaluation score.

[0185] In an exemplary embodiment, the first evaluation module 802 is also used to obtain the number of tags in the thought chain and the tag number threshold corresponding to the target prompt word; the tags correspond to the steps in the reasoning answer; when the number of tags in the thought chain is greater than or equal to the tag number threshold, the step completeness evaluation score is determined according to the number of tags in the thought chain.

[0186] In an exemplary embodiment, the first evaluation module 802 is also used to obtain each target paragraph in the thought chain and the previous paragraph corresponding to each target paragraph; calculate the similarity change rate of each paragraph based on each target paragraph and the corresponding previous paragraph; and determine the logical coherence evaluation score of the thought chain based on the similarity change rate of each paragraph.

[0187] In an exemplary embodiment, the first evaluation module 802 is further configured to obtain a first evaluation score based on the step completeness evaluation score and the corresponding first weight, the logical coherence evaluation score and the corresponding second weight, and the terminology accuracy evaluation score and the corresponding third weight.

[0188] In an exemplary embodiment, a large model reasoning and evaluation device includes: a weight determination module for determining a first weight, a second weight, and a third weight based on the acquired parameter quantity and mapping relationship of the model to be tested; the mapping relationship includes the relationship between the first weight, the second weight, the third weight and the parameter quantity.

[0189] In an exemplary embodiment, the second evaluation module 803 is also used to obtain the number of tags in the thought chain, and determine the first rule score based on the number of tags; perform term distance detection on the thought chain and the reasoning answer; determine the second rule score; perform keyword coverage detection on the thought chain and the reasoning answer to determine the third rule score; perform banned word matching on the thought chain and the reasoning answer to determine the fourth rule score; and obtain the second evaluation score based on the first rule score, the second rule score, the third rule score and the fourth rule score.

[0190] Each module in the aforementioned large-scale model reasoning and evaluation device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0191] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store large model evaluation data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a large model reasoning evaluation method is implemented.

[0192] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0193] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0194] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0195] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0196] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0197] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0198] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A large model reasoning evaluation method, characterized in that: The method comprises: Obtaining the answer output by the model to be tested according to the target test question; the answer includes a thought chain and a reasoning answer; Performing a multi-dimensional semantic evaluation on the thought chain to obtain a first evaluation score; Performing rule verification on the thought chain and the reasoning answer to obtain a second evaluation score; An evaluation result of the answer is obtained based on the first evaluation score and the second evaluation score.

2. The method according to claim 1, characterized in that Obtaining the answer output by the model to be tested according to the target test question includes: Obtaining an initial test question for inputting the model to be tested, wherein the initial test question carries a domain label; Initiate a data loading request to the test case library based on the domain label; receiving key data of a target domain returned by the test case library in response to the data loading request; wherein the domain label corresponds to the target domain in a one-to-one manner; Determining the target test questions based on the initial test questions and key data of the target field; The target test question is input into the model to be tested, and the answer output by the model to be tested is received.

3. The method according to claim 1, characterized in that The multi-dimensional semantic evaluation of the thought chain to obtain a first evaluation score includes: The first evaluation score is obtained by performing a multi-dimensional semantic evaluation on the thought chain through a multi-expert model, wherein the multi-expert model is a large model with a parameter amount greater than or equal to the parameter amount of the model to be tested.

4. The method according to claim 3, characterized in that Before performing multi-dimensional semantic evaluation on the thought chain through a multi-expert model, the method further includes: Determining a target prompt word through a test processing program based on the tags in the thought chain and the relationship between the tags and prompt words in the thought chain established in advance; the target prompt word is used to activate the multi-expert model; the tags are used to identify the domain of the test case library from which the thought chain originates; The thought chain is input into the multi-expert model in the text format of the target prompt word through the test processing program.

5. The method according to claim 4, characterized in that The performing a multi-dimensional semantic evaluation on the thought chain to obtain the first evaluation score includes: Conducting integrity assessment on the thought chain to obtain a step integrity assessment score; When the thought chain meets the preset integrity condition, the logical coherence of the thought chain is evaluated to obtain a logical coherence evaluation score; When the thought chain satisfies a preset completeness condition and a preset logical coherence condition, evaluating the terminology accuracy of the thought chain to obtain a terminology accuracy evaluation score; The first evaluation score is determined according to the step completeness evaluation score, the logical coherence evaluation score, and the terminology accuracy evaluation score.

6. The method according to claim 5, characterized in that The integrity assessment of the thought chain is performed to obtain a step integrity assessment score, including: Obtaining the number of tokens in the thought chain and a threshold value of the number of tokens corresponding to the target prompt word; the tokens correspond to steps in the reasoning answer; When the number of tokens in the thought chain is greater than or equal to the token number threshold, a step completeness evaluation score is determined according to the number of tokens in the thought chain.

7. The method according to claim 5, characterized in that The logical coherence of the thought chain is evaluated to obtain a logical coherence evaluation score, including: Obtaining each target paragraph in the thought chain and the previous paragraph corresponding to each target paragraph; Calculating a similarity change rate of each paragraph based on each target paragraph and each corresponding previous paragraph; The logical coherence evaluation score of the thought chain is determined according to the similarity change rate of each paragraph.

8. The method according to claim 5, characterized in that The determining the first evaluation score according to the step completeness evaluation score, the logical coherence evaluation score, and the terminology accuracy evaluation score comprises: The first evaluation score is obtained based on the step completeness evaluation score and the corresponding first weight, the logical coherence evaluation score and the corresponding second weight, and the terminology accuracy evaluation score and the corresponding third weight.

9. The method according to claim 8, characterized in that The method for determining the first weight, the second weight, and the third weight includes: The first weight, the second weight and the third weight are determined according to the acquired parameter quantity and mapping relationship of the model to be tested; the mapping relationship includes the relationship between the first weight, the second weight, the third weight and the parameter quantity.

10. The method according to claim 1, characterized in that The performing rule verification on the thought chain and the reasoning answer to obtain a second evaluation score includes: Obtaining the number of tags in the thought chain, and determining a first rule score according to the number of tags; Performing term distance detection on the thought chain and the reasoning answer; determining a second rule score; Performing keyword coverage detection on the thought chain and the reasoning answer to determine a third rule score; performing a banned word match on the thought chain and the reasoning answer to determine a fourth rule score; The second evaluation score is obtained based on the first rule score, the second rule score, the third rule score, and the fourth rule score.

11. A large model reasoning and evaluation device, characterized in that: The device comprises: An acquisition module is used to obtain the answer output by the model to be tested according to the target test question; the answer includes a thought chain and a reasoning answer; A first evaluation module is used to perform a multi-dimensional semantic evaluation on the thought chain to obtain a first evaluation score; A second evaluation module is used to perform rule verification on the thought chain and the reasoning answer to obtain a second evaluation score; An evaluation result determination module is configured to obtain an evaluation result of the answer based on the first evaluation score and the second evaluation score.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images