Device for evaluating problem processing capability of large language model under incomplete information

Abstract

The application provides a device for evaluating the problem processing capability of a large language model under incomplete information, which has the following characteristics: an evaluation entity generation module is used to generate a hierarchical concept list and similar entities according to existing entity guessing data; a simple difficulty guessing module is used to obtain a corresponding simple difficulty guessing result and a simple difficulty guessing round number according to entities and corresponding underlying concepts; a general difficulty guessing module is used to obtain a corresponding general difficulty guessing result and a general difficulty guessing round number according to entities and corresponding intermediate layer concepts; a difficult difficulty guessing module is used to obtain a corresponding difficult difficulty guessing result, a difficult difficulty guessing round number and a question and answer record according to entities, corresponding underlying concepts and similar entities; and an analysis and calculation module is used to calculate a problem processing capability result under incomplete information. In summary, the device can objectively and accurately evaluate the problem processing capability of a large language model under incomplete information in multiple aspects.

Description

Technical Field

[0001] This invention relates to the field of problem-solving evaluation, and more specifically to an evaluation device for problem-solving capabilities under incomplete information of a large language model. Background Technology

[0002] Incomplete information scenarios include missing information, uncertainty, and misinformation, which can be encountered in fields such as business negotiations, military strategy, medical diagnosis, and legal judgments. Problem-solving ability under incomplete information refers to the ability to effectively process existing information, make reasonable inferences, and make decisions in the absence of comprehensive data. This ability is crucial in real life because it is impossible for us to possess all the information needed for decision-making.

[0003] This capability is also crucial for large language models, or LLMs, as it reflects not only their logical reasoning ability but also their ability to adjust strategies in constantly changing environments, thereby significantly improving their decision-making quality and robustness across various domains. Therefore, a natural question arises: Do LLMs possess the ability to solve problems in scenarios with incomplete information?

[0004] Previous research on problem-solving capabilities in incomplete information scenarios has primarily focused on simulating complex decision-making environments through games such as Werewolf, Poker, and Avalon. These games require players to make decisions without complete information, often involving deception and strategic planning to conceal their true identities. "Who's the Spy?" is another incomplete information game that requires players to deduce whether they are a spy based on descriptions from others. In this example, the player holding the "bread" is the spy and needs to distort facts to describe themselves, such as "it usually requires more sugar, fat, and eggs," to avoid being caught. However, even advanced LLMs like GPT-3 and GPT-3.5, which perform well in general natural language processing tasks, still have limitations in effective decision-making under incomplete information environments.

[0005] Information processing is a key capability for solving problems in scenarios with incomplete information, such as Minesweeper and the Twenty Questions game. The Twenty Questions game requires players to ask a series of yes / no questions to guess a given entity, which can effectively assess LLMs' creativity, knowledge retrieval, and multi-hop reasoning abilities. However, games like the Twenty Questions game do not adequately assess LLMs' information processing and problem-solving abilities because they lack the deception and strategic complexity required to identify misleading clues and develop adaptive strategies based on limited or false information. Furthermore, games like Werewolf and Spyfall are highly subjective, making it challenging to effectively assess LLMs' abilities in scenarios with incomplete information.

[0006] In summary, existing evaluation methods are unable to objectively and accurately assess the problem-solving capabilities of large language models under incomplete information. Summary of the Invention

[0007] This invention is made to solve the above-mentioned problems, and its purpose is to provide an evaluation device for problem processing capabilities under incomplete information of a large language model.

[0008] This invention provides an evaluation device for the problem-solving ability of a large language model under incomplete information, used to obtain the result of the problem-solving ability of the large language model under incomplete information. It features the following components: an evaluation entity generation module, used to generate multiple entities and their corresponding hierarchical concept lists and similar entities based on existing entity guessing data, wherein the hierarchical concept lists include bottom-level concepts and intermediate-level concepts; a simple difficulty guessing module, used to enable the large language model to be evaluated to perform simple difficulty entity guessing based on the entities and their corresponding bottom-level concepts, obtaining the corresponding simple difficulty guessing result and the number of simple difficulty guessing rounds; and a normal difficulty guessing module, used to perform simple difficulty entity guessing based on the entities and their corresponding bottom-level concepts. The corresponding intermediate layer concept enables the large language model to be evaluated to perform entity guessing of normal difficulty, obtaining the corresponding normal difficulty guessing result and the number of normal difficulty guessing rounds; the difficult difficulty guessing module is used to enable the large language model to be evaluated to perform difficult difficulty entity guessing based on the entity and its corresponding underlying concept and similar entities, obtaining the corresponding difficult difficulty guessing result, the number of difficult difficulty guessing rounds, and question and answer records; the analysis and calculation module is used to calculate the problem processing capability result under incomplete information based on all easy difficulty guessing results, easy difficulty guessing rounds, normal difficulty guessing results, normal difficulty guessing rounds, difficult difficulty guessing results, difficult difficulty guessing rounds, and question and answer records.

[0009] The problem-solving ability evaluation device under incomplete information of a large language model provided by this invention may also have the following features: the analysis and calculation module includes: a simple difficulty analysis unit, used to obtain the simple difficulty accuracy win rate, the simple difficulty round win rate, and the simple difficulty total win rate based on multiple simple difficulty guess results and the number of simple difficulty guess rounds; a normal difficulty analysis unit, used to obtain the normal difficulty accuracy win rate, the normal difficulty round win rate, and the normal difficulty total win rate based on multiple normal difficulty guess results and the normal difficulty guess rounds; and a difficult difficulty analysis unit, used to obtain the difficult difficulty accuracy win rate, the difficult difficulty round win rate, the difficult difficulty total win rate, and the reasoning score based on multiple difficult difficulty guess results, the difficult difficulty guess rounds, and the question-and-answer records. The problem-solving ability results under incomplete information include the simple difficulty accuracy win rate, the simple difficulty round win rate, the simple difficulty total win rate, the normal difficulty accuracy win rate, the normal difficulty round win rate, the normal difficulty total win rate, the difficult difficulty accuracy win rate, the difficult difficulty round win rate, the difficult difficulty total win rate, and the reasoning score.

[0010] The evaluation device for problem-solving ability under incomplete information of a large language model provided by the present invention may also have the following features: In the simple difficulty guessing module, the simple difficulty entity guessing is a "20 questions" game. The starting point of the "20 questions" game is the underlying concept. When the large language model to be evaluated guesses successfully in the "20 questions" game, the simple difficulty guessing result is 1. When the large language model to be evaluated fails to guess in the "20 questions" game, the simple difficulty guessing result is 0. The number of simple difficulty guessing rounds is the total number of question-and-answer rounds experienced by the large language model to be evaluated in the "20 questions" game.

[0011] The evaluation device for problem-solving ability under incomplete information of a large language model provided by the present invention may also have the following features: wherein, in the simple difficulty analysis unit, the total win rate of simple difficulty is the average of the simple difficulty accuracy win rate and the simple difficulty round win rate, the simple difficulty accuracy win rate is the average of all single simple difficulty accuracy win rates, the single simple difficulty accuracy win rate is the product of the simple difficulty guess result and 100, the simple difficulty round win rate is the average of all single simple difficulty round win rates, and the single simple difficulty round win rate is the product of the reciprocal of the simple difficulty guess round number and 100.

[0012] The evaluation device for problem-solving ability under incomplete information of a large language model provided by the present invention may also have the following features: In the ordinary difficulty guessing module, the ordinary difficulty entity guessing is a "20 questions" game. The starting point of the "20 questions" game is an intermediate layer concept. When the large language model to be evaluated guesses successfully in the "20 questions" game, the ordinary difficulty guessing result is 1. When the large language model to be evaluated fails to guess in the "20 questions" game, the ordinary difficulty guessing result is 0. The number of ordinary difficulty guessing rounds is the total number of question-and-answer rounds experienced by the large language model to be evaluated in the "20 questions" game.

[0013] The evaluation device for problem-solving ability under incomplete information of a large language model provided by the present invention may also have the following features: wherein, in the normal difficulty analysis unit, the total win rate of normal difficulty is the average of the accuracy win rate of normal difficulty and the round win rate of normal difficulty, the accuracy win rate of normal difficulty is the average of all single normal difficulty accuracy win rates, the single normal difficulty accuracy win rate is the product of the normal difficulty guess result and 100, the round win rate of normal difficulty is the average of all single normal difficulty round win rates, and the single normal difficulty round win rate is the product of the reciprocal of the number of rounds guessed for normal difficulty and 100.

[0014] The evaluation device for problem-solving ability under incomplete information of a large language model provided by the present invention may also have the following features: In the difficulty guessing module, the difficulty entity guessing is a "20 questions" game. The starting point of the "20 questions" game is the underlying concept. When the large language model to be evaluated guesses successfully in the "20 questions" game, the difficulty guessing result is 1. When the large language model to be evaluated fails to guess in the "20 questions" game, the difficulty guessing result is 0. The difficulty guessing round number is the total number of question-and-answer rounds experienced by the large language model to be evaluated in the "20 questions" game. The question-and-answer record is all the guessing questions generated by the large language model to be evaluated in the "20 questions" game.

[0015] The evaluation device for problem-solving ability under incomplete information of a large language model provided by this invention may also have the following features: In the difficulty analysis unit, the total win rate of difficulty is the average of the accuracy win rate and the round-count win rate of difficulty; the accuracy win rate of difficulty is the average of all single-round accuracy win rates of difficulty; the single-round accuracy win rate of difficulty is the product of the difficulty guess result and 100; the round-count win rate of difficulty is the average of all single-round win rates of difficulty; and the single-round-count win rate of difficulty is the product of the reciprocal of the round-count guess of difficulty and 100. The reasoning score is the average of all single reasoning scores. The success of misidentification is determined by the question-and-answer records. If it is, the single reasoning score is 1; if not, the single reasoning score is 0. Each guessed question and its corresponding entity in the question-and-answer records is input into the existing large language model to obtain the association results between each guessed question and the entity. The association result is "related" or "unrelated". It is determined whether there are two or more consecutive association results of "related" followed by an association result of "unrelated", or whether there are two or more consecutive "related" and "unrelated" alternating results. If so, the misidentification fails; if not, the misidentification succeeds.

[0016] The evaluation device for problem processing capability under incomplete information of a large language model provided by the present invention may also have the following feature: wherein, in the difficulty guessing module, at least one incorrect answer result is randomly provided, and the incorrect answer result is generated based on similar entities.

[0017] The evaluation device for problem processing capability under incomplete information of a large language model provided by the present invention may also have the following features: wherein the evaluation entity generation module includes: a scoring unit, including an existing large language model, for performing a commonality score on each guessed entity in the entity guessing data; an entity selection unit, for selecting multiple guessed entities as entities from all guessed entities based on the commonality score; a concept generation unit, including an existing large language model, for generating a corresponding hierarchical concept list for each entity; and a similar entity generation unit, including an existing large language model, for generating corresponding similar entities for each entity.

[0018] The role and effect of invention

[0019] According to the present invention, the evaluation device for problem-solving ability under incomplete information of a large language model generates entities, hierarchical concept lists, and similar entities that can be used for question-and-answer guessing through an evaluation entity generation module. Then, it tests the problem-solving ability of the large language model under incomplete information at three levels: easy difficulty guessing module, medium difficulty guessing module, and hard difficulty guessing module. Finally, it uses an analysis and calculation module to obtain the quantitative problem-solving ability results of the large language model under the three different difficulty levels. Therefore, the evaluation device for problem-solving ability under incomplete information of a large language model according to the present invention can objectively and accurately evaluate the problem-solving ability of a large language model under incomplete information in multiple aspects. Attached Figure Description

[0020] Figure 1 This is a block diagram of the large model problem processing and evaluation device in an embodiment of the present invention;

[0021] Figure 2 This is a speculative schematic diagram of an entity that is a tiger in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the process for evaluating problem-solving capabilities under incomplete information in a large language model, as described in an embodiment of the present invention. Detailed Implementation

[0023] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the evaluation device for problem processing capabilities under incomplete information of a large language model.

[0024] This embodiment provides an evaluation device for problem-solving capabilities under incomplete information of a large language model, hereinafter referred to as the large model problem-solving evaluation device, which is used to obtain the results of problem-solving capabilities of the large language model to be evaluated under incomplete information.

[0025] In this embodiment, the large language model is positioned as the questioner in the game, thereby more effectively assessing its ability in deductive and creative questioning, and exploring the domain of understanding and knowledge in depth. Therefore, the large model question processing assessment device is used to guide the large language model to be evaluated to ask questions, and to evaluate the question processing ability of the large language model to be evaluated under incomplete information based on the questions raised.

[0026] In this embodiment, a large-scale model problem-solving evaluation device is constructed based on the "guessing game" of 20 questions proposed by Wal-079sorth in 1882. In addition to evaluating the questioning and knowledge retrieval capabilities of the large language model to be evaluated, it further evaluates key skills such as error identification, backtracking, and path planning, thereby providing a more complete evaluation of the problem-solving capabilities of the large language model to be evaluated under incomplete information.

[0027] Figure 1 This is a block diagram of the large model problem processing and evaluation device in an embodiment of the present invention.

[0028] like Figure 1 As shown, the large model problem processing and evaluation device 100 includes an input module 10, an evaluation entity generation module 20, a simple difficulty guessing module 30, a normal difficulty guessing module 40, a hard difficulty guessing module 50, an analysis and calculation module 60, and a control module 70 that controls the operation of the above modules.

[0029] Input module 10 is used to input the large language model to be evaluated.

[0030] The evaluation entity generation module 20 is used to generate multiple entities and their corresponding hierarchical concept lists and similar entities based on existing entity guessing data, including a scoring unit 201, an entity selection unit 202, a concept generation unit 203, and a similar entity generation unit 204.

[0031] Scoring unit 201 includes an existing large language model used to assign a commonality score to each guessed entity in the entity guessing data. In this embodiment, the existing entity guessing data comes from an open-source dataset of 20 questions, containing 78,890 question-answer pairs. To avoid the large language model to be evaluated being unable to guess uncommon entities due to insufficient knowledge, an existing large language model, specifically GPT3.5 in this embodiment, is used to generate a commonality score for each guessed entity. This score ranges from 1 (uncommon) to 3 (more common).

[0032] The entity selection unit 202 is used to select multiple guessed entities as entities from all guessed entities based on the commonness score. In this embodiment, the top 10,000 common guessed entities are selected as entities.

[0033] The concept generation unit 203 includes an existing large language model, used to generate a corresponding hierarchical concept list for each entity. In this embodiment, the large language model of the concept generation unit 203 is GPT3.5.

[0034] The hierarchical concept list includes at least bottom-level and intermediate-level concepts. Both bottom-level and intermediate-level concepts describe the categories of their corresponding entities. The categories corresponding to intermediate-level concepts are broader or more abstract than those corresponding to bottom-level concepts. In this embodiment, the hierarchical concept list also includes third-level concepts, whose categories are broader or more abstract than those corresponding to intermediate-level concepts. This embodiment does not strictly adhere to academic classifications in generating the hierarchical concept list to maintain the logic and clarity of the concept hierarchy. For example, the hierarchical concept list for the entity "dog" is "[domestic pet, terrestrial mammal, mammal]", rather than the strict biological hierarchical classification "[Canis, Canidae, Carnivora]", thus increasing the difficulty of guessing.

[0035] The similar entity generation unit 204 includes an existing large language model, used to generate corresponding similar entities for each entity. In this embodiment, the large language model of the similar entity generation unit 204 is GPT3.5. For example, the similar entity generated by GPT3.5 for the entity "dog" is "wolf". In this embodiment, the similar entities conform to the underlying concepts of the corresponding entities.

[0036] The simple difficulty guessing module 30 is used to enable the large language model to be evaluated to perform simple difficulty entity guessing based on the entity and its corresponding underlying concept, and to obtain the corresponding simple difficulty guessing result and the number of simple difficulty guessing rounds.

[0037] In the simple difficulty guessing module 30, the simple difficulty entity guessing is a "20 Questions" game. The starting point of this "20 Questions" game is the underlying concept.

[0038] When the large language model under evaluation guesses correctly in the "20 questions" game, the easy difficulty guess result is 1. When the large language model under evaluation fails to guess correctly in the "20 questions" game, the easy difficulty guess result is 0. The easy difficulty guess round number is the total number of question-and-answer rounds experienced by the large language model under evaluation in the "20 questions" game.

[0039] The normal difficulty guessing module 40 is used to enable the large language model to be evaluated to perform normal difficulty entity guessing based on the entity and its corresponding intermediate layer concept, and obtain the corresponding normal difficulty guessing result and the number of normal difficulty guessing rounds.

[0040] In the normal difficulty guessing module 40, the normal difficulty entity guessing is a "20 Questions" game. The starting point of this "20 Questions" game is the intermediate layer concept.

[0041] When the large language model under evaluation guesses correctly in the "20 Questions" game, the guess result for normal difficulty is 1. When the large language model under evaluation fails to guess correctly in the "20 Questions" game, the guess result for normal difficulty is 0. The number of rounds for normal difficulty guessing is the total number of question-and-answer rounds that the large language model under evaluation goes through in the "20 Questions" game.

[0042] The difficulty guessing module 50 is used to enable the large language model to be evaluated to guess the difficulty of entities based on the entities and their corresponding underlying concepts and similar entities, and to obtain the corresponding difficulty guessing results, difficulty guessing rounds, and question and answer records.

[0043] In the difficulty guessing module 50, the difficulty entity guessing is represented by the "20 Questions" game. The starting point of this "20 Questions" game is the underlying concept.

[0044] When the large language model under evaluation guesses correctly in the "20 Questions" game, the difficulty guess result is 1. When the large language model under evaluation fails to guess correctly in the "20 Questions" game, the difficulty guess result is 0. The number of rounds for difficulty guessing is the total number of question-and-answer rounds that the large language model under evaluation goes through in the "20 Questions" game. The question-and-answer record is all the guessing questions generated by the large language model under evaluation in the "20 Questions" game.

[0045] In this embodiment, the process of the "20 questions" game in the simple difficulty guessing module 30, the normal difficulty guessing module 40, and the hard difficulty guessing module 50 is the same. The following describes the game process.

[0046] In this embodiment, a single "20 Questions" game lasts for 20 rounds, or 20 question-and-answer rounds. In each round, the large language model to be evaluated poses a question (i.e., a guessing question) based on all known information. The corresponding guessing module judges and answers the question based on the entity's relevant characteristics. The answer is "yes" or "no," completing one question-and-answer round. The initial known information is the starting point, and the known information for each round consists of all questions generated in previous rounds, their corresponding answers, and the starting point. The game ends when the large language model to be evaluated poses a question that guesses the entity and the answer is "yes," or when the large language model to be evaluated still fails to guess the entity correctly after 20 rounds.

[0047] In this embodiment, the simple difficulty guessing module 30, the normal difficulty guessing module 40, and the hard difficulty guessing module 50 all include an existing large language model. This large language model is used to judge the entity and the question to obtain the answer. In this embodiment, the large language models of the simple difficulty guessing module 30 and the normal difficulty guessing module 40 provide correct answers based on the entity and the question. However, the large language model of the hard difficulty guessing module 50 provides at least one incorrect answer in a single "20 questions" game, thus misleading the evaluation of the question. This incorrect answer is generated using similar entities. For example, if the entity is a tiger and the similar entity is a leopard, then the question is "Does it often live in trees?", the corresponding incorrect answer is "yes," and the corresponding correct answer is "no."

[0048] If the large language model to be evaluated correctly guesses the entity within 20 rounds, the guess is successful; if the large language model to be evaluated fails to guess the entity within 20 rounds, the guess fails. The number of question-and-answer rounds is the total number of question-and-answer rounds that have been experienced by the end of the game. If the entity is not correctly guessed, the number of question-and-answer rounds is 20.

[0049] Figure 2 This is a speculative schematic diagram of an entity representing a tiger in an embodiment of the present invention.

[0050] like Figure 2 As shown, the target entity is a tiger, and the corresponding similar entity is a leopard. The bottom-level concept is "Pantherinae" (Felidae), and the intermediate-level concept is "Felidae" (Cat family). The tiger's characteristics include "striped," "ground-based," "hunting large prey," and "dusk activity." The leopard's characteristics include "nocturnal activity," "hunting small prey," "arboreal activity," and "spotted."

[0051] Building upon the above, in the simple difficulty guessing module 30 (simple mode), the large language model to be evaluated starts guessing entities from "Pantheraea subfamily". Question 1 might be "Does it have stripes?", with the corresponding answer being "yes"; Question 2 might be "Does it often move on the ground?", with the corresponding answer being "yes", and so on, until the game ends.

[0052] In the medium difficulty guessing module 40, the large language model to be evaluated starts guessing entities from "feline". Question 1 might be "Is it large?", with the corresponding answer being "yes"; Question 2 might be "Does it have stripes?", with the corresponding answer being "yes"; Question 3 might be "Does it often move on the ground?", with the corresponding answer being "yes", and so on, until the game ends.

[0053] In the difficult guessing module 50, i.e., the complex mode, the large language model to be evaluated starts guessing entities from "Pantheraea subfamily". Question 1 might be "Does it have stripes?", and the corresponding answer is "yes"; Question 2 might be "Does it often live in trees?", in which case a random incorrect answer is provided, which is a confusion error, and the corresponding answer is "yes"; Question 3 might be "Does it hunt large prey?", and the corresponding answer is "yes"; Question 4 might be "Does it often live at dusk?", and the corresponding answer is "yes", and so on, until the game ends.

[0054] The analysis and calculation module 60 is used to calculate the problem-solving ability result under incomplete information based on all the simple difficulty guessing results, the number of rounds of simple difficulty guessing, the normal difficulty guessing results, the number of rounds of normal difficulty guessing, the difficult difficulty guessing results, the number of rounds of difficult difficulty guessing, and the question and answer records. It includes a simple difficulty analysis unit 601, a normal difficulty analysis unit 602, and a difficult difficulty analysis unit 603.

[0055] The simple difficulty analysis unit 601 is used to obtain the simple difficulty accuracy win rate, the simple difficulty round win rate, and the simple difficulty total win rate based on multiple simple difficulty guess results and the simple difficulty guess round number.

[0056] In the easy difficulty analysis unit 601, the overall win rate for easy difficulty is the average of the easy difficulty accuracy win rate and the easy difficulty round win rate. The easy difficulty accuracy win rate is the average of all single easy difficulty accuracy win rates. The single easy difficulty accuracy win rate is the product of the easy difficulty guess result and 100. The easy difficulty round win rate is the average of all single easy difficulty round win rates. The single easy difficulty round win rate is the product of the reciprocal of the easy difficulty guess round number and 100.

[0057] The normal difficulty analysis unit 602 is used to obtain the normal difficulty accuracy win rate, the normal difficulty round win rate, and the normal difficulty total win rate based on multiple normal difficulty guess results and the normal difficulty guess round number.

[0058] In the Normal Difficulty analysis unit, the overall win rate for Normal Difficulty is the average of the Normal Difficulty accuracy win rate and the Normal Difficulty round-count win rate. The Normal Difficulty accuracy win rate is the average of all individual Normal Difficulty accuracy win rates. The single Normal Difficulty accuracy win rate is the product of the Normal Difficulty guess result and 100. The Normal Difficulty round-count win rate is the average of all individual Normal Difficulty round-count win rates. The single Normal Difficulty round-count win rate is the product of the reciprocal of the Normal Difficulty guess round number and 100.

[0059] The difficulty analysis unit 603 is used to obtain the accuracy win rate, the round win rate, the total win rate, and the reasoning score of the difficulty based on multiple difficulty guessing results, the number of rounds of difficulty guessing, and the question and answer records.

[0060] In the difficulty analysis unit 603, the overall win rate for difficult difficulty is the average of the accuracy win rate and the round-count win rate for difficult difficulty. The accuracy win rate for difficult difficulty is the average of all individual accuracy win rates for difficult difficulty. The accuracy win rate for a single difficult difficulty attempt is the product of the difficult difficulty guess result and 100. The round-count win rate for difficult difficulty is the average of all individual round-count win rates for difficult difficulty. The round-count win rate for a single difficult difficulty attempt is the product of the reciprocal of the number of rounds guessed for difficult difficulty and 100. The reasoning score is the average of all individual reasoning scores.

[0061] The system determines whether misleading identification was successful based on the question-and-answer records. If successful, the single reasoning score is 1; otherwise, it is 0. Each guessed question and its corresponding entity from the question-and-answer records is input into the existing large language model to obtain the association results between each guessed question and the entity, which are either "related" or "irrelevant." The system then determines whether two or more consecutive association results of "related" are followed by an association result of "irrelevant," or whether two or more consecutive "related" and "irrelevant" results alternate. If so, misleading identification fails; otherwise, it succeeds. In this embodiment, the large language model for the difficulty analysis unit 603 is GPT4. It uses the prompt "This is an entity reasoning task. Please determine whether there is an association between the given question and the target entity," along with "Target entity: a" and "Question: b," where a is the entity and b is the corresponding guessed question, to generate the corresponding association results for GPT4.

[0062] In this embodiment, a "20 Questions" game is used to allow the large language model under evaluation to guess entities based on a starting point, thereby objectively assessing the model's world knowledge and reverse thinking ability. This embodiment also tests whether the large language model can avoid navigating to incorrect answers and rethink the correct reasoning path by confusing the results of incorrect evaluation questions. The error detection capability of the large language model is reflected through a single reasoning score.

[0063] Among them, the problem-solving ability results under incomplete information include the accuracy win rate of easy difficulty, the win rate of the number of rounds of easy difficulty, the total win rate of easy difficulty, the accuracy win rate of normal difficulty, the win rate of the number of rounds of normal difficulty, the total win rate of normal difficulty, the accuracy win rate of hard difficulty, the win rate of the number of rounds of hard difficulty, the total win rate of hard difficulty, and the reasoning score.

[0064] The control module 70 stores the control program that controls the operation of each module.

[0065] The following description, in conjunction with the accompanying drawings, explains the process of evaluating the problem-solving capabilities of a large language model with incomplete information using the large model problem-solving evaluation device 100.

[0066] Figure 3 This is a schematic diagram of the process for evaluating problem-solving capabilities under incomplete information in a large language model, as described in an embodiment of the present invention.

[0067] like Figure 3 As shown, the evaluation of problem-solving capabilities under incomplete information in a large language model includes the following steps:

[0068] Step S1: Input the large language model to be evaluated using input module 10.

[0069] Step S2: The simple difficulty guessing module 30 uses the entity and its corresponding underlying concept to make simple difficulty entity guessing of the large language model to be evaluated, and obtains the corresponding simple difficulty guessing result and the number of simple difficulty guessing rounds.

[0070] Step S3: Using the normal difficulty guessing module 40, based on the entity and its corresponding intermediate layer concept, the large language model to be evaluated performs normal difficulty entity guessing, and obtains the corresponding normal difficulty guessing result and the number of normal difficulty guessing rounds.

[0071] Step S4: The difficulty guessing module 50 uses entities and their corresponding underlying concepts and similar entities to enable the large language model to be evaluated to perform difficulty guessing, and obtain the corresponding difficulty guessing results, difficulty guessing rounds, and question and answer records.

[0072] Step S5: The analysis and calculation module 60 calculates the problem-solving ability result under incomplete information based on all the simple difficulty guessing results, the number of rounds of simple difficulty guessing, the normal difficulty guessing results, the number of rounds of normal difficulty guessing, the difficult difficulty guessing results, the number of rounds of difficult difficulty guessing, and the question and answer records.

[0073] In this embodiment, the problem-solving ability evaluation method under incomplete information of existing large language models, namely the llama2-70B model, the llama2-7B model, the Claude2 model, and the GPT3.5 model, is applied. The round-count win rate, accuracy win rate, and inference score of each large language model are shown in the table below:

[0074]

[0075]

[0076] The first column in the table above represents the large language model. The second to fourth columns represent the round-count win rate, accuracy win rate, and reasoning score for each large language model, respectively. The round-count win rate reflects the knowledge level of the large language model, the accuracy win rate reflects the model's logical ability, and the reasoning score represents the model's intelligence level to some extent. Based on the actual manual testing and analysis of each large language model by the evaluators, the problem-solving ability of each large language model under incomplete information, from highest to lowest, is: llama2-70B≈Claude2>GPT3.5≈llama2-7B. The corresponding reasoning scores are also ranked the same. Therefore, the large language model problem-solving evaluation device 100 can objectively and quantitatively assess the problem-solving ability of large language models under incomplete information.

[0077] The role and effect of the embodiments

[0078] According to the evaluation device for problem-solving capabilities under incomplete information of a large language model involved in this embodiment, the entity generation module generates entities, hierarchical concept lists, and similar entities that can be used for question-and-answer guessing. Then, the device tests the problem-solving capabilities of the large language model under incomplete information at three levels: easy difficulty guessing module, medium difficulty guessing module, and hard difficulty guessing module. Finally, the analysis and calculation module obtains the quantitative problem-solving capability results of the large language model under the three different difficulty levels. In summary, this device can objectively and accurately evaluate the problem-solving capabilities of a large language model under incomplete information in multiple aspects.

[0079] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A device for evaluating problem-solving ability under incomplete information of a large language model, used to obtain the intelligence level result of the large language model to be evaluated, characterized in that, include: The evaluation entity generation module is used to generate multiple entities and their corresponding hierarchical concept lists and similar entities based on existing entity guessing data. The hierarchical concept list includes bottom-level concepts and intermediate-level concepts. The bottom-level concepts and the intermediate-level concepts are used to describe the categories of the corresponding entities. The categories corresponding to the intermediate-level concepts are broader or more abstract than the categories corresponding to the bottom-level concepts. The simple difficulty guessing module is used to enable the large language model to be evaluated to perform simple difficulty entity guessing based on the entity and its corresponding underlying concept, and to obtain the corresponding simple difficulty guessing result and the number of simple difficulty guessing rounds; The normal difficulty guessing module is used to enable the large language model to be evaluated to perform normal difficulty entity guessing based on the entity and its corresponding intermediate layer concept, and to obtain the corresponding normal difficulty guessing result and the number of normal difficulty guessing rounds; The difficulty guessing module is used to enable the large language model to be evaluated to guess the difficulty of the entity based on the entity and its corresponding underlying concept and similar entity, and to obtain the corresponding difficulty guessing result, difficulty guessing round number and question and answer record; The analysis and calculation module is used to calculate the intelligence level result based on all the simple difficulty guessing results, the number of rounds of simple difficulty guessing, the normal difficulty guessing results, the number of rounds of normal difficulty guessing, the difficult difficulty guessing results, the number of rounds of difficult difficulty guessing, and the question-and-answer records. The analysis and calculation module includes: The simple difficulty analysis unit is used to obtain the simple difficulty accuracy win rate, the simple difficulty round win rate, and the simple difficulty total win rate based on multiple simple difficulty guess results and the simple difficulty guess round number. The normal difficulty analysis unit is used to obtain the normal difficulty accuracy win rate, the normal difficulty round win rate, and the normal difficulty total win rate based on multiple normal difficulty guess results and the normal difficulty guess round number. The difficulty analysis unit is used to obtain the accuracy win rate, the round-count win rate, the total win rate, and the reasoning score for the difficult difficulty based on multiple difficulty guessing results, the number of rounds of difficulty guessing, and the question-and-answer records. The intelligence level results include the accuracy win rate for easy difficulty, the round-count win rate for easy difficulty, the total win rate for easy difficulty, the accuracy win rate for medium difficulty, the round-count win rate for medium difficulty, the total win rate for medium difficulty, the accuracy win rate for hard difficulty, the round-count win rate for hard difficulty, the total win rate for hard difficulty, and the reasoning score. In the simple difficulty guessing module, the simple difficulty entity guessing is a "20 questions" game. The starting point of the "20 Questions" game is the aforementioned underlying concept. When the large language model to be evaluated successfully guesses in the "20 questions" game, the easy difficulty guess result is 1. When the large language model to be evaluated fails to guess in the "20 questions" game, the easy difficulty guess result is 0. The number of rounds for guessing the difficulty level refers to the total number of question-and-answer rounds experienced by the large language model to be evaluated in the "20 questions" game. The evaluation entity generation module includes: The scoring unit, including an existing large language model, is used to score the commonality of each guessed entity in the entity guessing data. An entity selection unit is configured to select multiple guessed entities as the entity from all the guessed entities based on the publicness score. A concept generation unit, including an existing large language model, is used to generate a corresponding hierarchical concept list for each of the entities; The similar entity generation unit includes an existing large language model, used to generate corresponding similar entities for each of the entities.

2. The evaluation device for problem processing capability under incomplete information of a large language model according to claim 1, characterized in that: in, In the simple difficulty analysis unit, the total win rate on simple difficulty is the average of the accuracy win rate on simple difficulty and the round-count win rate on simple difficulty. The accuracy win rate for easy difficulty is the average of all individual accuracy win rates for easy difficulty. The single-round easy difficulty accuracy win rate is the product of the easy difficulty guess result and 100. The win rate for easy difficulty rounds is the average of the win rates for all single easy difficulty rounds. The win rate for a single easy difficulty round is the product of the reciprocal of the easy difficulty guess round number and 100.

3. The evaluation device for problem processing capability under incomplete information of a large language model according to claim 1, characterized in that: in, In the normal difficulty guessing module, the normal difficulty entity guessing is the "20 Questions" game. The starting point of the "20 Questions" game is the intermediate layer concept. When the large language model to be evaluated successfully guesses in the "20 questions" game, the guess result for the normal difficulty level is 1. When the large language model to be evaluated fails to guess in the "20 questions" game, the guess result for the normal difficulty level is 0. The number of guessing rounds of normal difficulty refers to the total number of question-and-answer rounds experienced by the large language model to be evaluated in the "20 questions" game.

4. The evaluation device for problem processing capability under incomplete information of a large language model according to claim 3, characterized in that: in, In the normal difficulty analysis unit, the total win rate for normal difficulty is the average of the accuracy win rate for normal difficulty and the round-count win rate for normal difficulty. The accuracy win rate for normal difficulty is the average of all individual accuracy win rates for normal difficulty. The single-round accuracy win rate on normal difficulty is the product of the normal difficulty guess result and 100. The win rate for rounds on normal difficulty is the average of the win rates for all single rounds on normal difficulty. The win rate for a single round on normal difficulty is the product of the reciprocal of the number of rounds guessed on normal difficulty and 100.

5. The evaluation device for problem processing capability under incomplete information of a large language model according to claim 1, characterized in that: in, In the difficulty guessing module, the difficulty entity is guessed as a "20 questions" game. The starting point of the "20 Questions" game is the aforementioned underlying concept. When the large language model to be evaluated successfully guesses in the "20 questions" game, the difficulty guess result is 1. When the large language model to be evaluated fails to guess in the "20 questions" game, the difficulty guess result is 0. The number of rounds for guessing the difficulty level refers to the total number of question-and-answer rounds experienced by the large language model to be evaluated in the "20 questions" game. The question-and-answer record consists of all the guessing questions generated by the large language model to be evaluated in the "20 Questions" game.

6. The evaluation device for problem processing capability under incomplete information of a large language model according to claim 5, characterized in that: in, In the difficulty analysis unit, the total win rate for difficult difficulty is the average of the accuracy win rate for difficult difficulty and the round-count win rate for difficult difficulty. The accuracy win rate for the Hard difficulty level is the average of all individual accuracy win rates for the Hard difficulty level. The single-run accuracy win rate for difficult difficulty is the product of the difficult difficulty guess result and 100. The win rate for the most difficult rounds is the average of the win rates for all single rounds on the most difficult difficulty. The win rate for a single round of Hard difficulty is the product of the reciprocal of the predicted round number for Hard difficulty and 100. The reasoning score is the average of all individual reasoning scores. The question-and-answer records are used to determine whether misleading information was successfully identified. If so, the single reasoning score is 1; otherwise, the single reasoning score is 0. Each of the guessed questions and the corresponding entities in the question-and-answer records is input into an existing large language model to obtain the association results between each guessed question and the entity. The association results are either "related" or "unrelated". If the association result is "related" after two or more consecutive "related" results, or if "related" and "unrelated" results alternate twice or more, then the identification of misleading information fails; otherwise, the identification of misleading information succeeds.

7. The evaluation device for problem processing capability under incomplete information of a large language model according to claim 6, characterized in that: in, In the difficulty guessing module, at least one incorrect answer is provided randomly. The incorrect response result is generated based on the similar entities.

Images