Evaluation method for nutrition management of large language model

Abstract

The invention discloses an evaluation method for nutrition management of a large language model. The method comprises the following steps: constructing an evaluation data set; submitting problems in the assessment data set to a to-be-assessed large language model; collecting answers of the big language model for the question, and classifying the answers according to the content of the answers; calculating the answer accuracy of the large language model according to the classification; the method relates to the technical field of big language model evaluation, reduces the influence of human factors and improves the objectivity and reliability of an evaluation result by constructing an evaluation data set and setting a clear evaluation standard, covers a plurality of aspects of nutriology, and has the advantages that the evaluation data set is established, the evaluation standard is set, the evaluation accuracy is improved, and the evaluation efficiency is improved. Comprising multiple dimensions of food and nutriology, individual and group nutrition management, public nutrition and nutrition education, diet management and the like, and the comprehensive ability of the large language model in the field of nutriology can be comprehensively evaluated.

Description

Technical Field

[0001] The present invention relates to the technical field of large language model evaluation, and in particular to an evaluation method for nutrition management of a large language model. Background Art

[0002] With the development of artificial intelligence technology, especially the progress of natural language processing (NLP), large language models (LLMs) have been widely used in many fields. These models are able to handle complex language tasks such as text generation, machine translation, and question-answering systems. In the field of healthcare, especially in nutrition management, large language models have attracted attention due to their powerful information processing capabilities and rapid response characteristics.

[0003] Traditional nutrition management relies on the expertise of registered dietitians who develop personalized diet plans based on the individual conditions of patients. However, with the aging of the population and the rising incidence of chronic diseases, the demand for professional nutritionists is growing, but the number of nutritionists is difficult to meet this demand. In addition, patients often need to obtain nutrition guidance frequently, which is a huge burden for nutritionists.

[0004] In recent years, some studies have begun to explore the potential application of large language models in nutrition counseling. These models are able to provide a large amount of information in a short period of time and can provide instant feedback based on the user's questions. However, since nutrition management involves complex biochemical processes and individualized medical needs, large language models face severe challenges in providing accurate and safe nutrition advice. Specifically, the model may generate advice based on incomplete or erroneous data, or give outdated information due to a lack of understanding of the latest medical research.

[0005] Currently, the evaluation methods for large language models in nutrition management are still in their infancy, lacking standardized evaluation processes and clear evaluation indicators. Most existing evaluation methods rely on subjective manual judgment and lack objective quantitative indicators. In addition, these methods are often limited to specific knowledge points and fail to fully consider the comprehensive performance of large language models in nutrition management.

[0006] Therefore, it is urgent to develop a systematic and standardized method to evaluate the expertise and application capabilities of large language models in the field of nutrition management to ensure that the information they provide is both accurate and safe. This evaluation method should take into account the breadth and complexity of nutrition, including but not limited to food nutritional composition, dietary needs of different populations, chronic disease management, and public nutrition policies. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides an evaluation method for nutritional management of large language models, which solves the problem that the current evaluation method for nutritional management of large language models is still in its early stages and lacks standardized evaluation processes and clear evaluation indicators.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: an evaluation method for nutritional management of a large language model, the method comprising:

[0009] S1. Construct an evaluation data set, the data set comprising registered dietitian examination questions, the questions including food and nutrition, individual and group nutrition management, public nutrition and nutrition education, and dietary management;

[0010] S2. Submitting the questions in the evaluation data set to the large language model to be evaluated;

[0011] S3, collecting the answers of the large language model to the question, and classifying them according to the content of the answers, wherein the classification includes at least correct, logical error, information error and calculation error;

[0012] S4. Calculate the accuracy of the answer of the large language model according to the classification;

[0013] S5. Performing manual review on the answers of the large language model, wherein the manual review includes labeling and cross-checking by professionals with medical background.

[0014] Preferably, it also includes:

[0015] Construct an evaluation dataset for ketogenic diet ingredient recommendations, which contains at least 95 classifications of common ingredients and beverages;

[0016] asking the large language model whether a particular ingredient is recommended for a ketogenic diet;

[0017] collecting responses from the large language model and comparing them with recommendations from nutrition experts;

[0018] The consistency ratio between the large language model recommendation and the nutrition expert recommendation is calculated.

[0019] Preferably, it also includes:

[0020] Construct a common problem evaluation dataset covering the basic concepts of ketogenic diet, diet implementation and adverse reactions;

[0021] Asking the common questions to the large language model;

[0022] collecting responses from the large language model and comparing them with responses from nutrition experts;

[0023] At least six clinical nutritionists with more than 10 years of working experience will evaluate the answers, and the evaluation will be divided into at least three levels: unacceptable, acceptable, and excellent.

[0024] Preferably, it also includes:

[0025] Defining evaluation criteria for the large language model, including but not limited to information accuracy, logical coherence, and computational accuracy;

[0026] Based on the evaluation criteria, the expertise level of the large language model in the field of nutrition management is determined.

[0027] Preferably, the registered dietitian examination questions are divided into multiple difficulty levels, and the difficulty levels at least include simple, general and difficult questions.

[0028] Preferably, the large language model includes but is not limited to ChatGPT and its subsequent versions.

[0029] Beneficial Effects

[0030] The present invention provides an evaluation method for nutrition management of a large language model, which has the following beneficial effects:

[0031] 1. Improve the standardization and objectivity of evaluation: By constructing an evaluation data set and setting clear evaluation criteria, the method of the present invention can provide a more systematic and standardized evaluation method, reduce the influence of human factors, and improve the objectivity and reliability of the evaluation results.

[0032] 2. Enhance the comprehensiveness of the evaluation: The method of the present invention covers multiple aspects of nutrition, including food and nutrition, individual and group nutrition management, public nutrition and nutrition education, and dietary management, and can comprehensively evaluate the comprehensive capabilities of the large language model in the field of nutrition.

[0033] 3. Improve the practicality of the evaluation: By simulating actual questions in the registered dietitian examination and specific application scenarios of the ketogenic diet, the method of the present invention can not only evaluate the basic theoretical knowledge of the large language model, but also test its performance in practical applications, thereby better evaluating its applicability and effectiveness in real scenarios.

[0034] 3. Ensure the safety of assessment: By carefully classifying and reviewing the answers generated by the large language model, the method of the present invention helps to identify and reduce misleading information in the model output, ensuring the safety and accuracy of nutritional recommendations.

[0035] 4. Promote model improvement: By recording and analyzing the types of model answer errors in detail, the method of the present invention provides a basis for the continuous improvement of large language models, which helps developers to optimize the model in a targeted manner and improve its performance in the field of nutrition management.

[0036] 5. Enhance user trust: By systematically evaluating the performance of large language models in the field of nutrition management, the method of the present invention can enhance the trust of medical workers and patients in these tools, thereby promoting their application in actual medical services.

[0037] 6. Promote academic research and technological progress: This invention provides a standardized evaluation framework, which helps promote academic research and technological progress in related fields and provides a reference standard for subsequent model development and evaluation.

[0038] In summary, the evaluation method proposed in the present invention can not only effectively evaluate the expertise of the large language model in the field of nutrition management, but also promote its improvement and development in practical applications, which has important practical value and social significance. DETAILED DESCRIPTION

[0039] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] The present invention provides a technical solution: an evaluation method for nutritional management of a large language model, the method comprising:

[0041] S1. Construct an evaluation data set, the data set comprising registered dietitian examination questions, the questions including food and nutrition, individual and group nutrition management, public nutrition and nutrition education, and dietary management;

[0042] S2. Submitting the questions in the evaluation data set to the large language model to be evaluated;

[0043] S3, collecting the answers of the large language model to the question, and classifying them according to the content of the answers, wherein the classification includes at least correct, logical error, information error and calculation error;

[0044] S4. Calculate the accuracy of the answer of the large language model according to the classification;

[0045] S5. Performing manual review on the answers of the large language model, wherein the manual review includes labeling and cross-checking by professionals with medical background.

[0046] Further, among them, food and nutrition mainly involves basic concepts of nutrition, food nutrition, food science, hygiene, etc.;

[0047] Individual and group nutrition management, including basic medical knowledge, nutrition assessment and intervention, dietary application for different populations, and disease-nutrition related issues;

[0048] Public nutrition and nutrition education, mainly involving some standards and regulations issued by the Chinese government, nutrition surveys and education, community nutrition and chronic disease management, environment and health;

[0049] Dietary management, including meal planning, preparation and hygiene monitoring;

[0050] Among them, logical errors are finding the necessary information for the answer but not reasoning it into the correct answer;

[0051] Information errors are when the answer uses wrong external information or fails to identify the key information in the question stem;

[0052] A calculation error is a response that resulted in a calculation error or incorrect estimate.

[0053] Also includes:

[0054] Construct an evaluation dataset for ketogenic diet ingredient recommendations, which contains at least 95 classifications of common ingredients and beverages;

[0055] asking the large language model whether a particular ingredient is recommended for a ketogenic diet;

[0056] collecting responses from the large language model and comparing them with recommendations from nutrition experts;

[0057] The consistency ratio between the large language model recommendation and the nutrition expert recommendation is calculated.

[0058] Furthermore, the 95 common ingredients and beverages include meat, seafood, eggs, vegetables, cooking oil, soy products, dairy products, nuts, fruits, starch, high-sugar foods, sugary and alcoholic beverages, etc.

[0059] Also includes:

[0060] Construct a common problem evaluation dataset covering the basic concepts of ketogenic diet, diet implementation and adverse reactions;

[0061] Asking the common questions to the large language model;

[0062] collecting responses from the large language model and comparing them with responses from nutrition experts;

[0063] At least six clinical nutritionists with more than 10 years of working experience will evaluate the answers, and the evaluation will be divided into at least three levels: unacceptable, acceptable, and excellent.

[0064] Furthermore, the evaluation is divided into three levels:

[0065] Unacceptable (the answer has professional loopholes, major language problems, and is generally unacceptable);

[0066] Acceptable (the answer is somewhat technical or part of the content is difficult to understand, but generally acceptable);

[0067] Excellent (the answer is professional, logical, easy to read and understand);

[0068] Also includes:

[0069] Defining evaluation criteria for the large language model, including but not limited to information accuracy, logical coherence, and computational accuracy;

[0070] Based on the evaluation criteria, the expertise level of the large language model in the field of nutrition management is determined.

[0071] The registered dietitian examination questions are divided into multiple difficulty levels, which include at least simple, general and difficult levels.

[0072] The large language model includes but is not limited to ChatGPT and its subsequent versions.

[0073] Example 1: Nutritional knowledge assessment:

[0074] 1.1 Building an evaluation dataset

[0075] Data source: Actual test questions from the official registered dietitian examination guide released by China were selected. These test questions cover four parts: food and nutrition, individual and group nutrition management, public nutrition and nutrition education, and dietary management.

[0076] Difficulty level: According to the official guidelines, the test questions are divided into three difficulty levels: A (easy), B (normal) and C (difficult).

[0077] Question volume: Choose a full exam with 200 questions to ensure comprehensiveness and representativeness of the assessment.

[0078] 1.2 Questioning and Recording

[0079] Questioning method: The test questions are input word by word into the large language model to be evaluated, such as ChatGPT or GPT 4.0.

[0080] Recording method: Record the model's answer to each question and organize it into a document for subsequent analysis.

[0081] 1.3 Answer Analysis

[0082] Classification criteria: Based on the model's response, classify it as correct, logically incorrect, informationally incorrect, or computationally incorrect.

[0083] Logical errors (the answer discovers necessary information but does not reason it into the correct answer);

[0084] Incorrect information (the answer uses incorrect external information or fails to identify key information in the question stem);

[0085] Calculation error (the answer contained a calculation error or incorrect estimate);

[0086] Manual review: Graduate students with a medical background review the model’s responses and cross-check uncertain labels.

[0087] 1.4 Statistical analysis

[0088] Calculate accuracy: Count the number of questions the model answers correctly and calculate the accuracy.

[0089] Error type statistics: Count the proportion of each type of error to understand the weak links of the model;

[0090] According to the above method, ChatGPT and GPT 4.0 were tested respectively, of which ChatGPT had an accuracy of 60.5% (121 / 200) and GPT 4.0 had an accuracy of 74.5% (149 / 200). In the four parts, their accuracy is shown in Table 1, among which the relative accuracy of the food and nutrition part is the lowest, and the relative accuracy of the public nutrition and nutrition education part is the highest. Their accuracy and error type distribution for the three difficulty level questions are also shown in Table 1. Most errors come from inaccurate information, with error rates of 64.6% and 70.6% for ChatGPT and GPT 4.0, respectively.

[0091] Table 1 Comparison of ChatGPT and GPT4.0 nutritionist exam evaluation (correctness and error distribution).

[0092]

[0093] In terms of overall accuracy, GPT 4.0 showed a higher accuracy rate (74.5%) in the registered dietitian exam, which is a significant improvement over ChatGPT's 60.5%. This shows that GPT 4.0 is more comprehensive and accurate in its grasp of nutritional science knowledge. However, both have relatively low accuracy rates in the food and nutrition section, while showing higher accuracy rates in the public nutrition and nutrition education section. The difference in accuracy may be due to GPT 4.0's larger scale, larger data set, and more advanced algorithms, which enable it to more accurately capture and reflect professional knowledge. Most of the errors they made were information errors, which can be improved with more targeted training data.

[0094] Example 2: Evaluation of Common Recommendations for Ketogenic Diets:

[0095] 2.1 Building a food database

[0096] Data source: 95 common food ingredients and beverages in the daily diet of Chinese residents were collected, including meat, seafood, eggs, vegetables, cooking oil, soy products, dairy products, nuts, fruits, starch, high-sugar foods, sugary and alcoholic beverages, etc.

[0097] Classification: Classify ingredients according to their nutritional content and the requirements of the ketogenic diet.

[0098] 2.2 Questioning and Recording

[0099] Questioning: For each ingredient, ask the large language model: "Would you recommend eating (food name) on a ketogenic diet?"

[0100] Recording method: Record the model's recommendations and organize them into a table.

[0101] 2.3 Comparative Analysis

[0102] Reference standard: Refer to the recommendations of the China Low Carb Diet Association on the above ingredients.

[0103] Consistency analysis: The consistency ratio between the statistical model recommendations and the recommendations of the China Low Carb Diet Association was analyzed to analyze the reasons for the inconsistency.

[0104] As described in the above method, the answers obtained by ChatGPT were compared with the expert consensus of the China Low-Carb Diet Association, and the recommendation or not of the obtained 95 foods were analyzed. The simplified results are shown in Table 2. In the category of non-recommended foods, the consistency between expert recommendations and ChatGPT recommendations was 80.70%. The inconsistency in recommended amounts was mainly concentrated in root vegetables and dried beans. In addition, there were inconsistencies in vegetable oils, cocoa butter substitutes, and vegetable fat powders. In the category of recommended foods, the consistency between the two was relatively high (94.87%), and only whole milk and macadamia nuts had recommendation conflicts.

[0105] Table 2. Consistency between ChatGPT and experts on different food recommendations for ketogenic diet

[0106]

[0107]

[0108] Example 3: Basic knowledge assessment of ketogenic diet:

[0109] 3.1 Building a Question Database

[0110] Question source: A selection of 28 frequently asked questions about ketogenic diet therapy, covering basic concepts, diet implementation, and adverse reactions.

[0111] 3.2 Questioning and Recording

[0112] Questioning method: Ask the above questions to the large language model and record its answers.

[0113] 3.3 Expert Review

[0114] Judging team: The judging panel consists of 6 nutritionists with more than 10 years of clinical experience.

[0115] Evaluation criteria: The model's answers are listed alongside expert opinions for evaluation by the jury. The evaluation is divided into three levels: unacceptable, acceptable, and excellent;

[0116] Unacceptable (the answer has professional loopholes, major language problems, and is generally unacceptable);

[0117] Acceptable (the answer is somewhat technical or part of the content is difficult to understand, but generally acceptable);

[0118] Excellent (the answer is professional, logical, easy to read and understand).

[0119] 3.4 Statistical results

[0120] Statistical scoring: Count the scores of each question and analyze the overall quality of the model's answers.

[0121] ChatGPT and the experts' responses to 28 common questions on ketogenic diet therapy were evaluated by 6 nutrition clinical experts. A total of 82 (48.81%) were rated as "excellent", 80 (47.62%) were rated as "acceptable", and 6 (3.57%) were rated as "unacceptable". Among them, 22 questions received all evaluations from 3 points and 2 points, and 6 questions received a 1 point evaluation. Of the 6 responses that received 1 point from the nutritionist judges, 3 responses (questions 10, 20, and 27) were considered to provide less information than the expert's response, 2 responses (questions 8 and 17) addressed the question from a different perspective, and 1 response (question 12) contained inaccurate information and incorrectly recommended avoiding red meat on the ketogenic diet. After evaluation, we queried question 12 in the model 10 more times and did not repeat the error of red meat. This result reminds us that the large language model is a statistical model that generates responses from its training data in an opaque or "black box" manner.

[0122] Through the above specific implementation methods, the present invention provides a systematic and standardized method to evaluate the professional knowledge and application ability of the large language model in the field of nutrition management, so as to ensure that the nutrition advice it provides is both accurate and safe. The specific implementation steps are summarized as follows:

[0123] Constructing a dataset: Select appropriate evaluation data sources to ensure the representativeness and comprehensiveness of the dataset.

[0124] Ask questions and record: Ask questions to the large language model in a predetermined format and record its answers.

[0125] Expert review: Organize experts to review the model’s responses to ensure the accuracy of the evaluation results.

[0126] Data analysis: Categorize and count the model’s responses and calculate accuracy and other metrics.

[0127] Result report: Summarize the analysis results and form an evaluation report, pointing out the advantages of the model and the direction of improvement.

[0128] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions. The sentence "includes an element defined by ... does not exclude the existence of other identical elements in the process, method, article or device including the element".

[0129] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for evaluating nutritional management of a large language model, characterized in that: The method comprises: S1. Construct an evaluation data set, the data set comprising registered dietitian examination questions, the questions including food and nutrition, individual and group nutrition management, public nutrition and nutrition education, and dietary management; S2. Submitting the questions in the evaluation data set to the large language model to be evaluated; S3, collecting the answers of the large language model to the question, and classifying them according to the content of the answers, wherein the classification includes at least correct, logical error, information error and calculation error; S4. Calculate the accuracy of the answer of the large language model according to the classification; S5. Manually review the answers of the large language model, wherein the manual review includes labeling and cross-checking by professionals with medical background.

2. The method for evaluating nutritional management of a large language model according to claim 1, characterized in that: Also includes: Construct an evaluation dataset for ketogenic diet ingredient recommendations, which contains at least 95 classifications of common ingredients and beverages; asking the large language model whether a particular ingredient is recommended for a ketogenic diet; collecting responses from the large language model and comparing them with recommendations from nutrition experts; The consistency ratio between the large language model recommendation and the nutrition expert recommendation is calculated.

3. The evaluation method for nutritional management of a large language model according to claim 1, characterized in that: Also includes: Construct a common evaluation dataset covering the basic concepts of the ketogenic diet, diet implementation, and adverse reactions; Asking the common questions to the large language model; collecting responses from the large language model and comparing them with responses from nutrition experts; At least six clinical nutritionists with more than 10 years of working experience will evaluate the answers, and the evaluation will be divided into at least three levels: unacceptable, acceptable, and excellent.

4. A method for evaluating nutritional management of a large language model according to any one of claims 1 to 3, characterized in that: Also includes: Defining evaluation criteria for the large language model, including but not limited to information accuracy, logical coherence, and computational accuracy; Based on the evaluation criteria, the expertise level of the large language model in the field of nutrition management is determined.

5. The method for evaluating nutritional management of a large language model according to claim 1, characterized in that: The registered dietitian examination questions are divided into multiple difficulty levels, and the difficulty levels at least include simple, general and difficult.

6. A method for evaluating nutritional management of a large language model according to any one of claims 1 to 3, characterized in that: The large language model includes but is not limited to ChatGPT and its subsequent versions.

Images